ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections
Authors: Chun-Han Yao
Amit Raj
Wei-Chih Hung
Michael Rubinstein
Yuanzhen Li
Ming-Hsuan Yang
Varun Jampani
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D articulated shape
animal body estimation
diffusion for 3D
TL;DR: We propose ARTIC3D, a self-supervised framework to reconstruct 3D articulated shapes from a sparse and noisy image collection in-the-wild.

Abstract: Estimating 3D articulated shapes like animal bodies from monocular images is inherently challenging due to the ambiguities of camera viewpoint, pose, texture, lighting, etc. We propose ARTIC3D, a self-supervised framework to reconstruct per-instance 3D shapes from a sparse image collection in-the-wild. Specifically, ARTIC3D is built upon a skeleton-based surface representation and is further guided by 2D diffusion priors from Stable Diffusion. First, we enhance the input images with occlusions/truncation via 2D diffusion to obtain cleaner mask estimates and semantic features. Second, we perform diffusion-guided 3D optimization to estimate shape and texture that are of high-fidelity and faithful to input images. We also propose a novel technique to calculate more stable image-level gradients via diffusion models compared to existing alternatives. Finally, we produce realistic animations by fine-tuning the rendered shape and texture under rigid part transformations. Extensive evaluations on multiple existing datasets as well as newly introduced noisy web image collections with occlusions and truncation demonstrate that ARTIC3D outputs are more robust to noisy images, higher quality in terms of shape and texture details, and more realistic when animated.

ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections
Chun-Han Yao!"
Amit Raj²
Wei-Chih Hung Yuanzhen Li
Michael Rubinstein 2
Ming-Hon Muugn Yang!
Varun Jampani
UC Merced
Google Research
Waymo Yonsei University
Abstract
Estimating 3D articulated shapes like animal bodies from monocular images is inherently challenging due the ambiguities of camera viewpoint, pose, texture, lighting, etc. We propose ARTIC3D, a self-supervised framework to reconstruct per-instance 3D shapes from sparse image collection in-the-wild. Specifically, ARTIC3D is built upon a skeleton- based surface representation and is further guided by 2D diffusion priors from Stable Diffusion. First, we enhance the in- put images with occlusions/truncation via 2D diffusion to obtain cleaner mask estimates and semantic features. Second, we perform diffusion-guided 3D opti- mization to estimate shape and texture that are of high-fidell sigh-fidelity and faithful to input images. We also propose novel technique calculate more stable image-level gradients via diffusion models compared to existing alternatives. Finally, we pro- duce realistic animations by fine-tuning the rendered shape and texture under rigid part transformations. Extensive evaluations on multiple existing datasets as well as newly introduced noisy web image collections with occlusions and truncation demonstrate that ARTIC3D outputs more robust to noisy images, higher quality in terms of shape and texture details, and more realistic when animated.
Introduction
Articulated 3D animal shapes are widely used in applications such as AR/VR, gaming, and content creation. However, the articulated models usually hard obtain manually creating them is labor intensive and 3D scanning real animals in the lab settings is highly infeasible. In this work, aim to automatically estimate high-quality 3D articulated animal shapes directly from sparse and noisy web image collections. This a highly ill-posed problem due to the variations across images with diverse backgrounds, lighting, camera viewpoints, animal poses, shapes, and textures, etc. In addition, we do not assume access to any 3D shape models or per-image annotations like keypoints and camera viewpoints in our in-the-wild setting.
While several recent methods [39, 33, 38] can produce animatable 3D shapes using a skeleton-based neural surface or pre-defined mesh template, the success is largely dependent on large-scale image datasets or manually- -fittered clean images for training or optimization. Moreover, the output 3D shapes and textures are usually unrealistic when viewed from novel viewpoints pose articulations. On the other hand, recent success generative diffusion models [25, 28, 27] shows that one generate high-quality images for a given text prompt. DreamFusion [21] and other recent works [15. 18, 23] further demonstrate the possibility to produce 3D objects/scenes simply using 2D diffusion multi-view supervision. In this work, leverage the powerful 2D diffusion prior
Work done as a student researcher at Google.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Noisy web images
Articulated shapes (input and novel views)
Textured outputs (input and novel views)
Animated
Fine-tuned animation
Figure Learning articulated 3D shapes from noisy web images. We propose ARTIC3D, a diffusion-guiied optimization framework to estimate the 3D shape and texture of articulated animal bodies from sparse and noisy image in the-wild. Results show that ARTIC3D outputs are detailed, animatable, and robust to occlusions truncation.
to learn 3D articulated shapes, aiming to reconstruct and animate 3D animals from sparse noisy online images without any 2D or 3D annotations, Intuitively, one can improve the quality of 3D reconstructions by utilizing diffusion prior similar to the score distillation sampling (SDS) loss proposed in DreamFusion [21]. In our experiments, nonetheless, we observe that naively applying the SDS loss on 3D surface optimization leads unstable and inefficient training, producing undesirable artifacts like noisy surfaces or ambiguous texture.
In this work, we present ARTIC3D (ARTiculated Image Collections in 3D), a diffusion uiffusiin-guided guided optimization framework learn articulated 3D shapes from sparse noisy image collections. We use the articulated part surface and skeleton from Hi-LASSIE [38], which allows explicit part manipulation and animation. We propose novel Decoder-based Accumulative Score Sampling (DASS) module that can effectively leverage 2D diffusion model priors from Stable Diffusion [27] for 3D optimization. In contrast to existing works that pack-propagate image gradients through the latent encoder, we propose decoder-based multi-step strategy in DASS, which we find to provide more stable gradients for 3D optimization. preprocessing scheme that diffusion model reason about occluded truncated regions. In addition, we also propose techniques to create realistic animations from pose articulations.
deal with noisy input images, we propose an input
We analyze ARTIC3D on the Pascal-Part [5] and LASSIE [39] datasets. To better demonstrate the robustness to noisy images, we extend LASSIE animal dataset [39] with noisy web animal images where animals are occluded and truncated. Both qualitative and quantitative results show that ARTIC3D produces 3D shapes and textures that are detailed, faithful to input images, and robust to partial observations. Moreover, our 3D articulated representation enables explicit pose transfer and realistic animation which are not feasible for prior ceffusiionggidded methods with neural volumetric representations. Fig. 1 shows sample 3D reconstructions and applications from ARTIC3D. The main contributions this work are:
We propose a diffusion-guided optimization framework called ARTIC3D, where we reconstruct 3D articulated shapes and textures from sparse noisy online images without using any pre-defined shape templates or per-image annotations like camera viewpoint or keypoints.
We design several strategies to efficiently incorporate 2D diffusion priors in 3D surface optimization, including input preprocessing, decoding diffused latents as image targets, pose exploration, and animation fine-tuning.
We introduce E-LASSIE, an extended LASSIE dataset [39], by collecting and annotating noisy web images with occlusions or truncation evaluate model robustness. Both qualitative and quantitative results show that ARTIC3D outputs have higher-fidelity compared to prior arts in terms of 3D shape details, texture, and animation.

---

2 Related Work
Animal shape and pose estimation. Earlier techniques on animal shape estimation used statistical body models [46, 45] that are learned either using animal figurines or large number of annotated animal images. Some other works [35, 34, 36, 37], use video inputs to learn articulated shapes by exploiting dense correspondence information video. However, these methods rely on optical flow correspondences between video frames, which are not available in our problem setting. Other techniques [12, [1] leverage a parametric mesh model and learn a linear blend skinning from images to obtain posed mesh for different animal categories. MagicPony [33] learns hybrid 3D representation of the animal instance from category specific image collections. Most related to our work are LASSIE [39] and Hi-LASSIE [38], which tackle the same problem setting of recovering 3D shapes from a sparse collection of animal images in the wild using either a manually annotated skeleton template or by discovering
erregory-speciffie template from image collections. However, these approaches require carefully curated input data and fail to handle image collections with partial occlusions, truncation, noise. By leveraging recent advances in diffusion models, support reconstruction on wider variety of input images.
3D reconstruction from sparse images. Several recent works [30, 42, 41, 32, 24, 2, 43, 3] have used implicit representations [19] to learn geometry and appearance from sparse image collections either by training in category specific manner or assuming access to multi-view consistent sparse images during inference. However, most of these approaches demonstrate compelling results only on rigid objects. Zhang et al. [42] is another closely related work that finds a neural surface representation from sparse image collections but requires coarse camera initialization. By learning a part based mesh shape and texture, our framework naturally lends itself to modeling and animating articulated categories such as animals in the wild without any additional requirements on camera parameters.
Diffusion prior for 3D. Diffusion models [27, 28, 44] have recently gained popularity for generating high resolution images guided by various kinds of conditioning inputs. Diffusion models capture the distribution of real data which can be used as score function to guide 3D generation with score- distillation sampling (SDS) loss as first described in DreamFusion [21]. Several recent approaches [18, 15, 17, 29, 26, 23] leverage the SDS loss to generate 3D representations from either text or single or sparse image collections. Drawing inspiration from these lines of work, we propose a novel Decoder- based accumulative Score Sampling (DASS) that exploits the high quality images synthesized by the decoder and demonstrate improved performance over naive SDS loss.
3 Approach
Given 10-30 noisy web images of animal species, ARTIC3D first preprocesses the images via 2D diffusion to obtain cleaner silhouette estimates, semantic features, and 3D skeleton initialization. We then jointly optimizes the camera viewpoint, pose articulation, part shapes and texture for each instance. Finally, we animate the 3D shapes with rigid bone transformations followed by diffusion- guided fine-tuning. Before introducing our diffusion-based strategies improve the quality of 3D outputs, we briefly review the skeleton-based surface representation similar to [39, 38], as well as Stable Diffusion [27] that we use as diffusion prior.
3.1 Preliminaries
While most 3D generation methods optimizes a volumetric neural field to represent 3D rigid ob- jects/scenes, we aim to produce 3D shapes that are articulated and animatable. To enable explicit part manipulation and realistic animation, we adopt a skeleton- based surface representation as in LASSIE [39] and Hi-LASSIE [38]. Nevertheless, unlike [39, 38] which naively sample surface texture from images, we optimize per-part texture images to obtain realistic instance textures from novel views. At a high level, [39, 38) focus on using geometry priors to learn detailed articulated shapes, whereas ARTIC3D further incorporates generative 2D diffusion priors to learn both shapes and texture in
challenging scenario with noisy images.
3D Skeleton. Given a user-specified reference image in the collection, Hi-LASSIE [38] automatically discovers 3D skeleton based on the geometric and semantic cues from DINO-VIT [4] feature clusters. The skeleton initializes a set of 3D joints and primitive part shapes, providing a good constraint of part transformation and connectivity. In our framework, obtain cleaner feature
3

---

Input preprecessing (Sec. 3,3)
articulated shape and optimization (Sec. 3,4)
Animation 3.5)
H8: LASSIE losses
Test recon loss
Right part motiion
Part strfhees
Temperal consstency
Surfect points
K.A.
wab noiey
mage
image
Per frame sarget
DASS: Decnder-based Accumulative Score Sampling (Sec. 3.2)
"A plitto of
Gradiest Dow
Datillation
FyAans
Stup gradicn
Fncrnker
Decoder
Image.x
Latents
Image larget
Enhanced images & Suture clusters
Inanee gnalianis
Figure 2: ARTIC3D overview, Given sparse web images of an animal species, ARTIC3D estimates the camera viewpoint, articulated pose, 3D part shapes, and surface texture for each instance. We propose novel DASS module to efficiently compute image-level gradients from stable diffusion, which are applied 1) input preprocessing, 2) shape and texture optimization, and 3) animation.
clusters by diffusing input images, then applying Hi-LASSIE as an off-the-shelf skeleton discovery method. For fair comparison with existing works, we use the same reference image for skeleton discovery as in [38] in our experiments. Please refer to [38] for further details on skeleton discovery.
Neural part surfaces. Following [38]. using the discovered 3D skeleton, reconstruct 3D part corresponding to each skeleton bone via deformable neural surface [42]. The neural surfaces are unit
parameterized by multi-layer perceptron networks (MIPs), mapping 3D surface points sphere to their xyz deformation. Given m uniformly sampled 3D points X E R33mm on a spherical surface,
deform the 3D shape of the i-th part through the part MI.P X - (X)). Then, the part surfaces are rigidly transformed by the scaling s; R, rotation R; E [R3x3 and translation R of each skeleton part i. The transformed part surface points V; in the global coordinate can be written as: V; = s; (()) + 4 Please refer to [38] for further details.
Stable Diffusion architecture. Stable Diffusion (SD) [27] is state-of-the art text-to-image gen- crative model that can synthesize high-quality images given a text prompt. SD mainly consists of 3 components: An image encoder & that encodes given image into latent code decoder network that converts the latent code back to image pixels; and a U-Net denoiser co that can iteratively denoise noisy latent code. We use SD as diffusion prior in our framework.
3.2 Decoder-based Accumulative Score Sampling (DASS)
To leverage the 2D diffusion prior for 3D shape learning. DreamFusion [21] proposes score distillation sampling (SDS) loss to distill the images rendered from random views and propagate the image-level gradients to Neural Radiance Field (NeRF) parameters. To reduce the computational cost and improve training stability, recent works like Latent-NeRF [18] and Magic3D [15] perform distillation on the low-resolution latent codes SD and back-proeagate the gradients through the SD image encoder E. Formally, let x be rendered image from 3D model and denote its latent codes from the SD image encoder E. At each score distillation iteration, the latent codes z are noised to random time step f, denoted 2t, and denoised by the U-Net denoiser es of the diffusion model. The image-level SDS gradients can then expressed as:
0=
V LDDS ( t)
Oz
where y denotes the guiding text embedding, is the random noise added to the latent codes, and a constant multiplier which depends on diffusion timestep The denoiser 6s uses guidance scale Wy to balance the text guidance and classifier-free guidance [8] of unconditional model.

---

(a) Clean v.s. noised background
(b) Encoder v.s. Decoder
Image
Input (clean)
Diffused
Input (noised)
Diffused
Encoder (SDS)
Decoder (DASS)
(c) Accumulation steps
(d) Diffusion timestep /
(c) Guidance weight Wy
n
X1 16 (DASS)
(DSS)
(DASS)
0.2
0.5
0.8
W/g W
W3 40
Figure 3: Ablative visualizations of the DASS method. From the example input image (top left), we show the updated image after one optimization iteration using various ways to obtain image-level gradients or parameter settings: (a) shows that noised background in the input image encourages DASS to hallucinate the missing parts; (b) compares the standard SDS mrok--ropagate gradients through encoder) and our DASS (decoder-based) losses; (c) justifies our accumulating latent gradient approach as it leads to cleaner decoded output; (d) indicates that small timestep mostly modifies the texture, whereas large timestep changes the geometry more (sometimes removes or creates body parts); (e) demonstrates high-contrast colors and slightly disproportioned body with higher guidance weight (diffusion prior is biased towards larger heads and frontal views). Note that (b) uses the clean input in (a) for better visualization, whereas (c). are obtained from the noised input.
Although this common SDS loss is effective in generating NeRFs from text, we observe that naively applying it in our framework leads to unstable and inefficient training. As shown in Fig. (b), the SDS gradients muck-propagated through the encoder are often quite noisy, causing undesirable artifacts on 3D shapes and texture. Moreover, it requires the extra computation and memory usage for gradient back propagation, limiting the training batch size and thus decreasing stability.
To mitigate these issues, we propose a novel Decoder-based Accumulative Score Sampling (DASS) module, an alternative calculate pixel gradients that are cleaner and more efficient. Fig. illustrates the proposed DASS module. At a high level, given an input image x, we obtain a denoised image x' from the decoder as reconstruction target, based on our observation that decoded outputs generally less noisy. As shown in Fig. 2, we pass a rendered image through the encoder & to obtain low-resolution latent codes, update the latents for steps via score distillation, then decode the updated latents with the decoder D as an image target. Formally, instead of explicitly calculating the partial derivative 0z/Ox, we use x D(= V2) as a proxy Vx, where V2 is the accumulated latent gradients over steps. This makes a linear assumption on D around latents 2, which we empirically find effective to approximate the pixel gradients. The target image x' D(2 V2) can be directly used as an updated input (Section 3.3) or to compute a pixel-level DASS loss Ldass [| D(2 Vz)) 2 in 3D optimization (Section 3.4). Since the DASS module only involves one forward pass of the encoder and decoder, it costs roughly half the memory consumption during training compared to the original SDS loss.
The visualizations in Fig. 3 demonstrate that DASS produces cleaner images than the original SDS loss one training step (b), and that the accumulated gradients can effectively reduce noise and fill in the missing parts (c). Moreover, we show that adding random noise to the background pixels can facilitate the shape completion by DASS (a). We also perform ablative analyses on other diffusion parameters like noise timestep (d) and guidance weight (e) in Fig. 3. In general, ARTIC3D favors moderate accumulation steps n E (3. 10) and lower timestep t (0.2, 0.5) since higher variance can lead to 3D results that are not faithful to the input images. Also, we use a lower guidance weight Wy E (10,30) so that our results do not suffer from over saturation effects common in prior works due to high guidance scale in SDS loss. We apply the DASS module in three different stages in our framework: input preprocessing, shape and texture optimization, and animation fine-tuning, which we describe in the following sections.
5

---

3.3 Input preprocessing for noisy images
Animal bodies in real-world images often have ambiguous appearance caused by noisy texture, dim lighting, occlusions, or truncation, as shown in Fig. 4. To better deal with noisy or partial observations, we propose novel method to enhance the image quality and complete the missing parts. Given sparse image collection { E " x3} (j {1...., } and N is typically between 10-30) of an animal species, we aim to obtain accurate silhouettes estimates {M, E "B* and clean semantic features {K3 E Rxxxxd for each instance. As shown in Fig. 2. we roughly estimate the foreground masks via clustering salient features extracted by a trained DINO- ViT [4] network. Then, we apply DASS to diffuse the background-man -makeed images, resulting in animal bodies with cleaner texture and complete shapes. Formally, we obtain an updated image I' by D(= Vz), where ET). Here, DASS serves as an image denoising and inpainting module, which can effectively generate a high-quality version of noisy input via n latent updates and single forward pass of 'D Following the poise-and-denois nature of diffusion models, we show in Fig. 3 (a) that manually adding Gaussian noise to the background pixels in input image encourages DASS to hallucinate the occluded parts while mostly preserving the visible regions. Finally, we re-apply DINO-ViT feature extraction and clustering [1] on the diffused images to obtain cleaner and more complete masks as well as semantic features. Fig. 2 (left) shows sample noisy input images and the corresponding output enhanced images and feature clusters. Note that Farm3D [9] uses SD [27] to generate animal images from text for 3D training, which, however, often contain irregular shapes (e.g., horses with 5 legs). On the contrary, our preprocessed images are more suitable for the sparse-image optimization framework since our goal is to reconstruct 3D shape and texture that are realistic and faithful to the input images.
3.4 Diffusion-guided optimization of shape and texture
Given the preprocessed images, silhouette estimates, and semantic features, we jointly optimize the camera viewpoint, pose articulation, 3D part shapes, and texture. Since we do not assume any 2D or 3D annotations, we follow Hi-LASSIE [38] and adopt an by-ssthes-s approach to reconstruct 3D shape and texture that are faithful to the input images. That is, we render the 3D part using a differentiable renderer [16] and compare them with the 2D images, pseudo ground-truth silhouettes, and DINO-ViT features. Fig. 2 (top) illustrates the shape and texture optimization.
LASSIE and Hi-LASSIE losses. Given the rendered silhouette MS and pseudo ground-truth A of instance j. the silhouette loss Las can be written as: Caul = >EIl M1 13.. LASSIE [39] and Hi-LASSIE [38] further leverage the 2D correspondence of DINO features between images of the same animal class to define a semantic consistency loss Leem. Luom can be interpreted as the Chamfer distance between 3D surface points and 2D pixels, enforcing the aggregated 3D point features to project closer to the similar pixel features in all images. To regularize the pose articulations and part shapes, [39, 38] also apply a part rotation loss Lror. Laplacian mesh regularization Clop- and surface normal loss Lnorm- The part rotation loss Lrot = E,
RI22 limits the angle offsets from resting pose, where R is the part rotations of instance j and R denotes the part rotations of shared resting pose. Ltop and Lnorm encourage smooth 3D surfaces by pulling each vertex towards the center of its neighbors and enforcing neighboring faces to have similar normals, respectively. We omit the details and refer the readers to [39, 38). Considering that the reconstruction (Luit, Leem) and regularization (Lrot. Liap: Lnorm) losses are generic and effective on articulated shapes, we use them in ARTIC3D along with novel texture reconstruction and DASS modules.
Texture reconstruction. Both [39, 38] directly sample texture from input RGB, resulting in unrealis- tic textures in occluded regions. To obtain more realistic textures, we also optimize a texture image T, for each part. The vertex colors C' R3xm are sampled via the pre-defined UV mapping S of surface points X. Formally, the surface color sampling of part i can be expressed as C; = TXX)( The sampled surface texture are then symmetrized according to the symmetry plane defined in the 3D skeleton. Note that the texture images are optimized per instance since the animals in web images can have diverse texture. Similar to the Clap, we enforce the surface texture to be close to input image when rendered from the estimated input view. The texture reconstruction loss is defined as:
Lieat - ElA o (fa - 13)1
(2)
6

---

where i denotes the clean input image of instance j after input preprocessing and M denotes the corresponding animal mask; I is the rendered RGB image from the estimated 3D shape and texture; and @ denotes element-wise product. The reconstruction loss masked by the estimated foreground silhouette so that the texture optimization is only effected by the visible non-occluded animal pixels.
Distilling 3D reconstruction. In addition to the aforementioned losses, we propose to increase the shape and texture details by distilling 3D reconstruction. Here, we use DASS as a critic to evaluate how well a 3ID reconstruction looks in its 2D renders, and calculate pixel gradients from the image target. Similar to prior diffusion-based methods [21, 18, 15], we render the 3D surfaces with random viewpoints, lighting, and background colors during training. Moreover, we design a pose exploration scheme to densify the articulation space in our sparse-image scenario. In particular, we randomly interpolate the estimated bone rotation ( RT²)) of two instances (ii, and generate new instance with novel pose R' aR + (1 R) for rendering, where a E (0, 1) is a random scalar, As such, we can better constrain the part deformation by diffusion prior and prevent irregular shape or disconnection between parts. As shown in Fig. 2, we then diffuse the latent codes of rendered images and obtain pixel gradients from the DASS module. The resulting gradients are back-propagatee to update the part surface texture, deformation MIP, bone transformation and camera viewpoints. In our experiments, we observe that the RGB gradients do not propagate well through the SoftRas [16] blending function, and we thus modify it with a layered blending approach proposed in [20].
Optimization details. The overall optimization objective can be expressed the weighted sum of all the losses L = -ieg orhi, where & = {sil, sem, rot, lap, norm, text, dass} as described above. We optimize the shared and instance-specifee shapes in two stages. That is, we first update the shared part MLPs along with camera viewpoints and pose parameters. Then, we fine-tune the instance-specific part MLPs and optimize texture images for each instance. All model parameters updated using an Adam optimizer [10]. We render the images at 512 x512 resolution and at 128 x 128 for the part texture images. More optimization details are described in the supplemental material.
3.5 Animation fine-tuning
One can easily animate the resulting 3D articulated animals by gradually rotating the skeleton bones and their corresponding parts surfaces. However, the rigid part transformations often result in disconnected shapes or texture around the joints. To improve the rendered animation in 2D, one can naively use DASS frame-by-framd on sequence of articulated shapes. However this can produce artifacts like color flickering and shape inconsistency across the frames. As a remedy, we further propose a fine-tuning step, called Temporal-DASS (T-DASS), to generate high-quality and temporally consistent 2D animations based on the ARTIC3D outputs. Given a sequence of part transformations from simple interpolation across instances or motion re-targeting, we render the 3D surfaces as video frames (JA E HW (k E {1, K}}} and encode them into latent codes {2x E xxxx through the SD encoder E. Then, we design a reconstruction loss Crecon and temporal consistency loss Ltemp to fine-tune the animation in the latent space. Similar DASS, we obtain the reconstruction targets } by accumulating latent SDS gradients VZu for multiple steps: 2/2 = 2 Vzk. The reconstruction loss can then be written as: Crecon = EE1(2s 12?. To enforce temporal consistency, we exploit our 3D surface outputs and calculate accurate 2D correspondences across neighboring frames. Specifically, for each latent pixel in frame 2k, we find the closest visible 3D surfaces via mesh rasterization, then backtrack their 2D projection in frame 2k-1, forming a dense 2D flow field FA E R hxxxx Intuitively, the corresponding pixels should have similar latent codes. Hence, use F& to perform temporal warpping the latent codes 2k-1, denoted as: warp( Fx), and define Lremp as:
K Ciemp = EI(EE - warp(26-1, F2) k=2
(3)
We fine-tune the latent codes {2x} with Crecon and Ltemp, where {Fx} are pre-computed and {2(] are updated in each iteration. Finally, we can simply obtain the RGB video frames by passing the optimized latent codes through the SD decoder (D(=s)). The proposed Lreemn encourages better shape and texture details in each frame, and Ltemp can effectively regularize latent updates temporally. Note that T-DASS optimizes the latent codes and takes temporal consistency into account, which different from DASS which operates on each image individually.
7

---

Figure 4: E-LASSIE samples. We extend LASSIE [39] image sets with 15 occluded or truncated images per animal class and annotate the 2D keypoints for evaluation. These noisy images pose great challenges sparse-image optimization since the per-instance 3D shapes can easily overfit to the visible parts and ignore the rest.
4 Experiments
Datasets.
Following [39, 38], evaluate ARTIC3D on the Pascal-Part [5] and LASSIE [39] images. From Pascal-Part, we obtain images of horse, cow, and sheep, as well as their 2D keypoints automatically computed using the ground-truth 2D part masks. The LASSIE dataset includes web images of other animal species (zebra, tiger, giraffe, elephant, kangaroo, and penguin) and 2D keypoint annotations. Each image collection contains roughly 30 images of different instances with diverse appearances, which are manually filtered so that the animal bodies are fully visible in the images. To evaluate the model robustness in a more practical setting, we extend the LASSIE image sets with several noisy images where the animals are occluded or truncated. In particular, we collect 15 additional web images (CC-licensed) per class and annotate the 2D keypoints for evaluation. We call the extended image sets E-LASSIE and show some examples in Fig. 4. For the experiments on E-LASSIE, we optimize and evaluate on all the 45 images in each set.
Baselines. We mainly compare ARTIC3D with LASSIE [39] and Hi-LASSIE [38] as we deal with the same problem setting, namely sparse image optimization for articulated animal shapes. For reference, we also compare the results with several lcarning-basee based methods like A-CSM [11], MagicPony [15], and Farm3D [9]. Note that these approaches
not directly comparable to
ARTIC3D since they train a feedforward network on large-scale image sets (not available in our scenario). Although related, some other recent works on 3D surface reconstruction either cannot handle articulations [12, 14, 31, 40] or require different inputs [13, 34, 36]. As a stronger baseline, we implement Hi- Hi-LASSIE+, incorporating the standard SDS loss in [27, 15, 18]
latent gradients through encoder) during Hi-LASSIE [38] optimization for shape and texture.
Evaluation metrics. Considering the lack of ground- tround-truth truth 3D annotations in our datasets, we follow a common practice [45, 11, 39, 38] to use keypoint transfer accuracy as a quantitative metric to evaluate 3D reconstruction. For each pair of images, we map the annotated 2D keypoints source image onto the canonical 3D surfaces, re-project them to the target image via the estimated camera, pose, and shape, and compare the transferred keypoints with target annotations. To further evaluate the quality of textured outputs, we compute CLIPP [22] features of the 3D output renders under densely sampled viewpoints, and calculate the feature similarity against text prompt well input images. While most prior arts on 3D shape generation [21, 23] only evaluate the image-text similarity, we also evaluate the image-image similarity since our outputs should be faithful to both the category-level textual description as well as -nstance-specific input images. We use a text prompt: "A photo of ** for each animal class **" in our experiments. A CLIP ViT-B/32 model is used compute the average feature similarity over 36 uniformly sampled azimuth renders at a fixed elevation of 30 degrees. We show the main results here and more quantitative and qualitative comparisons in the supplemental material, including animation videos, user study, and more detailed ablation study.
Qualitative results. Fig. 1 shows some sample outputs of ARTIC3D. In Fig. 5, we compare the visual results of Hi-LASSIE, Hi-LASSIE+, and ARTIC3D on the E-LASSIE images. Both Hi-LASSIE and Hi-LASSIE+ produce irregular pose and shape for the invisible parts. Regarding surface texture, Hi-LASSIE reconstructs faithful texture from the input view but noisy in novel views, since naively samples vertex colors from the input images. The output texture of Hi-LASSIE+ is generally less
8

---

Input
Hi-LASSIE
Hi-LASSIE+
ArticD
(ours)
Figure 5: Visual comparison of ARTIC3D and other baselines. For each input image, we show the 3D textured outputs from input (upper) and novel (lower) views. The results demonstrate that ARTIC3D is more robust to noisy images with occlusions or truncation, producing 3D shape and texture that are detailed and faithful the input images.
noisy thanks to the SDS loss. By comparison, ARTIC3D accurately estimates the camera viewpoint, pose, shape, and texture even with the presence of occlusions truncation. Note that there exists a tradeoff between realism and faithfulness to input images due to the highly ill-posed nature of our problem setting, especially for the unseen/occluded surface. For instance, whether colored or black-and-whit output is preferred given grey-scale/poorly illuminated image, or whether we preserve the noisy texture (small occlusions like dirt, water splash, shadow caused by rough surface / objects) in the original image. Since the dense surface visibility is hard to obtain, we optimize the shape and texture that are slightly biased towards realism (detailed and clean texture that resembles the animal class) in this paper. In our ablative analysis of the DASS module (Fig. 3), we show that we can control the realism-faithfulness tradeoff by tuning the diffusion timestep and guidance weight Wgg. Specifically, larger t and Wy allows DASS to hallucinate shape and texture that are not present in the original image. addition, one can enforce more faithful reconstruction by setting higher weight of the texture reconstruction loss Oteet- More qualitative comparisons against other baselines can be found in the supplemental material.
Quantitative comparisons. We show comparisons of the keypoint transfer accuracy (PCK) in Tables 1. On both LASSIE and E-LASSIE image sets, Hi-LASSIE+ produces marginal PCK gain from Hi-LASSIE [38] by naively applying the SDS loss. ARTIC3D, on the other hand, achieves consistently higher PCK than the baselines, especially on the noisy E-LASSIE images. The results demonstrate that our diffusion-guidel strategies can effectively learn more detailed, accurate, and robust 3D shapes. The Pascal-Part results in Tab 2 further show that ARTIC3D performs favorably against the state-of-the-art optimization-basse proaches. In Table 3, we show the CLIP similarity comparisons on the E-LASSIE images, which indicate that our textured outputs are more faithful to both the input images (instance-level) and text prompt (class-level) for most animal classes.
methods and are comparable to learning-basee ap-
Animation and texture transfer. Fig. 6, we compare the animations before and after our fine- tuning step via T-DASS. While the skeleton-based representation allows easy animation via rigid part transformation, the output part shapes and texture are often disconnected and irregular around the joints. The results show that T-DASS effectively produce high-qualis animations that are detailed in shape and texture and temporally consistent between frames. addition to animation, our 3D part surfaces also enables convenient controllable syntheses like texture transfer and pose transfer between different instance or animal classes. Several examples of texture transfer are shown in Fig. 7. More visual results with video results of these applications are shown in the supplemental material.
Limitations. ARTIC3D relies on the 3D skeleton discovered by Hi-LASSIE [38] to initialize the parts. If the animal bodies are occluded or truncated in most images, the skeleton initialization tends be inaccurate, and thus limiting ARTIC3D's ability form realistic parts. Although our input preprocessing method can mitigate this issue to some extent, fluffy animals (e.g.sheep) with

---

Table 1: Keypoint transfer evaluations the LASSIE [39] and E-LASSIE image sets. We report the average PCK@0.0 (1) all pairs of images. ARTIC3D performs favorably against the optimization-ban
Table 2: Keypoint transfer results on Pascal-Part [6]. We report the mean PCK @0.1 (f) on all pairs of
prior arts on all animal classes. The larger performance gap in the E-LASSIE demonstrates that ARTIC3D is robust to noisy images.
images. indicates learning- srrning-baaed models which trained large- scale image set.
Mathod
Image set Elephant Gimfffe Kangaroo Penguin
Tiger
Zehra
Method
Hoese
Cow
Sheep
LASSIE [39]
LASSIE
40.3
60.5
31.5
40.6
62.4
63.3
UMR [14]
24.4
HI-LASSIE (38)
LASSIE
42.7
61.6
35.0
44.4
63.1
64.2
A-CSM [11]
32.9
26.3
28.6
MagiclPnyy
ARTIC3D
LASSIE
44.1
61.9
36.7
45.3
64.0
64.8
Farm3D" [9]
42.5
40.2
32.8
HI-LASSIE [38]
E-LASSIE
37.6
54.3
31.9
41.7
57.4
60.1
LASSIE [39]
42.2
37.5
27.5
HI-LASSIE+
E-LASSIE
38.3
54.8
32.8
41.8
57.7
61.3
HI-LASSIE (38]
43.7
42.1
29.9
ARTIC3D
ARTIC3D
44.4
43.0
31.9
Table 3: CLIP similarity (1) evaluations on the E-LASSIE images. For each animal class, we calculate cosine similarities s1/s2, where s1 is the image-iman image similarity (against masked input image) and s2 is the image-text similarity (against text prompt).
Method
Elephant
Giraffe
Kangaroo
Ponguin
Tiger
Zebra
Hi-LASSIE [38]
80.0 26.3
85.2 29.66
77.4
85.8 / 30.8
79.7 25.6
83.8 27.4
Hi-LASSIE+
79.0/ 27.7
84.7 300.2
78.3 29.1
82.9 32.3 /2.3
/5.3/255.3
81.9 27.6
ARTIC3D
82.6/ 28.4
85.3/
81.6 29.9
85.5 33.1
80.0 27.8
84.1 29.4
Rigil transfem
Fine tunel
Target
Thansfemod
Figure 6: Animation fine-tuning, Compared
Figure Texture transfer. Our part surface repre- sentation enables applications like pose or texture transfer. Given a source shape and target texture, we show the transferred texture between instances (left)
to the original animated outputs via rigid trans-
formation (top), our animation fine-tuning (bot- tom) effectively improves the shape and tex-
ture details, especially around animal joints.
and animal species (right).
ambiguous skeletal configuration can still pose challenges in skeleton discovery. In addition, the front-facing bias diffusion models sometimes lead unrealistic texture like multiple faces, which also affects our reconstruction quality. See the supplemental material for failure cases.
5 Conclusion
We propose ARTIC3D, a diffusion guided framework to reconstruct 3D articulated shapes and texture from sparse and noisy web images. Specifically, we design novel DASS module to efficiently calculate pixel gradients from score distillation for 3D surface optimization. We use the DASS module and its variants the input preprocessing of noisy images; shape and texture optimization; as well as the animation fine-tuning stages. In addition to the existing Pascal-Part and LASSIE datasets, also evaluate on the self-collected E-LASSIE images with occlusions and truncation. Results these datasets demonstrate that ARTIC3D produces more robust, detailed, and realistic reconstructions against prior arts. As our future work, we hope to extend ARTIC3D more general articulated objects.
10

---

References
[1]
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep ViT features as dense visual descriptors. arXiv preprint arXiv:2112.05814 2021. 6
[2]
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. NeRD: Neural reflectance decomposition from image collections. In CVPR, pages 12684-12694, 2021. 3
[3] Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan Barron, Hendrik Lensch, and Varun Jampani. SAMURAL: Shape and material from unconstrained real-world arbitrary image collections. NeurlPS, 2022. 3
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- superviedd vision transformers. In ICCV, pages 9650-9660) 2021. 3, 6 [5] Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, pages
1971-1978, 2014. 2, 8
[6] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The PASCAL visual object classes ( (VOC) challenge. IJV, 88(2)300--338, 2010. 10
[7] Shubham Goel, Angjoo Kanazawa, and Jitendra Malik. Shape and viewpoint without keypoints. In ECCV,
pages 88-104, 2020.
[8] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12558 2022.
[9] Tomas Jakab, Ruining Li, Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Farm3d: Learning articulated 3d animals by distilling 2d diffusion. arXiv preprint arXiv:2304. 10535, 2023. 6, 8, 10
[10] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv: /4/2.69801 2014. 7
[11] Nilesh Kulkarni, Abhinav Gupta, David F Fouhey, and Shubham Tulsiani. Articulation-awarl canonical
surface mapping. In CVPR, pages 452-461, 2020. 3, 8, 10
[12] Nilesh Kulkamni, Abhinav Gupta, and Shubham Tulsiani. Canonical surface mapping via geometric cycle consistency. In ICCV, pages 2202-2211, 2019. 3,8
Ming-Hsuan [13] Xueting Li. Sifei Liu, Shalini De Mello, Kihwan Kim, Xiaolong Wang. Ming Hsuan Yang, and Jan Kautz.
Online adaptation for consistent mesh reconstruction in the wild. NeurIPS, 33:1 15099-15019, 2020.
[14] Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang. and Jan Kautz. Self supervised single-viev view 3D reconstruction via semantic consistency. In ECCV, pages 677-693, 2020.
8, 10
[15] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211. 10440, 2022. 1, 3, 4, 7,
[16] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based
3D reasoning. CVPR, pages 7708-7717, 2019. 6,7
[17] Luke Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Realfusion: 360 {(deg)
reconstruction of any object from a single image. arXiv preprint arXiv:2302. 106663 2023. 3
[18] Gal Metzer, Elad Richardson, Or Patashnik. Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape- guided generation of 3d shapes and textures. arXiv preprint arXiv:2211.07660 2022. 4,
[19] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, pages 405-421, 2020. 3 [20] Tom Monnier, Matthew Fisher, Alcxci A Efros, and Mathicu Aubry. Share with thy neighbors: Single-view reconstruction by cross-instance consistency. In ECCV, pages 285-303, 2022. 7
[21] Ben Poole. Ajay Jain, Jonathan Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion.
arXiv preprint arXiv:2209.
2022. 1, 2, 3, 4. 7,8
[22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, pages 8748- 9748-8763, 2021. 8
[23] Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, et al. Dreambooth3d: Subject-driven text-to-3d generation. arXiv preprint 2330v 13508, 2023. 3, 8
[24] Amit Raj. Michael Zollhofer, Tomas Simon. Jason Saragih, Shunsuke Saito, James Hays, and Stephen Lombardi. Pixel-aligned volumetric avatars. CVPR, pages 11733-11742, 2021. 3
[25] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh. Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ICML, pages 8821-8831, 2021. 1
[26] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing of 3d shapes. arXiv preprint XXXi:23302.0172/1 2023. 3
[27] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolttion image synthesis with latent difffusion models. In CVPR, pages 10684 10695, 2022. 1, 3, 4, 6,8
11

---

[28] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily L Denton. Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. NeurIPS, 35:36479-36494, 2022. 1.
[29] Uriel Singer, Shelly Sheynin, Adam Polyak. Oron Ashual, lurii Makarov. Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. arXiv preprint Xiv:2300.. 1280, 2023. 3
[30] Prune Truong. Maric-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. Spart: Neural radiance fields from sparse and noisy poses. arXiv preprint arXiv:2211.1173, 2022.
[31] Shubham Tulsiani, Nilesh Kulkarni, and Abhinav Gupta. Implicit mesh reconstruction from unannotated
image collections. arXiv preprint arXiv:2007 000000 2020. 8
[32] Qianqian Wang. Zhicheng Wang. Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron,
Ricardo Martin-Brualla Noah Snavely, and Thomas Funkhouser. Ibmet: Learning multi-view image-based
rendering. In CVPR, pages 4690 16990 2021.
[33] Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Magicpony: Learning articulated 3d animals in the wild. arXiv preprint (tfi::2211.1499/ 2022. 1, 3, 10
[34] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang. Deva Ramanan, William T Freeman, and Ce Liu. LASR: Learning articulated shape reconstruction from a
monocular vidco. In CVPR, pages 15980-15989 2021. 3,8 8
[35] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva Ramanan.
VISER: Video-specific surface embeddings for articulated 3d shape reconstruction. NeurIPS, 34, 2021. [36] Gengshan Yang. Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. BANMo: Building animatable 3D neural models from many casual videos. arXiv preprint arXiv:21 127661, 2021.
3,
[37] Gengshan Yang, Chaoyang Wang, N Dinesh Reddy, and Deva Ramanan. Reconstructing animatable categories from videos. arXiv preprint arXiv:2305. 06351, 2023. 3
[38] Chun-Han Yao, Wci-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming-Hsuan Yang, and Varun Jampani. Hi-lassie: High-fidelity articulated shape and skeleton discovery from sparse image ensemble. arXiv
preprint arXiv:2212. /042, 2022. 1, 2, 3, 4, 6,
10
[391 Chun-Han Yao, Wci-Chih Hung, Yuanzhen Li, Michael Rubinstoin, Ming-Hsuan Yang, and Varun Jampani. Lassie: Learning articulated shapes from sparse image ensemble via 3d part discovery. arXiv preprint arXiv:2207.00 0334444,4 2022. 1, 2, 3, 10
[40] Yufci Ye, Shubham Tulsiani, and Abhinav Gupta. Shelf-supervieed mesh prediction in the wild. In CVPR,
pages 8843 8852, 2021. 8
[41] Alex Yu, Vickie Yc, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In CVPR, pages 4578-4587, 2021.
[42] Jason Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan. NeRS: Neural reflcctance surfaces for sparse-view 3D reconstruction in the wild. NeurlPS, 34, 2021. 3,
[43] Kai Zhang, Fujun Luan, Qianqian Wang. Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical Gaussians for thysics-basee material editing and relighting. In CVPR, pages 5453-5462, 2021.
[44] Lvmin Zhang and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv
preprint arX2: 05543, 2023. 3
[45] Silvia Zuffi, Angjoo Kanazawa, Tanya Berger- Wolf, and Michael J Black. ThreeD Safari: Learning to estimate zebra posc, shape, and texture from images "in the wild". In ICCV, pages 5359-5368, 2019. 3, 8
[46] Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael Black, 3D menagerie: Modeling the 3D shape and pose animals, In CVPR, pages 1336566333 2017. 3
12


Reviews:
[{"review_id": "m5NqFyWmbq", "reviewer": "Reviewer_MsPL", "summary": "This paper introduces an articulated 3D shape reconstruction method from noisy web images with the help of diffusion models. The authors use a diffusion method to enhance the noisy input images to get clean reference 2D images and masks. Then, skeleton-based surface representations are optimized from the reference images. Then, fine-tuning improves the animations of the reconstructed shapes.", "questions": "1. How does the diffusion-guided optimization of shape and texture perform? It would be nice to see the ablation study on this part to distinguish this method from LASSIE[39] and Hi-LASSIE[38].\n2. How do you deal with the extra noise or overcorrection from the diffusion model?  The preprocessed image may look like an average animal in the species from the diffusion models.", "limitations": "The authors mentioned some limitations but not enough for me. Like the possible noise introduced from Stable Diffusion as mentioned above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The overall narrative of the paper is sound and readable.\n2. The authors propose to produce clean input reference images from noisy web images using diffusion models as a preprocessing step.\n3. The authors propose Decoder-based Accumulative Score Sampling  (DASS) to improve efficiency and reduce artifacts.\n4. The authors designed a fine-tuning step to allow better animation of the reconstructed objects.", "weaknesses": "1. The reconstruction part heavily relies on previous works like LASSIE[39] and Hi-LASSIE[38], it seems that the authors did not contribute very much to the core algorithm in this reconstruction task. Diffusion models helped improve the results, but most of the contribution is in the data cleaning and preprocessing.\n2. Diffusion models help in image preprocessing, but Stable Diffusion would also produce results from its knowledge based on the text prompt. Therefore we can observe that some textures of the reconstructed objects are different from the reference input images, even for the unoccluded parts.\n3. Figure 2 is not clear enough to show the whole workflow of the proposed methods, it is hard to relate the DASS module with the shape and texture optimization.\n4. Lacking ablation studies, it is unclear whether the Distilling 3D reconstruction is working in section 3.4, without the Distilling 3D reconstruction part, the reconstruction technique would be mostly relied on LASSIE[39] and Hi-LASSIE[38] as I mentioned above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "---\n**Contribution beyond LASSIE / Hi-LASSIE**\n\nPlease see “Contribution beyond LASSIE / Hi-LASSIE” in the General Response above.\n\n\n\n---\n**Unfaithful texture from Stable  Diffusion**\n\nPlease see “Unfaithful texture from Stable Diffusion” in the General Response above.\n\n\n---\n**“Figure 2 is not clear enough”**\n\nWe thank the reviewer for the feedback and will update Figure 2 in the manuscript. We also provide a more detailed overview figure through this [anonymous link](https://www.dropbox.com/scl/fi/7abzk19c7cunubatnsp84/illustration.png?rlkey=hnlzefqxwvxf75hboktjhelw1&dl=0) and will add it to the supplemental document.\n\n\n\n---\n**Ablation studies on distilling 3D reconstruction**\n\nOur ablation study of individual components is shown in Table 1 in the supplemental document, and we further report the detailed PCK gain (compared to Hi-LASSIE) in Table-T3 below. Note that the proposed DASS loss for 3D reconstruction brings 1.0-2.1% PCK gain from Hi-LASSIE without input preprocessing.\n\n**Table-T3**: PCK@0.05 on the E-LASSIE image sets (higher the better).\n| Method            | Input enhance. | $L_{dass}$ | Elephant    | Giraffe        | Kangaroo   | Penguin    | Tiger          | Zebra        |\n| :-----------        |  :-----------:      |  :-----------: | :-----------: | :-------:       | :-----------: | :---------:  | :-----:          | :-------:      |\n| Hi-LASSIE       |                         |                     |  37.6           | 54.3           | 31.9           | 41.7           | 57.4           | 60.1           |\n| ARTIC3D         |                         |  v                 |  38.8 (+1.2) | 56.1 (+1.8) | 34.0 (+2.1) | 42.7 (+1.0) | 58.5 (+1.1) | 61.9 (+1.8) |\n| ARTIC3D         |  v                     |                     |  39.0 (+1.4) | 57.3 (+3.0) | 34.6 (+2.7) | 43.4 (+1.7) | 58.5 (+1.1) | 62.4 (+2.3) | \n| ARTIC3D (full)  |  v                     |  v                 |  39.8 (+2.2) | 58.0 (+3.7) | 35.3 (+3.4) | 43.8 (+2.1) | 59.3 (+1.9) | 63.0 (+2.9) |\n"}, {"review_id": "jnGQ0BgsTV", "reviewer": "Reviewer_V4Hg", "summary": "This paper proposes a new framework, named ARTIC3D, to address the task of 3D reconstruction of articulated shapes and texture from noisy and few images. It is based on pre-trained diffusion models. Specifically, the authors use a novel decoder-based accumulative score sampling (DASS) to replace score distillation sampling (SDS). This has been used in many other frameworks to calculate pixel gradients in order to use diffusion priors more efficiently. Besides, they also extend the LASSIE dataset with more annotated data, called E-LASSIE, which could be useful for future works , especially for robustness of 3D reconstruction models.", "questions": "n/a", "limitations": "Limitations have already been elaborated clearly in their paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Good writing.\n\n(2) Plentiful experiments to prove the effectiveness of their framework and necessity of each module.\n\n(3) Novelty of the method (DASS) to better implement 2D diffusion priors in 3D reconstruction tasks, especially when big and clean datasets are not available. Detailed techniques are elaborated in the method section, including how to use them in preprocessing noisy input image, shape and texture optimization, and animation fine-tuning.\n\n(4) Extension of the LASSIE dataset with more annotated images to a new dataset, E-LASSIE, which could be useful for future research in this area, especially for evaluating the 3D reconstruction model robustness.", "weaknesses": "(1) This framework is highly based on LASSIE, both the method and the necessary input. Specifically, this framework needs the 3D skeleton from a pre-trained LASSIE model. And the framework also shares many parts with LASSIE's. But as mentioned in the Strengths part, the authors propose a new method to better incorporate diffusion models in 3D reconstruction and they propose a new dataset.\n\n(2) Though the title claims \"learning from noisy web image collections\", the noise actually only involves truncation and occlusions. There are also many other types of noise that have not been explored, including illumination variations, too small instances, multiple instances, etc., which are common in web images. They also uses DINO-VIT to do the foreground-background segmentation, which is acceptable but tricky, since the noisy background is also an important and common difficulty when dealing with web images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "---\n**Contribution beyond LASSIE / Hi-LASSIE**\n\nPlease see “Contribution beyond LASSIE / Hi-LASSIE” in the General Response above.\n\n\n---\n**”There are many other types of noises that are not explored”**\n\nWe thank the reviewer for the feedback and will revise the wording in the manuscript. Please note that ARTIC3D can deal with general noise beyond occlusion/truncation to some extent as long as the DINO-ViT feature clusters are robust. For instance, several images in the E-LASSIE dataset are either greyscale, poorly illuminated, or contain small instances and noisy backgrounds, as shown in the examples through this [anonymous link](https://www.dropbox.com/scl/fi/ykuubnkgn1j5emo6d4tl9/noisy_images.png?rlkey=mfzegdj9az9t5q0wl34ql27xv&dl=0). Although not within the scope of this paper, automatic filtering of web images is an important next step towards real-world application."}, {"review_id": "gwKNYJ3pSN", "reviewer": "Reviewer_GaMw", "summary": "This paper proposes a method to reconstruct the shape and texture of articulated objects from noisy web image collections. To achieve this, ARTIC3D proposed a diffusion-based 2D image enhancement module DASS, and then reconstructed the shape and texture maps using Hi-LASSIE. Moreover, to increase the animation results, ARTIC3D introduced a T-DASS module for animation fine-tuning. Experiments on the E-LASSIE dataset show that this method can produce high-fidelity animation results from noisy inputs.", "questions": "--ARTIC3D optimized the textured images per instance(L220-211). However, the appearance of a single animal may also vary with the lighting conditions. How does this method deal with this problem?\n--ARTIC3D ustilizes the T-DASS module to enhance animation. It would be great if the paper can also include some discussions on rendering speed and other relevant computational costs.\n--The T-DASS module may lead to 3D inconsistency. How does this method solve this problem?", "limitations": "--The shapes and textures generated by this method is inaccurate and blurry, and are not faithful to the input images in some cases.\n--The DASS module may lead to 3D inconsistency.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "--ARTIC3D can directly reconstruct the shapes and texture maps of articulated objects from noisy web image collections, greetly increase the robotness of Hi-LASSIE.\n--The proposed DASS and T-DASS modules are novel, intuitive and effective.\n--The paper is well-writen and easy to follow.", "weaknesses": "--The texture map obtained by ARTIC3D is not good enough. To solve this problem, ARTIC3D relies on the diffusion-based DASS module and the animation fine-tuning module to achieve high-fidelity animation and novel view rendering results. However, this may lead to 3D inconsistency. How does this method solve this problem?\n--Although enhanced by T-DASS, the animation results are still blurry and the texture moves over time. Also, the T-DASS module seems to make the results blurrier than directly using DASS module.\n--The reconstructed texture and shape are not faithful to the input image in some cases. The DASS module change the shape and appearance of the input images. Moreover, the images generated by this module may lose their 3D consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "---\n**3D inconsistency and unfaithful texture from Stable Diffusion**\n\nMost diffusion-based 3D generation methods like DreamFusion [21] only rely on the 2D diffusion prior to produce 3D shapes and texture, and thus they are prone to inconsistent outputs from different views (e.g. multiple faces on one animal body). ARTIC3D, on the other hand, combines the 2D diffusion prior, semantic correspondence, and 3D geometric priors (silhouettes from multiple views/poses, 3D part surface and pose regularizations, etc), which largely mitigates the 3D inconsistency issue. For more discussion on texture reconstruction, please see “Unfaithful texture from Stable Diffusion” in the General Response above.\n\n\n---\n**Blurry animation results with T-DASS**\n\nAs discussed in supplemental Sec. 2.3, rigid transformation produces static texture during motion and undesirable gaps around joints, and per-frame DASS outputs contain sharp details but are flickering and temporally inconsistent. We propose T-DASS as an alternative to find a better tradeoff between high-fidelity details and temporal smoothness. Although the tradeoff might be suboptimal, we argue that T-DASS makes a good contribution considering that the animation is created from a **single image without any annotations** and that no prior method can achieve this.\n\n\n---\n**Lighting conditions**\n\nConsidering that our problem setting is highly ill-posed (sparse, unannotated, and uncorrelated images), we do not explicitly model lighting and thus the texture image incorporates both albedo and lighting information of the input image. Note that our diffusion-based DASS module can recover natural texture from poorly illuminated or shadowed images.\n\n\n---\n**Rendering speed and computational costs**\n\nTo render 30 frames at 512x512 resolution with rigid part transformation, it takes roughly 2-3 minutes on a single GTX 1080 GPU.\t Fine-tuning the animation frames with the DASS module for 300 iterations increases the total rendering time to 8 minutes. With the T-DASS module, the total fine-tuning + rendering time is roughly 10 minutes. Since the 2D flow fields and temporal consistency loss are computed on the low-resolution (64x64) feature maps, the T-DASS module marginally increases the rendering time by 2 minutes per 30 frames to enforce temporal consistency."}, {"review_id": "27EIune8rL", "reviewer": "Reviewer_atAr", "summary": "ARTIC3D is a self-supervised framework that reconstructs 3D articulated shapes and textures of animals from sparse and noisy online images. It uses a skeleton-based surface representation and 2D diffusion priors to enhance the input images and guide the 3D optimization. It also enables realistic animation by fine-tuning the rendered shape and texture under rigid part transformations. ARTIC3D outperforms prior methods in terms of shape and texture fidelity, robustness to occlusions and truncation, and pose transferability. The authors also introduce E-LASSIE, an extended dataset with noisy web images, to evaluate model robustness.", "questions": "* The usage of SD improves texture quality on a perceptual level. However, as seen in many figures (ex 3, 4 in supp, 3 main) it mainly hallucinates the texture to appear realistic however further apart from the image used as reference. For example, the reconstructed elephant looks nothing like the image, similarly the kangaroo, tiger, zebra. The work is presented as reconstruction work and as such diverging from the images can't be considered a proper reconstruction. Have you examined a way to mitigate this texture drift? \n\n* The user studies on animation are flawed. A 55% preference from 100 users means that ARTIC3D'a animations are only slightly better than rigid transform which is close enough to random choice. Could you explain in detail what sample size was used for the user study? Whether the studies were cherry picked prior to the user study? I find the user study explanation to lack a lot of key details.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* ARTIC3D is a self-supervised framework that can reconstruct 3D articulated shapes and textures of animals from sparse and noisy online images, without relying on any pre-defined shape templates or per-image annotations. This makes it scalable and adaptable to different animal species and poses.\n* The method leverages 2D diffusion to enhance the input images by removing occlusions and truncation, and to extract semantic features and 3D skeleton initialization. This improves the quality and robustness of the 3D outputs, as well as the efficiency and stability of the optimization process.\n* Usage of diffusion-guided 3D optimization to estimate shape and texture that are faithful to the input images and consistent across different viewpoints and poses. It also introduces a novel technique to calculate more stable image-level gradients via diffusion models, which enhances the convergence and robustness of the optimization.\n* ARTIC3D produces realistic animations by fine-tuning the rendered shape and texture under rigid part transformations, which preserves the articulation and details of the 3D shapes. It also enables explicit pose transfer and animation, which are not feasible for prior diffusion-guided methods with neural volumetric representations.", "weaknesses": "* The manuscript is not easy to follow for anybody who is unfamiliar with LASSIE and HI-LASSIE. This work is a step forward from Hi-LASSIE using Stable Diffusion to avoid several pitfalls with respect to optimization and image pre-processing.\n\n* ARTIC3D depends on the 3D skeleton initialization from Hi-LASSIE [38], which may be inaccurate for occluded or truncated animal bodies, resulting in unrealistic part shapes. Also, it struggles with fluffy animals with ambiguous skeletal configuration, such as sheep, which pose challenges in skeleton discovery and shape reconstruction.\n\n* The reconstruction results are not significantly better than Hi-LASSIE in most case 1% better, however it is not clear from the manuscript whether this is statistically significant or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "---\n**”More details of LASSIE and Hi-LASSIE are needed”**\n\nWe thank the reviewer for the feedback and will add more details to the preliminaries section (3.1) in the manuscript. \n\n\n---\n**”3D skeleton from Hi-LASSIE may not work for occluded/truncated images”**\n\nOur 3D skeleton initialization is performed on the reference image after input preprocessing with DASS, which can effectively complete the occluded or truncated animal bodies. That is, our skeleton is more robust and accurate compared to Hi-LASSIE in the partial-body cases. Fluffy animals are a common challenge in our ill-posed problem setting. However, we are able to produce reasonable reconstruction of sheep as shown in supplemental Figure 8, which is a considerable improvement from prior works.\n\n\n---\n**Reconstruction performance gain**\n\nAs shown in Table 1 of the main paper, ARTIC3D achieves 0.6-1.7% PCK gain on the clean LASSIE images and 1.9-3.7% PCK gain on the noisy E-LASSIE images compared to Hi-LASSIE. It demonstrates a consistent advantage on ARTIC3D over Hi-LASSIE on all animal classes, especially in the occlusion/truncation cases. Moreover, the CLIP similarity evaluation in Table 3 also shows consistently favorable textured reconstructions by ARTIC3D. For more detailed discussion, please see “Contribution beyond LASSIE / Hi-LASSIE” in the General Response above.\n\n\n---\n**Unfaithful texture from Stable Diffusion**\n\nPlease see “Unfaithful texture from Stable Diffusion” in the General Response above.\n\n\n---\n**Details of user study**\n\nIn our user study, we randomly select 3 examples per animal class from the E-LASSIE and Pascal-Part datasets. More details of the user study will be added to the manuscript. As shown in the video and discussed in supplemental Sec. 2.3, rigid transformation produces static texture during motion and undesirable gaps around joints, and per-frame DASS outputs contain sharp details but are flickering and temporally inconsistent. We propose T-DASS as an alternative to find a better tradeoff between high-fidelity details and temporal smoothness. Despite the small gap, the user study still shows an overall preference of T-DASS considering both realism and temporal consistency. Please also note that all three methods (rigid transformation,  DASS, and T-DASS) are our contributions, which enable animation from a **single image without any annotations** and that no prior method can achieve this.\n"}]

Author rebuttal:
# General Response

We thank the reviewers for the constructive feedback. We address the common concerns in the General Response and specific comments in the individual response to each reviewer.

---
### **Contribution beyond LASSIE / Hi-LASSIE**

While ARTIC3D deals with the same reconstruction task and leverages the skeleton/shape representation as in LASSIE and Hi-LASSIE, there are several key differences that allow ARTIC3D to handle occluded/truncated images and produce detailed texture in novel views. At a high level, LASSIE and Hi-LASSIE focus on **using geometry priors to learn detailed articulated shapes**, whereas ARTIC3D proposes to **incorporate generative 2D diffusion priors in a more challenging scenario with noisy images**. We emphasize that **combining such 3D geometry and 2D diffusion priors** is challenging, and that the proposed DASS module can effectively improve the results in all 3 stages (input preprocessing, 3D reconstruction, and animation). 

As shown in Table 1 in the manuscript and Table-T1 below, Hi-LASSIE+ (Hi-LASSIE with SDS loss) marginally improves the PCK on E-LASSIE images by 0.1-1.2% by naively applying the common SDS loss, while ARTIC3D achieves a 1.9-3.7% PCK gain. Our CLIP similarity evaluation in Table 3 also shows consistently favorable textured reconstructions by ARTIC3D in different views. Moreover, our ablation study (supplemental Table 1 and Table-T2 below) demonstrates that the DASS loss for 3D reconstruction leads to 1.0-2.1% PCK gain from Hi-LASSIE without input preprocessing. We believe these PCK gains from ARTIC3D compared to prior works are considerable.

**Table-T1**: PCK@0.05 on the E-LASSIE image sets (higher the better).
| Method            | Elephant | Giraffe | Kangaroo | Penguin | Tiger | Zebra |
| :-----------          | :----------: | :-------: | :------------: | :---------: | :-----: | :------: |
| Hi-LASSIE       | 37.6           | 54.3            | 31.9            | 41.7           | 57.4            | 60.1           |
| Hi-LASSIE+     | 38.3 (+0.7) | 54.8 (+0.5) | 32.8 (+0.9) | 41.8 (+0.1) | 57.7 (+0.3) | 61.3 (+1.2) |
| ARTIC3D         | 39.8 (+2.2) | 58.0 (+3.7) | 35.3 (+3.4) | 43.8 (+2.1) | 59.3 (+1.9) | 63.0 (+2.9) |


**Table-T2**: PCK@0.05 on the E-LASSIE image sets (higher the better).
| Method            | Input enhance. | $L_{dass}$ | Elephant | Giraffe | Kangaroo | Penguin | Tiger | Zebra |
| :-----------          |  :-----------:        |  :-----------:   | :-----------: | :-------: | :-----------: | :---------: | :-----: | :-------: |
| Hi-LASSIE       |                         |                    |  37.6           | 54.3            | 31.9            | 41.7           | 57.4            | 60.1           |
| ARTIC3D         |                         |  v                |  38.8 (+1.2) | 56.1 (+1.8) | 34.0 (+2.1) | 42.7 (+1.0) | 58.5 (+1.1) | 61.9 (+1.8) |
| ARTIC3D (full) |  v                     |  v                |  39.8 (+2.2) | 58.0 (+3.7) | 35.3 (+3.4) | 43.8 (+2.1) | 59.3 (+1.9) | 63.0 (+2.9) |


---
### **Unfaithful texture from Stable  Diffusion**

Due to the highly ill-posed nature of our problem setting, there exists a tradeoff between realism and faithfulness to input images, especially for the unseen/occluded surface. For instance, whether a colored or black-and-white output is preferred given a greyscale/poorly illuminated image, or whether we preserve the noisy texture (small occlusions like dirt, water splash, or shadow caused by rough surface / objects) in the original image. Since the dense surface visibility is hard to obtain, we optimize the shape and texture that are slightly biased towards realism (detailed and clean texture that resembles the animal class) in this paper. 

In our ablative analysis of the DASS module (Figure 3), we show that **we can control the realism-faithfulness tradeoff** by tuning the diffusion timestep $t$ and guidance weight $w_g$. Specifically, larger $t$ and $w_g$ allows DASS to hallucinate shape and texture that are not present in the original image. In addition, one can enforce a more faithful reconstruction by setting a higher weight of the texture reconstruction loss $\alpha_{text}$ (L223). We also show additional visual results on the tradeoff through this [anonymous link](https://www.dropbox.com/scl/fi/hshbqmf2gemq0ou32qa7f/faithfulness.png?rlkey=m6qt98hk5kuvuqoeoz96l2o3q&dl=0). 

Although we acknowledge that the current tradeoff is not optimal and some example outputs are not faithful to the input images, our evaluation of image-to-image CLIP similarity in Table 3 shows that ARTIC3D outputs are still generally more faithful to input images across different views compared to the existing methods. We believe that ARTIC3D is a good first step in this novel scope, and automatically finding the best tradeoff for each animal class/instance forms an interesting future work.


Meta-review: All the reviewers recognize the contributions and interest of the paper. The AC agrees and is happy to recommend acceptance.

The ablation results from the rebuttal seem essential to assess the contribution, it should be included in the final version of the paper.
Decision: Accept (poster)