YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals
Authors: Sandeep Mishra
Oindrila Saha
Alan Bovik
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Animal generation
Text-to-3D
Diffusion models
Primary area: generative_models
TL;DR: YouDream generates anatomically and geometrically consistent 3D animals following a 3D pose prior, thus also enabling the generation of novel unseen animals.

Abstract: 3D generation guided by text-to-image diffusion models enables the creation of visually compelling assets. However previous methods explore generation based on image or text. The boundaries of creativity are limited by what can be expressed through words or the images that can be sourced. We present YouDream, a method to generate high-quality anatomically controllable animals. YouDream is guided using a text-to-image diffusion model controlled by 2D views of a 3D pose prior. Our method is capable of generating novel imaginary animals that previous text-to-3D generative methods are unable to create. Additionally, our method can preserve anatomic consistency in the generated animals, an area where prior approaches often struggle. Moreover, we design a fully automated pipeline for generating commonly observed animals. To circumvent the need for human intervention to create a 3D pose, we propose a multi-agent LLM that adapts poses from a limited library of animal 3D poses to represent the desired animal. A user study conducted on the outcomes of YouDream demonstrates the preference of the animal models generated by our method over others. Visualizations and code are available at https://youdream3d.github.io/.

YOUDREAM: Generating Anatomically Controllable Consistent Text-to-3D Animals
Sandeep Mishra Oindrila Saha and Alan C. Bovik!
UUniversity of Texas at Austin, UUniversity of Massachusetts Amherst sandy mi mishra@uteass edu, osaha@umass edu, bovikdece utexas. edu
Abstract
3D generation guided by text-to-image diffusion models enables the creation of visually compelling assets. However previous methods explore generation based image text. The boundaries of creativity are limited by what can be expressed through words or the images that can be sourced. We present YOUDREAM, a method to generate high-quality anatomically controllable animals. YOUDREAM is guided using a text-to-image diffusion model controlled by 2D views of a 3D pose prior. Our method is capable of generating novel imaginary animals that previous text-to-3D generative methods are unable to create. Additionally, our method preserve anatomic consistency in the generated animals, prior approaches often struggle. Moreover, we design fully automated pipeline for generating commonly observed animals. To circumvent the need for human intervention to create a 3D pose, we propose multi-agent LLM that adapts poses. from a limited library of animal 3D poses to represent the desired animal. user study conducted on the outcomes of YOUDREAM demonstrates the preference of the animal models generated by our method over others. Visualizations and code are available at https://youdream3...ihuub
where
1 Introduction
Text-to-3D generative modeling using diffusion models has seen
growth recently with
methods utilizing text-to-image (T21) Poole et (2022), x Chen et al. (2023a); Zhu al. (2023): Seo et al. (2023). (text*camera)-to -t--image (TC21) Shi et al. (2023 Li et al. (2023) and (imagetcamera)- to-image (IC21) Liu al. (2023); Wang and Shi (2033) : Ye et al. (2023) diffusion models. These methods are widely accepted by AI enthusiasts, content creators, and 3D artists create high- quality 3D content. However, generating 3D assets using such methods is dependent on what can be expressed through text or the availability of an image faithful to the user's imagination. In this work, we provide more control to the artist to bring their creative imagination to life. YOUDREAM generate high-quality 3D animals based on any 3D skeleton, by utilizing 2D pose-controlled diffusion model which generates images adhering to 2D views of a 3D pose. Using depth, edge, and scribble has also been explored for controllable image generation Zhang al. (2023). However, in 3D context, pose offers both 3D consistency as well as room for creativity. Other controls are restrictive edge/depth/boundann of 2D views of pre-existing object is used to provide control, thus limiting the generated shape to be very similar to the existing asset. We show that the multi-view consistency offered by our 3D pose prior results in the generation of anatomically and geometrically consistent animals. Creating 3D pose control also requires minimal human effort. To further alleviate this effort, also present a multi-agent LLM setup that generates 3D poses for novel animals commonly observed in nature.
Equual contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

HiFA
MVDream
YouDREaM (ours)
"A zoomed photo of llama with octopus tentacles body"
realistic mythical with two pairs of wings two long thin lion-like tails"
"A dragon with three heads separating from the neck"
"A giraffe with dragon wings"
six legged lioness, fierce beast, pouncing, ultra realistic, 4k"
"Golden ball with wings"
Figure I: Creating unreal creatures. Our method generates imaginary creatures based on an artist's creative control. We show that these creatures cannot be generated faithfully only based text. Each row depicts a 3D animal generated by HiFA, MVDream, and YOUDREAM (left to right) using the prompt mentioned below the row. We present 3D pose controls used to create these in the Sec. (results best viewed zoomed in).

---

3D generation guided by T2I models involves using the gradient computed by Score Distillation Sampling (SDS) al. (2022) to optimize 3D representation such as a NeRF NeRF Mildenhall et al.) (202T). During any intermediate step of the training, a rendered image captured by random camera is added to Gaussian noise and passed to a T2I diffusion model, along with a directional prompt. The diffusion model estimates the added noise, which in turn is used to create a denoised image. In effect, this process pushes the rendered image of the NeRF representation slightly closer to the denoised image during each iteration. Thus, any unwanted semantic or perceptual issues arising in the denoised image are also transferred to the NeRF This is especially problematic for deformable objects such as animals, where variations in pose views often results in the Janus-head problem, dehydrated assets, and geometric and anatomical inconsistencies.
TC2I diffusion models, which encode camera parameters and train using 3D objects from various views learn multi-view consistency and thus are able to produce better geometries. However, they lack in diversity owing to the limited variation in training data, as compared to text-to-image models. Along with this, methods using IC2I diffusion models also face the problems arising from Novel View Synthesis (NVS), which requires hallucination of unseen regions along with accurate geometric transformation of observed parts. While these camera guided diffusion models perform better than T2I models in many cases, their limited diversity and lack of control limit the creativity of their users. By utilizing a 3D pose prior, YOUDREAM consistently outperforms previous methods that use T21 diffusion models, in terms of generating biologically plausible animals. Despite not being trained on any 3D data, our method also outperforms the 3D-aware TC2I diffusion model MVDream (Shi et al. (2023) in text-to-3D animal generation in terms of "Naturalness", "Text-Image Alignment" and CLIP score (see Sec.
3D 3DD-consistency for human avatar creation has been explored extensively in recent works Cao et al. (2023): (Hunn et al. (2024); Kolotouros et al. (2024); Hong et al. (2022); Zhhag et al. (2024a) (022). These models rely a 3D human pose and shape prior, usually the SMPL Loper al. (2023) or SMPL-X Pavlakos Pavlaos et al. (2019) model. This strategy can represent a variety of geometrically consistent human avatars. However, representing the animal kingdom is challenging owing to its immense diversity which cannot be represented using any existing parametric models. Sizes and shapes vary considerably across birds, reptiles, mammals, and amphibians, hence until now, single shape or pose prior exists that can represent all tetrapods. Parametric models such as SMAL Zuffi let al. (017)) and MagicPony Wu et al. (0235) suffer from severe diversity issues, and hence cannot be used as pose or shape prior. Thus to circumvent human effort in generating a 3D pose prior for animals prevalent in nature, present method for automatic generation of diverse 3D poses using a multi-agent LLM supported by a small library of animal 3D poses. Additionally, we present a method automatically generate an initial shape based on a 3D pose, which is utilized for NeRF initialization.
In summary, YOUDREAM offers the following key contributions:
TetraPose ControlNet, trained tetrapod animals across various familics, that enables the generation of diverse animals at test time, both real and unreal.
a multi-agent LLM that can generate the 3D pose of any desired animal in a described state, supported by small library of 16 predefined animal 3D poses for reference.
user-friendly tool to create/modify 3D poses for unreal creatures. The same tool automati- cally generates an initial shape based on the 3D skeleton.
a pipeline to generate geometrically and anatomically consistent animals based on an input text by adhering to 3D pose prior.
2 Related Work
The field of 3D animal generation has rapidly advanced due to studies that offer methods and insights for modeling animal structures and movements in 3D. IMAL|Zuffi et al. (2017) introduced method to fit a parametric 3D shape model, derived from 3D scans, to animal images using 2D keypoints and segmentation masks, with extensions to multi-view images Zuffi et al. (2018). The variety of animals able to be represented by SMAL is severely limited. Subsequent efforts, such as LASSIE Yao et al. (2022) 2023 2024), have focused on deriving 3D shapes directly from smaller image collections by identifying self-supervised semantic correspondences to discover 3D parts. Succeeding work

---

represent animals using parametric model Dakab et al. (2023); ; Wu et al. (2023a b) Li et al. (2024)
learnt from images or videos. Despite these advances, these methods are class-specific and lack in the diversity of animals that can be represented. YOUDREAMiS able to generate a great variety of animals including those that have not been observed previously with higher details (Fig. [6).
High quality text-to-3D asset generation has been fueled by the availability of large-scale diverse datasets of text-image pairs and the success of text-to-image contrastive and generative models trained on them. Contrastive methods such as CLIP Radford et al. (202I) and ALIGN Jia et al.
(202T) learn a common embedding between visual and natural language domains. Generative methods like Imagen Saharia et al. (2022) and Stable Diffusion Rombach et al. (2022) utilize a diffusion model learn to generate images given text latents. These methods inherently learn to understand the appearance of entities across various views and poses. Text-to-3D generative modeling
methods Mohammad Khalid et al. (2022); Jain et al. (2022); Wang al, (2023 ; Poole et al. (2022) exploit this information by using these text-image models to guide the creation of 3D representations by NeRFs MMilenhlll et al. (202T). The quality of 3D assets produced by these early methods suffer from several issues such smooth geometries, saturated appearances, problem. Subsequent recent methods have ameliorated these problems by the use of modified loss functions Wang et al. (2024); Zhu et al. (2023), using Deep Marching Tetrahedra Shen et al.] (202T) for 3D representation Chen et al. (2023a), . and modified negative prompt weighing strategies Armandpour et al. (2023). However these methods still fail to produce anatomically correct animals, often producing implausible geometries or even extra or insufficient limbs. Prior work 3DFuse (Sco al. (2023) uses sparse point clouds predicted from images as depth control for T21 diffusion, however still produces anatomically inconsistent animals due to the inaccuracy of image-to-pone cloud predictors and high dependency
well geometric issues
such as the Janus (multi-hean
the initial generated image (Fig. 4 .. Recently, 3D-aware diffusion models trained on paired text-3D datasets by encoding camera parameters have been used generate 3D assets Tang et al. (023): (Shi et al. (2023). As these methods learn using various views of 3D objects, they rarely produce geometric inconsistency. However these methods are limited by the variety of 3D data available, which is quite scarce as compared to image data that T2I diffusion models have been trained on. These are trained using 3D object databases such as Objaverse Deitke et al.) (2023) and Objaverse-XL XL et al. (2024) which are considerably smaller than text-image paired datasets such as LAION-5B Schuhmann et al. (2022) used for training T2I diffusion models. Thus, they often struggle to follow the text input faithfully in case of complex prompts (Fig. D By comparison, our method accurately follows the text prompt owing to the use of T2I diffusion models trained on vast image data. YOUDREAM strictly adheres to input 3D pose priors, thus producing geometrically consistent and anatomically correct animals.
Large Language Models (LLMs) have been explored in the context of 3D generation and editing previously. LLMS have been used Yin et al. (023);; (iddiqui et al. (2023) to generate and edit shapes using an embedding space trained on datasets such as ShapeNet Chang et al. (2015). Prior work have also used LLMs to generate code for 3D modeling software, such as Blender, to create objects Yuan al. (2024) and scenes (Sun et al. (2023); Hu et al. (2024). These methods produce impressive results suggesting at LLMs' 3D understanding capability, but explore limited variety of generation often limited to shapes, or generate layouts/scenes. 3D pose generation with L.LMs using text as input has been recently explored for humans. ChatPose (Feng et al. (2024) and MotionGPT Zhhng et al. (2024b) generate pose parameters for SMPL model based on textual input. LLMs have also been previously shown to accurately reason about anatomical differences of animals Menon and Vondrick (2022);
Saha et al. (2024). In this work, we show a novel application of off-the-shelf LLMs for generalized 3D pose generation based on the name of an animal supported by a library of animal 3D poses.
User-controlled generation has been introduced in several studies Zhang et (2023): Mou et al. (2024). and has gained widespread adoption among artists for crafting remarkable illustrations, including artistic QR codes to interior designs. However, the use of user control in 3D is still under- explored. Recent works such as MVControl Li et al.) (2023) and Control3D Chen et al. (2023b) guide the 3D generation process using 2D condition image of a single view. By contrast, the generation process in YOUDREAM is guided using 2D views of a 3D pose which is dependent on the sampled camera pose. This strategy not only allows YOUDREAM to take in specialized user control but also ensures multi-view geometric consistency.

---

A
Pose Generation
NeRF Rendering
Tikerd
Becee weltove dialicc flrnce
y Lopel
Vinder
Clbsurvar
Modiliee
Viewpoint Sumpling
Cherk Hothrr
Fesch Prsee
Figure 2: Automatic pipeline for 3D animal generation. Given the name of an animal and textual pose description, we utilize mullii-ggent LLM to generate a 3D pose (ø) supported by a small library of animal names paired with 3D poses. With the obtained 3D pose, we train NeRF to generate the 3D animal guided by a diffusion model controlled by 2D views (opres) of Φb
3 Method
Multi-view sampling from T2I diffusion models for 3D generation is guided using directional prompt extensions such as "<user_ "<user_lext, front view" and "<user "<user_text>, side view". Such control signal is ambiguous due to, 1) directional text remaining unchanged over range of camera parameters, 2) T2I diffusion models generating deformable entities in various poses for the same view. Thus we utilize 3D pose as a stronger guidance to maintain consistency over different views. To do this in 3D consistent manner we design
model to generate 2D image samples following the projection geroj of 3D pose 6i of an animal, 2) method to generate the 3D pose & of novel animal y using a limited library of 3D poses (b) of commonly observed animals in nature and a multi-agent LLM pose editor, and 3) method to create 3D animals given an animal name y and 3D pose Φ. Our 3D model is represented using Neural Radiance Fields (NeRF Mildenhall et al. (202I)).
3.1 TetraPose ControlNet
To train a model to follow pose control we require images of animals with annotated poses. Datasets released by Banik et al. (2021) and Ng al. (2022) provide 2D pose annotations of animal images spanning a large number of species, compared to the limited diversity available in 3D animal pose datasets Xu al. 20223) ; [Badger al. (2020). We thus utilize these 2D pose datasets by learning to map the 2D pose of an animal to its captured image. We define such datasets of animal species y/+ )//=1 where (D) is the number of image-pose pairs in the dataset. This learned mapping can then be used to generate multi-view image samples consistent with a 3D pose o. The mapping is represented by ControlNet that produces animal images across mammals, amphibians, reptiles and birds following 2D input pose condition @proj learned by minimizing the following objective:
corresponding animal images Ij, and their 2D pose of as the set D {x,,, 0f"" yy .2))
LControlNet = oor N(..) [lle- cø (eet t, By ooros ||2
(I)
where 20 -... The above objective aims to learn network co estimate the noise added to input image =g (or xry) to form a noisy image 2, given time-steps 4, text yj and pose condition S The network ep is represented by the standard U-Net architecture of diffusion models (Stable Diffusion in this case) with a trainable copy of the U-Net's encoder attached to it using trainable zero convolution layers. We provide training details in Sec. F
The trained TetraPose ControlNet can be used generate pose-controlled images of tetrapods, including mammals, reptiles, birds, and amphibians. The model performs well for out-of-domaal 2D pose inputs of animals not seen during training. also performs well with inputs consisting of modified 2D poses that include extra appendages such as multiple heads, limbs, wings, and/or tails. While T2I diffusion models inherently provide a huge diversity to the generated outputs, the control module provides strong controlling signals to generate appropriate body parts in the right positions, alleviating the problem of T2I diffusion models producing inconsistent multi-view images when prompted using directional texts only.

---

Roseate
Crocodile
Hippo
Spoonbill
Flamingo
Giraffe
Horse
Figure 3: Qualitative examples of pose editing using multi-agent LLM setup. For each example, the green box denotes the desired animal, while the blue box is the animal retrieved from the 3D pose library by Finder LLM (TT). We show the pose modification performed by the joint effort of Observer (TO) and Modifier (TM) for three instances.
3.2 3D Pose Generation aided by Multi-agent LLM
Generating a 3D pose based on a text is not trivial as text-to-pose a many-to-many mapping. Existing 3D animal pose datasets are not diverse or vast enough to learn this mapping for variety of animals. Thus we leverage LLMs which are pre-trained on expansive textual datasets, and thus can reason about anatomical proportions of various animals. We find that LLMs do produce good 3D poses using only a text input, instead we use L.MSs to adapt a input 3D pose to represent a novel animal. We created limited library consisting of 16 animal 3D poses for this purpose.
Given a library animals = {y), ),)) consisting of 3D keypoint positions Φ; and animal names y: V. utilize multi-agent LLM setup for creating 3D pose for any desired animal y and pose description The agents include 1) Finder (Ty). 2) Observer (To), and 3) Modifier (TM)) Let the keypoint names representing any animal be K. and let the bone sequence which defines the skeleton S. Given K. the Finder selects the animal in B that is "anatomically closest" to the desired animal y as (Ye: o2) F (y, B, K.). "Anatomically closest" is defined the animal whose 3D pose will require minimal modificattions/upaaee to represent y. Given the keypoint definitions K., bone sequence S, the desired animal name y. the animal Ye selected by mr, and the pose description the Observer generates O = To (ye- V: U. K.). O represents a plan describing which keypoints of ye should be adjusted along with adjustments to represent the 3D pose of the desired animal in the described pose Based on the observations O, the Modifier updates the 3D positions of the keypoints, Φe. of the closest animal to Φ aa (&e: O). Thus we obtain the 3D keypoint positions of the desired animal y in the described pose p. We find that this multi-agent procedure more stable and accurate than using a single LLM for pose generation (see Sec. B We are able to represent diverse animals observed in nature using this setup. Fig. presents examples pose editing using our described setup. As ground truth text-to-3D poses for animals do not exist and the described task is many-to-many problem, quantitative evaluation difficult to obtain. Thus we conducted a user study to evaluate the efficacy of our method (details in Sec. 4). We describe the contents of our library B and the prompts to LLMSs in detail in Sec. and Sec.
instructions for the Modifier implement the suggested
3.3 Pose Editor and Shape Initializer
To facilitate easy creation and editing of 3D poses, we present a user-friendly tool to modify, add, or delete joints and bones. We also provide method in this tool to automatically generate initial shape based on the 3D skeleton using simple 3D geometries such as cylinders, cones, and ellipses. We this shape pre-train NeRF before fine-tuning using diffusion based guidance. Details of this tool are presented in Sec. F
3.4 Bringing Bones to Life
We want to create 3D animals given input text and 3D pose d. We adopt the Score Distillation Sampling (SDS) method proposed in DreamFusion Poole et al. (202)), adapted for our TetraPose ControlNet. The SDS loss gradient can now be represented as:
0z
V Co,s = E(g(n, c))) = Exc.,e w(t) (Ep(Ze: v(c),, gros (c)) -e)
(2)
On

---

"An elephent standing on concrete"
"A red male northern cardinal flying with wings spread out"
"A tiger
AA Tyrannoseurus rex
3DFuse
Fantasia3D
HiFA
DREAM (ours)
Figure 4: Comparison generating animals observed in nature. We compare with baselines which use T21 diffusion (with official open-source code) for the automatic generation of text-to-31) animals. Unlike the baselines, our method produces high quality anatomically consistent animals.
where y represents the trainable parameters of the NeRF, 0 the frozen diffusion model parameters, represents the sampled camera parameters, is the number of time-steps, and w(() timestep- dependent weighting function. z denotes the latent encoded using encoder & for the image rendered from the NeRF g e) for camera c. y(c) represents the directional text created based on camera c. while @ (c) is the 2D projection of the 3D pose & for camera Additionally, we also utilize an image domain loss weighted by the prper-parameter ARGB which reduces flickering and produces more solid geometry (Fig 14):
LRGB ARGB Er,c. (w(t) 1l c) ())
(3)
where 9(1), c) is the image rendered from the NeRF and (2) is the denoised image decoded using decoder 2 from the denoised latent 2.
Since our TetraPose ControlNet is trained on much smaller number of images compared to Stable Diffusion, it loses diversity. To improve diversity and generation capability, we propose use control scheduling and guidance scheduling. We observe that higher control scale provides strong signal for geometry modeling whereas higher guidance scale provides strong signal for appearance modeling, Since geometry is perfected in the initial stages and appearance in the latter, we propose reduction control scale and increase guidance scale over training iterations. This helps us create out-of-domain assets with significant style variety (see Fig. g Our strategy is formulated as:
control_scate = cos( π
train step,
train_step,
2 max_step (control control + controlmán.
train step
guidance_sst scale
(guidance, guidance min) + guidance mins
max_step
where train_step is the current training step and max_ster is total training iterations. The variables controlmaz- controlmin, guidance mer. guidance min are hyperparameters. We show that a linear scheme is better for guidance scheduling, while cosine is better for control scheduling in Sec. B

---

Experiments
In this section, we compare YOUDREAM against various baselines and evaluate the effect of various components of our method. We show qualitative comparison with text-to-3D methods which are guided by T21 diffusion models for common animals observed in nature. We also compare with MVDream which uses a (text + camera)-to-image diffusion model trained on 3D objects. It should be noted that our method does not use any 3D objects for training, yet is able to deliver geometrically consistent results. We conduct user study quantitatively evaluate our method against these baselines. We also compute CLIP score following previous work [Shi et al. (2023), shown in Sec. Additionally, we present ablations over the various modules that constitute YEUDREAM.
Generating animals observed nature. In Fig. we compare our method against 3DFuse Seo Chen et al. (023a), and HiFA Zhu et al. (2023) for generating common animals. s. HiFA and Fantasia suffer from anatomical inconsistency. while 3Dss more consistent in some cases due to the use of depth control. However, 3DFuse is highly dependent the point cloud prediction leading to the generation of implausible geometry, for example in the case of elephant and T-Rex. It should be noted that generating results using Fantasia3D required extensive parameter tuning, which has also been indicated by the authors in their repository. All results are generated using the same seed for fair comparison. We use the default hyperparameter settings of each baseline except Fantasia3D, The text full body" is appended at the end of the prompt for all basclines, as we observed that the methods generate truncated animals in many cases. We generate common animals using our fully automated pipeline, where we use LLM for pose editing sourced by a library of 3D poses. Tiger is generated by our multi-agent LLM based on German Shepherd, northern cardinal is made from an eagle, while elephant and Tyrannosaurus rex are part of our library. In all cases, our method visibly outperforms baselines in terms of perceptual quality and 3D consistency.
et al. (023), Fantasia3D Chenn
Generating unreal creatures. major advantage of pipeline is that it can be easily used generate non-existent creatures, especially those not explain- able through text. These can be generated robustly us- ing our method when the user provides a skeleton of their concept. We use our pose editor tool to generate the results shown in Fig. [ where YOUDREAM pro- duces stunning unreal creatures. We show the pose controls we use in Sec. [F] Notably, MVDream Shi al. (2023) struggled to follow the textual prompt as such creatures are not represented in existing 3D datasets; producing incorrect results such as "Wam- pus cat", cat-like creature in American folklore, with three legs instead of six in Fig. row 5. In some aspects HiFA attempted to follow the prompt (owing to its usage of T21 Stable Diffusion) such as produc- ing couple of tentacles in Fig. D row 1 and more than one head in row 3, but produces geometrically inconsistent results in all cases. Again use seed o, default hyperparameter settings for baselines and append ", full body" at the end of the prompts except for 'golden ball'.
Naturalness preference
HVDrram
SDFuse
VeuDream 60.5%
2.0% Fantasiado
Text-Image Alignment preference
AVDroom 266 3%%
J0Fuce
TO d%
YouBroatt
Pantasta30
62.8%
0.0%
Figure 5: User Study. User preferences on 1) Naturalness and 2) Text-Image alignment averaged over 32 participants and 22 text-to- 3D generated assets reveals the superiority of our proposed method.
Subjective Quality Analysis. We conducted volun- tary user study PludhWwith with 32 participants to subjectively evaluate the quality of our 3D generated assets. The participants were shown side-by-side videos of assets generated using the same prompt input by YOUDREAM(UTTs) HiFA, Fantasia3D, 3DFuse, and MVDream, and were asked to select the best model under the categories 1) Naturalness and 2) Text- Image alignment. The participants were instructed to judge naturalness on the basis of geometrical and anatomical consistency/correctness perceptual quality, artifacts, and details present in the videos.
"This work involved human subjects or animals in its research. Approval of all ethical and experimental procedures and protocols was granted by the Institutional Review Board (IRB), University of Texas. Austin, under FWA No. 00002030 and Protocol No. 2007-11-0066

---

shape X
posa
X
shape
pose control
shape
X
pose control
shape
posa
Back
Figure 6: Ablation over the effect of initial shape and pose control. The initial shape helps in producing clean geometry. while pose control helps to maintain 3D consistency,
guidance scheduling X
guidance scheduling v/ control schaduling
guidance scheduling control scheduling
X
guidonce scheduling v control schedilling
control scheduling
Figure 7: Ablation over scheduling techniques. Using either guidance or control scaling produces unnatural color, using neither produces artifacts such as grass at feet owing to lower diversity of ControlNet compared to Stable Diffusion.
Text-image alignment preference was self-explanatory. A total of 22 prompts and their corresponding 3D assets generated using each model were shown to each participant, thereby accumulating a total of 1408 user preferences. Of the 22 prompts, 13 involved naturally existing animals while the remaining 9 included unreal and non-existent animals. The collected user preferences are shown Fig We observe 60-62% user preference in both the preference categories for our model, strongly indicating the superior robustness and quality YOUDREAM.
We also tested the efficacy of our multi-agent LLM based pose generator via subjective study. We request 16 novel 3D poses of different animals from the multi-agent LLM which uses 16 pre-defined animal poses in our animal pose library. The requested animals were such that there was a high chance using each reference animal pose the library. The participants were shown paired videos of rotating 3D poses, consisting of the pose taken from the library (left-side video, 'reference animal') and the novel pose (right-side video, 'requested animal') generated. Since the participants were not experts in animal anatomy, they were also provided multi-view view images of each animal under their video. They were asked to mark 'Yes" or 'No" for the question: "If this 3D pose <reference pose video> represents 'reference animal' in 'reference pose' pose. Could this 3D pose <generated pose video> represent 'requested animal' in 'requested pose' pose?" The study consisted of the same 32 participants and each subject voted for all the 16 novel poses. Subjects agreed that the generated pose correctly represents 'requested animal' 91 % of the time. Kangaroo (standing pose) generated by the multi- multi-agent LLM using the pose of T-Rex (standing pose) received the lowest agreement among all pairs with 8 out of 32 votes being 'No'. Detailed description of the pose library, the generated poses, and partieulars of the human study provided in Sec. [
Ablation. We present ablation over the effect of using initial shape and pose control in Fig. Without pose control refers to using vanilla Stable Diffusion. Without using initial shape or control, the Janus head problem occurs. With initial shape but without pose control, the geometry improves but still sees the appearance of another head on the elephant's backside. Using pose control without initial shape produces visibly good results, however using both initial shape and pose control results in much cleaner geometry.
In Fig. we show the effect of our scheduling strategies. Without guidance or control scaling, the result has grassy texture at the feet which could be owing to seeing most elephants on grass during TetraPose ControlNet training on limited animal pose data. Using only one kind of scheduling produces incorrect color, showing that both scaling techniques go hand-in-hande

---

5 Conclusion
We presented YOUDREAM, a method to create anatomically controllable and geometrically consistent 3D animals from text prompt and 3D pose input. Our method facilitates the generation of diverse creative assets through skeleton control, which cannot expressed through language and difficult to provide guidance image, especially for unseen creatures. Additionally, we presented pipeline for automatic generation of 3D pose for animals commonly observed in nature by utilizing multi- agent LLM setup. Our 3D generation process enjoys multi-vier view consistency by utillizing 3D pose as a prior. We quantitatively outperform prior work in terms of "Naturalness" and TexxtImmage Alignment" evidenced in the user study.
Acknowledgment
The authors would like thank the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing compute resources that have contributed to the research results reported in this paper. URL: http://www.tacc..ttexaa.dlld
References
Mohammadreza Armandpour, Ali Sadeghian, Huangjie Zheng, Amir Sadeghian, and Mingyuan Zhou. Re- imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arXtv:23044. 04968, 2023.
Mare Badger, Yufu Wang, Adarsh Modh, Ammon Perkes, Nikos Kolotouros, Bernd G Pfrommer, Marc F Schmidt, and Kostas Daniilidis. 3d bird reconstruction: a dataset, model, and shape recovery from a single view. In European Conference on Computer Vision, pages 1-17. Springer, 2020.
Prianka Banik, Lin Li, and Xishuang Dong. A novel dataset for keypoint detection of quadruped animals from images. arXiv preprint arXiv: 108, 13958, 2021.
Yukang Cao, Yan-Pei Cao. Kai Han, Ying Shan, and Kwan-Yee Wong. Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models. arXiv preprint arXiv:2304.009116 2023.
Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 151.00:51 2015.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22246-22256, 2023a.
Yang Chen, Yingwei Pan, Ychao Li, Ting Yao, and Tao Mci. Control3d: Towards controllable text-to-3d generation. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1188-1156. 2023b.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13153, 2023.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024.
Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J. Black. Chatpose: Chatting about 3d human pose. In CVPR, 2024.
Fangzhou Hong, Mingyuan Zhang, Liang Pan. Zhongang Cai, Lei Yang. and Ziwei Liu. Avatarclip: Zero-shot text-driven generation and animation 3d avatars. arXiv preprint XXXi:2205.008355 2022.
Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A Ross, Cordelia Schmid, and Alireza Fathi. Scenecraft: An 1l agent for synthesizing 3d scene as blender code. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.

---

Yukun Huang, Jianan Wang, Ailing Zeng, He Cao, Xianbiao Qi. Yukai Shi, Zheng-Jun Zha, and Lei Zhang. Dreamwaltz: Make scene with complex 3d animatable avatars. Advances in Neural Information Processing Systems, 36, 2024.
Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 867-876, 2022.
Tomas Jakab, Ruining Li, Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Farm3d: Learning articulated 3d animals by distilling 2d diffusion. arXiv preprint ar arXiv:2304. 10535, 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning. pages 4904-4916. PMLR, 2021.
Nikos Kolotouros, Thiemo Alldieck, Andrei Zanfir, Eduard Bazavan, Mihai Ficraru, and Cristian Sminchisescu. Dreamhuman: Animatable avatars from text. Advances in Neural Information Processing Systems, 36, 2024.
Zhiqi Li, Yiming Chen, Lingzhe Zhao, and Peidong Liu. Mvcontrol: Adding conditional control to multi-view diffussin for controllable text-to-3d generation. arXiv preprint arXiv:2311.14499 2023.
Zizhang Li, Dor Litvak, Ruining Li, Yunzhi Zhang, Tomas Jakab, Christian Rupprecht, Shangzhe Wu, Andrea Vedaldi, and Jiajun Wu. Learning the 3d fauna of the web. arXiv preprint arXiv:2401.0240,, 2024.
Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen, Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching. arXiv preprint 2rXiv:231..11288 2023.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov. Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zemo-shot one image to 3d object. In Proceedings of the IEEE/CVP International Conference Computer Vision, pages 9298-9309 2023.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851-866. 2023.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.0718 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 1061 2021.
Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia 2022 Conference papers, pages 1-8, 2022.
Chong Mou, Xintao Wang. Liaggbnn Xie, Yanze Wu, Jian Zhang. Zhongang Qi, and Ying Shan. adi-adapter: Learning adapters to dig out more controllable ahility for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296-4304, 2024.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding, ACM transactions graphics (TOG), 41(4):1-15, 2022.
Xun Long Ng, Kian Fng Ong, Qichen Zheng, Yun Ni, Si Yong Yeo, and Jun Liu. Animal kingdom: A large and diverse dataset for animal hchavior understanding. In Proceedings of the IEEF/VVF Conference on Computer Vision and Pattern Recognition, pages 19023-19034 2022.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975 10985, 2019.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pages 8748 8763. PMLR, 2021.

---

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffussio models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10684-10695, 2022.
Oindrila Saha, Grant Van Horn, and Subhransu Maji. Improved zero-shot classification by adapting vlms with text descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
pages
Chitwan Saharia, William Chan. Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Chasemipour Raphael Gontijo Lopes, Burcu Karagol Ayan. Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-
36494, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278-25594, 2022.
Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Hyeonsu Kim, Jaehoon Ko, Junho Kim, Jin-Iwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint arXiv:2303.07957 2023.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6101, 2021.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. The Twelfth International Conference on Learning Representations, 2023.
Yawar Siddiqui. Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießner. Meshgpt: Generating triangle meshes with decoder-only transformers. arXiv preprint 21X1v:231..1545, 2023.
Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang. Zishan Qin, and Stephen Gould. 3d-gpt: Procedural 3d modeling with large language models. arXiv preprint arXiv:2310.1294 2023.
Jiaxiang Tang. Stable-drcamfusson:: Text-to-3d with stahle-diffusion, 2022. http://gthuhhcom/saaawweyyaaaaa dreamfusion.
Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi-view image generation with correspondence-wware diffusion. arXiv, 2023.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEFF/CVP Conference on Computer Vision and Pattern Recognition, pages 12619-12629, 2023.
Peng Wang and Yichun Shi. Imagedream: image-prompt multi-vicw diffusion for 3d generation. arXiv preprint arXiv:2312.02200/ 2023.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High- fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Dove: Learning deformable 3d objects by watching videos. International Journal of Computer Vision, 131(10))2623-28) 2634, 2023a.
Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Magicpony: Learning articulated 3d animals in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8802, 2023h.
Jiacong Xu, Yi Zhang, Jiawei Peng, Wufei Ma. Artur Jesslen, Pengliang Ji, Qixin Hu, Jiehua Zhang, Qihao Liu, Jiahao Wang, et al. Animal3d: comprehensive dataset 3d animal pose and shape. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9099-9109, 2023.
Chun- Han Yao, Wei-Chil Hung, Yuanzhen Li, Michael Rubinstein, Ming-Hsuan Hsuan Yang, and Varun Jampani. Lassie: Learning articulated shapes from sparse image ensemble via 3d part discovery. Advances in Neural Information Processing Systems, 35:15296 15308, 2022.

---

Chun-Han Yao, Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Mingg-ssuan Yang, and Varun Jampani. Hi-lassie: High-fidelity articulated shape and skeleton discovery from sparse image ensemble. In Proceedings of IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4853-4862, 2023.
Chun-Han Yao, Amit Raj. Wei-Chih Hung. Michael Rubinstein, Yuanzhen Li, Ming- Ming-Hsuan Yang. and Varun Jampani. Artic3d: Learning robust articulated 3d shapes from noisy web image collections. Advances in Neural Information Processing Systems, 36, 2024.
Jianglong Ye, Peng Wang, Kejie Li, Yichun Shi, and Heng Wang. Consistent- 1-to-3: Consistent image 3d view synthesis via geometry-awee aware diffusion models. arXiv preprint arXiv:2310.03020 2023.
Fukun Yin, Xin Chen, Chi Zhang. Biao Jiang, Zibo Zhao, Jiayuan Fan, Gang Yu, Taihao Li, and Tao Chen. Shapegpt: 3d shape generation with a unified multi-modal language model. arXiv preprint arXiv:2311.1700 2023.
Zeqing Yuan, Haoxuan Lan, Qiang Zou, and Junbo Zhao. 3d 3d-premise: Can large language models generate 3d shapes with sharp features and parametric control? arXiv preprint vrXiv:2401.06445 2024.
Huichao Zhang, Bowen Chen, Hao Yang, Liao Qu, Xu Wang. Li Chen, Chao Long, Feida Zhu, Daniel Du, and Min Zheng, Avatarverse: fiigh-quaitty & stable 3d avatar creation from text and pose. In Proceedings of the AAAI Conference Artificial Intelligence, pages 7124-111 123-713,, 2024a.
Jianfeng Zhang, Zihang Jiang. Dingdong Yang. Hongyi Xu. Yichun Shi, Guoxian Song, Zhongcong Xu, Xinchao Wang, and Jiashi Feng. Avatargen: 3d generative model for animatable human avatars. In European Conference on Computer Vision, pages 668 685. Springer, 2022.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to rext-to-image diffusion models. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 3836-3847 38473847.22 2023.
Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli general-purpose Ouyang. Motiongpt: Finetuned Iims are general purpose motion generators. In Proceedings of the AAAI
Conference on Artificial Intelligence, pages 7368 7376, 2024b.
Junzhe Zhu. Peiye Zhuang, and Sanmi Koyejo. Hifa: High-fidelity text-to-3d generation with advanced diffusion guidance. In The Twelfth International Conference on Learning Representations, 2023.
Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael Black. 3D menagerie: Modeling 3D shape and pose of animals. In JEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
Silvia Zuffi, Angjoo Kanazawa, and Michael Black. Lions and tigers bears: Capturing non-rigid, 3d, articulated shape from images. In Proceedings of the IEEE Conference on Computer Vision and Patern Recognition, pages 3955 3963, 2018.

---

Appendix
A
More Results
We show our method's performance for varied styles in Fig. Even though our ControlNet is trained on images of animals in the wild, YOUDREAM produces assets with significant style alteration. This attainable because of our control scheduling and guidance scheduling approach, which ensures consistent geometry to be formed in initial iterations with higher control scale and style being finalized during the later iteration with higher guidance scale.
zoomed out DSLR photo of gold eagle statue
soft cute tiger plush toy, in standing position
a realistic lizard with a magician's hat
an elephant kicking a soccer ball
Figure 8: Results compositional and style prompts. We show our method performs well while generating animals with style alterations or object interactions.
Single LLM
Multi-agent LLM
Single LLM
Multi-agent LLM
Single LLM
Multi-agent LLM
Hippo
Greater Flamingo
Horse
Figure 9: Pose generation using single LLM vs our multi-agent LLM setup. For "Hippo", "Greater Flamingo", and "Horse", we show 2D view of the pose generated by single LLM compared to our multi-agent setup.
B Additional Ablations
In Fig. we show that using single L.LM agent performs much worse in generating 3D poses compared to multi-agent setup which includes Finder, Observer and Modifier GPTs.

---

Pose condition
Gazelle
Horse
Baboon
OpenPose ControlNet
TetraPose ControlNet
Figure 10: Comparison with OpenPose ControlNet for generating animals. OpenPose ControlNet produces the animal in the prompt for "Horse" and "Baboon", but either does not follow control or makes unnatural anatomy. For "Gazelle" meaningless image is produced.
"Golden ball with wings"
"Golden ball with two pairs of wings"
MVDream
YOUDREAM (ours)
Figure 11: Toy example showing inefficacy of text prompt. We show that pose control helps to add the additional wings at the desired location. MVDream makes two wings for "Golden ball with two pairs of wings" with differently shaped wings compared to "Golden ball with wings".
Since our ControlNet guided 3D generation pipeline can produce out-of-domain animals well, the question arises if OpenPose ControlNet can be utilized to generate animals. We show in Fig. that OpenPose ControlNet produces artificial looking images for animals. We use a "human on all fours" " image to obtain the pose for OpenPose and generate similar keypoint orientation for our TetraPose format. Even though the pose is unnatural for animals, with hips and shoulder very close to spine, TetraPose ControlNet produces clean images following the pose.
toy example based on golden ball with wings is presented in Fig. to show that text by itself can be ambiguous to convey meaning. When prompted for two wings, MVDream produces modified pair of wings, whereas YOUDREAM follows the user pose control to produce four wings. YouDream performs significantly better for many prompts involving real animals such pangolin and giraffe.

---

In Fig. 13 we show the effect of different guidance and control scheduling strategies. Note that for Seed 123
Seed
Seed 3456
Seed 23456
Figure 12: Variation with seed. Our method is robust across seeds and generates slightly different faces and stripes for various seeds.
Guidance Schedulingg cosine Control Scheduling cosine
Guidance Scheduling cosine Control Scheduling linear
Guidance Scheduling: linear Control Scheduling linear
Guidance Scheduling - linear Control Scheduling cosine
Figure 13: Comparison of various scheduling techniques. Using cosine strategy for both produces oversaturation, while using cosine strategy for guidance scheduling and linear for control scheduling produces oversmooth textures at the legs. Results of using both linear scheduling is closest to our strategy, but is lesser textured (notice feet and ears).
Fig. 15 shows results for both the animals generated using MVDream and YOUDREAM. Even though MVD MvDream is 3D aware model, it still produces artificial looking results in many cases. While results generated using YOUDREAM are much more natural perceptually and contain realistic textures found in the respective animals.
We show that our method does not require seed tuning for generating consistent results Fig. Variation in textures and shapes can be seen across seed.
all, guidance scale increases while control scale reduces.
We show that not using LaGB loss produces holes and flickering in generated assets. We show the normals for elephant and tiger for this purpose.
Comparison with 3D Animal Model. 3D animal reconstruction method based on image inputs. Given an input image 3DFaaa failed to capture high-frequency details and follow the input image (see tail and snout in Fig. [6]. whereas our method produced highly detailed animal given the input pose and text, which closely followed the input pose control.
We compared our method against 3DFauna Li et al. (2024),

---

Without Laos
With Leas
Without Leoa
With Lega
Figure 14: Effect of using LRGB. Not using LRGB results is hollow geometry and flickering. The chin of the tiger appears and disappears based on view, a view where the chin disappeared has been chosen.
a pangolin
giraffe
YOUDREAM
MVDream
Figure 15: More Comparison with MVDream. We compare our method with MVDreem for simple prompts. MVDream results are clearly missing the texture of the scaly body of the pangolin, while their giraffe has a tok-like geometry and hence unnatural. In contrast YOUDREAM produces very realistic results.
C Comparison with more text-to-3D baselines
We also compare with other text-to-3D generative methods guided by T2I models. These include Stable Dreamfusion Tang (2022). Prolific Wang et al. (2024), and LucidDreamer Liang et al. (2023). In Fig. 17 we show that all these methods suffer geometric and anatomic inconsistencies, as well as fail to capture the text faithfully.
D Exploring severely Out-of-Domain cases
We explore generating animals well out-of-domain with respect to our animal library (see Sec. G We show in Fig. 18 that we can generate "clownfish" and "four-legged tarantula" without any human intervention using our fully automatic pipeline comprising of the multi-agent LLM pose editor and the 3D generation pipeline. Our multi-agent L.LM setup has been explored in the context of four-limbed animals and generating more appendages is a direction of future work.
E
Scaling to higher dimension NeRF
In Fig. 19 we show that we can scale 3D generation to higher dimension NeRF without any changes hyperparameters. can be observed that scaling to the larger NeRF improves the sharpness of the asset considered and results in crisper textures.

---

3D Fauna
Image to 3D
Ours Text Pose to 3D
Input Image
Predicted
Input Pose
Predicted Normal
Figure 16: Comparison with 3DFauna. Our method produces more detailed geometry compared to the baseline.
Stable Drcamfusion
PromifcDreammrr
LucidDreamer
YouDream
Front View
An elephant
Side View
a llama with octopus
Front View
zoomed out photo
tentacles body
Side View
Figure 17: Comparison with additional prior art methods. Even though LucidDreamer performs better than Stable Dreamfusion and ProaificDreamer, it shows the same failures discussed in the main paper.
Implementation details
Poses used to generate 3D animals main paper. Fig. 20 shows the 2D views of 3D poses used to generate the 3D animals in main paper.
TetraPose ControlNet Training:
We used annotated poses from the Aw AwA-pose Banik et (2021) and Animal Kingdom Ng et al. (2022) datasets to train ControlNet in similar way as the original
paper, which uses stable diffusion version 1.5. AwA-pose consists of 10k annotated images covering 35 quadruped animal classes, while Animal Kingdom provides 33k annotated images spanning 850 species, including mammals, reptiles, birds, amphibians, fishes and insects. From a combined set of 43k samples, we carefully selected subset including only mammals, reptiles, birds, and amphibians. We also eliminated any sample having less than 30% of its keypoints annotated. The curated dataset consists of 13k annotated samples. To increase diversity in learning, and to improve

---

Multi Multi-gent aae--ggentt
Pose guided 3D geseraion
Roseate Spoonbil
Clomnish
Multi-agent LLM
Pose-guded gDgmerai
Shepherd
Four-legged Larantula
Figure 18: Generating more OOD assets through automatic pipeline. Using multi-agent LLM setup we first generate the 3D poses "clownfish" and "four-legged tarantula". We then use the produced 31 poses guide our 3D generation. We observe that the multi-agent LLM pose editor chooses "Roseate Spoonbill" as the base 3D pose to be modified into "Clownfish", and "German Shepherd" is chosen for modifying "four-legged tarantula".
NeRF Dimensions
128
NeRF Dimensions
256 256 256
Figure 19: Increasing NeRF dimensions. On increasing each NeRF dimension by 2x we generate sharper and cleaner 3D asset for the prompt "a tiger" without any change in hyperparameters.
test-time generation at any scale and transformations, used combination data augmentation strategies consisting of random rotations, translations, and scaling while training so as to handle highly varied and heavily occluded 2D pose samples during 3D generation. The model was trained over 229k iterations with batch size of 12, constant learning rate of 1e-5, on a single Nvidia RTX 6000. The model converged after around 120k iterations and would not overfit even up to 200k iterations, owing in part to the augmentation strategy.
3D Pose editing and Shape generation: We used the following 18 keypoints represent every quadruped: left eye, right eye, nose, neck end, x thighs, knees, 4 x paws, back end, and tail end. For the upper limbs of birds, i.e. wings, their front thighs, knees, and paws are defined in accordance with how their upper limbs move. The user can begin with any initial pose from the animal library and modify its keypoinns using the Balloon Animal Creator Tool. This tool was developed using THREE.js and can be run on any web browser. The tool provides buttons for the following functions: 1) add extra head. add extra limb front, 3) add extra limb back, and 4) add extra tail. After appropriate modification of the pose the user can press the button to create mesh around bones. This button press invokes calls various functions defined to create each body part, based on their natural appearances using simple mesh components such as ellipses (eyes and torso), cylinders (neck, tail, and limbs), and cones (nose). The combined mesh and the corresponding keypoints can be downloaded by clicking the Export Mesh and Save Keypoints button. An example of this process used for creating the three headed dragon using the Balloon Animal Creator tool depicted in Fig. 21

---

Figure 20: Snapshots of 3D poses used for generating objects in the main paper. For a 2D view of each object, we show the corresponding 2D view of the 3D pose.
Centine
Ale Ores Ceck
AMl
Figure 21: 3D Pose editing and Shape generation. We show snapshots of our 3D pose creator tool with all functionalities.
Mesh depth guided NeRF initialization: The mesh downloaded in the previous step was used to provide depth maps the pre-trained depth guided ControlNet, which produces the gradient loss by SDS, which in turn is used to pre-train the NeRF. The pre-training helps achieve a reasonable initial state for the NeRF weights, which can then be refined in the final pose-guided training stage. The diffusion model was pre-trained for 10,000 iterations using the Adam optimizer with a learning rate randomly sampled in
of le - 3 and a batch size of 1. During training, the camera positions spherical coordinates, where the radius, azimuth, and polar angle of camera position were sampled from [1.0, 2.0], [O, 360], and [60, 120].
Pose-guided SDS for NeRF fine-tuning: Finally, we fine-tune the NeRF using the pre-trained ControlNet to provide 2D pose guidance SDS. The gradients computed using the noise residual from SDS were weighted a similar manner as DreamFusions where w(t) = of and f was annealed
iter using f = fmar (tmaz tman) V totai_iters' We set fmax to be 0.98, t.in to 0.4. Similar to
the previous stage, we trained the model over total iters 10, 000 using the same settings for the optimizer. Using cosine annealing, we reduced the control from an initial value of to a final value of 0.2, while updating guidance scate linearly from guidancemin. 50 to guidance These settings helped reduce the impact of ControlNet gradually over the training process, while improving quality by gradually increasing strength classifier-free guidance. The camera positions were randomly sampled as in stage 1, as were the radius, azimuth, and polar angle of the camera. ÅRGB was set to 0.01. The 3D avatar representation renders images directly in the RGB space of IR 1128x128x3 We use Instant-NGP Müller et al. (2022) as the NeRF representation. The pre-training stage, if used, takes less than 12 minutes to complete, while the fine-tuning stage takes less than 40 minutes to complete on single A100 40GB GPU.
100.

---

Computational Resources: All the experiments pertaining to YOUDREAM and 3DFuse were run on Nvidia A100 40GB GPU. Few experiments for MVDream and all experiments of HiFA required running on A 1000 80GB GPU, while all experiments for Fantasia3D were run on 3xA 100 40GB GPUs.
G Animal Library
Our animal library B contains total of 16 animal/pose combinations:
Giraffe
Elephant
German Shepherd
Eagle sitting
Eagle flying
American Crocodile
Tree Frog
Roseate Spoonbill sitting
Roseate Spoonbill flying
Raccoon standing four legs
Raccoon standing on two legs
T-Rex
Lizard
Tortoise
Bat
Otter
All common animals results shown in this paper are either using these 3D poses or poses modified from one of these by our multi-agent LLM. The library entries are chosen intuitively such that each has significant anatomical variation from the others so as to cover the large range of variety observed in the animal kingdom.
H Multi-agent LLM Implementation Details and Scope
We use the recently released "GPT-40" API of OpenAl with max_tooees as 4096 and temperature as 0.9. The keypoints are represented as a dictionary in JSON format and converted to a string to be appended to the text prompts of the observer and modifier LLMs. The observer LLM is instructed using the system prompt about details of the 3D coordinate space and relations among the various keypoints. The bone sequence which represents the connections between the various keypoints is also provided as a list to the observer GPT's prompt to reason about relative anatomy based on bone lengths. Finally, the multi-agent LLM outputs keypoint dictionary in the same format provided to it. The Multi-agent LLM is able to generate various animals by taking reference from a set of 16 3D animal poses. However, this setup can generate poses for animals that well out-of-domain of these 16 animals as shown in Fig. [8] The Multi-agent LLM supports generating animals that can be represented using four limbs. Generating more than four-limbed animals such insects using the LLM setup is direction of future work. We open-source this setup along with our project code.
I Evaluation using CLIP score score
Based on the user study, clear that users majorly prefer either MVDream or YOUDREAM. Hence we also compute CLIP similarity score for each of the two methods as the average CLIP score over 9 views of each of the prompts used for the user study. TableTsshone that our method outperforms MVDream based on the CLIP similarity score. We use the VIT-B/32 model for evaluation.

---

Choose the best model based on naturainess.
Samples for analysis
NEXT
Naturalnsss factors: Geometrical and anatomical coneisennyylommecceess perceptual quality, artiacts and details
User Study Interface for Naturalness preference: A snapshot of the interface displaying rotating videos of the results generated by the five chosen models. The user was provided with sample images of the real animal for analyzing anatomical consistency.
Figure 22:
MVDREAM
YOUDREAM
CLIP Score
29.78
30.86
Table 1: CLIP similarity score comparison for MVDream and YOUDREAM
J User Study Details
Graduate students at the University Texas at Austin volunteered for participating the user study, All the information regarding the preferences requested in the study, the judgement criteria, and operating the interface were provided at the beginning of the study. A consent form documenting the purpose of the study, risks involved in the study, duration of the study, compensation details, and contact details for grievances was signed by each user before the beginning of their study session. Screenshots of the user study interfaces shown in Fig, 22 and Fig. 23 The 22 prompts used for generating the 3D assets used in the user study are as follows:
1. A giraffe
2. A lizard
3. A raccoon standing two legs
4. A tiger
5. A lion
6. A red male northern cardinal flying with wings spread out
7. A roseate spoonbill flying with wings spread out
8. A Tyrannosaurus rex
9. A pangolin
10. A bear walking
11. A horse
12. A mastiff
13. A soft cute tiger plush toy, in standing position

---

Choose the best model based on naturainess.
pose represents:
Animal: German Shepherd Pose: Standing
this 3D pose represent Bear in walking pose?
Samples for analysis
Yes
No
NEXT
Figure 23: User Study Interface for Generated Pose Preference: snapshot of the interface displaying rotating pose video of the 'reference animal' (left side of the interface) used by the multi- agent LLM for generating 3D pose of 'requested animal' (right side of the interface) in 'requested pose'. The user provided with real samples of the 'reference animal' and the 'requested animal' (one side and one front-view each for better anatomical analysis.
14. An elephant standing on concrete
15. A dragon with three heads separating from the neck
16. A realistic mythical bird with two pairs of wings and two long thin lion-like tails
17. Golden ball with wings
18. A six legged lioness, fierce beast, pouncing. ultra realistic, 4k
A giraffe with dragon wings
20. A zoomed out photo llama with octopus tentacles body
21. A zoomed out DSLR photo gold eagle statue
22. Golden ball with two pairs of wings
K Animation using Pose Sequence
YOUDREAM can also be used to generate animated videos by generating 3D assets for every pose from pose sequence. In Fig. 24 we show frames chosen from pose sequence and the corresponding render of their generated 3D mesh. Despite this, generating a longer animation sequence using YOUDREAM would be highly resource expensive and time consuming task. We hope this work will inspire further exploration of efficient methods for controlled animation.
L Limitations and Discussion
While our method produces high quality anatomically consistent animals, the sharpness and textures can improved by utilizing a number of tricks used by recent papers. We use 128x 128 NeRF, while our baseline HiFA uses 512x512, while MVDream uses 256x 256. We use smaller NeRF for the sake of lower time complexity compared baselines. Other tactics such as using DMTet or regularization techniques also plug-and-pla -laay for our method and may improve sharpness.

---

Animation. Bottom-row: Sampled pose frames from a pose sequence of tiger walking. Top-row: Camera captured image of the 3D mesh corresponding to the view of the 3D pose shown below it in the bottom-row.
Figure 24:
We show several diverse examples of automatically generating common animals found in nature. However there could exist unusually shaped animals whose 3D pose cannot be satisfactorily generated using our multi-agent L.LM setup. these cases, manual editing of 3D pose might be required over the LLM generated 3D pose. However, we believe pose editor tool is highly interactive and user-friendly, thus requires very low human effort modify poses.
Broader Impact. Al generated art has been widely used in recent times. YOUDREAM enables artists gain more control over their creations, thus making the process of content creation easier. As our method uses Stable Diffusion, it inherits the biases of that model. TetraPose ControlNet training uses existing open-source animal pose datasets instead of internet scraped images, hence avoiding any copyright issues.
Licenses
URL https: http://////nnnhhooosss/////uuuu http: HiFF
Citation
License
Zhu et al.
2023
Apache License 2.0
https:
com/ wtt.aauwwoo.////C/A///////////// 3Fuue
Sen ct al.
20555
N/A
Lctps://githut
com//orrllaalaa SCUT/ CCUTT/Fattaiaa
Chen et a 2023a
Apache License 2.0
Shi et al.
2023
MIT License
https:
Tang 2022
Apache License 2.0
Wang
2024
Apache License 2.0
Dreamer
Liang
MIT License
https:/
Zhang et al.
2023
Apache License 2.0
Bankk al.
202
MIT License
X1I et al.
2023
N/A

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:[N]]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?

---

Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
cttppsscccpbbbccpppeeecccc guwdes/ddeeeeee Cooeeeoodeesssseeeeee / casssnnnnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: The experiments conducted in this paper do not have either ground truth well-defined metric for evaluation. We have reported CLIP score following prior work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics atttps:/neeuris. http://aarr...cc//bbbb.cccccccaa
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "k6O1aqGT8H", "reviewer": "Reviewer_127z", "summary": "The paper proposes a novel technique called YouDream for text-guided animal generation. Specifically, the authors first propose a multi-agent LLM that's capable of generating a 3D pose of the text-described animal. Secondly, YouDream includes a TetraPose ControlNet to generate the images based on the projection of the 3D pose. Last but not least, the authors design a tool for generating 3D rough shapes based on the 3D pose, which will serve as the initialization for the generation process. Extensive experiments, including qualitative, quantitative, and ablation studies, have been included to demonstrate the effectiveness of the network.", "questions": "Please see weakness listed above.", "limitations": "Limitation has been discussed in the Appendix.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The strengths of the proposed paper can be summarized as:\n\n+ The paper proposes a multi-agent LLM to generate 3D pose of animals, which is novel and interesting for 3D generation.\n\n+ The results are impressive, showcasing better performance than existing techniques.", "weaknesses": "The weakness of the proposed paper can be summarized as:\n\n- Missing comparisons with GaussianDreamer, LucidDreamer, etc, which are more recent works than 3DFuse, Fantasia3D, and HiFA. Meanwhile, it would also be better to compare with DreamFusion, Magic3D, and prolificdreamer to further demonstrate.\n\n- The ablation in Fig. 6 is not positive. Actually: it seems that the initial shape provides more contributions, while the pose control doesn't affect much.\n\n- It would also be good to show quantitative evaluations using CLIP-based metrics.\n\n- The methods can only process 16 categories of animals. Although it's a limitation of current datasets, it will still hurt the method's performance and bring the question of whether we need the technique specifically designed for these animals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_We appreciate the reviewers' thorough evaluation of our work, including recognition of our novel contributions and results. We thank them for their insightful suggestions to enhance our paper. If our response has adequately addressed their concerns and provided new insights, we would be grateful if they consider revising their score._\n\n__Q1. Comparison with more prior art__\n\nWe thank the reviewer for the suggestion. We added comparisons with LucidDreamer, Stable DreamFusion, and ProlificDreamer in Fig.(a) of the rebuttal pdf and report the same inconsistencies. Among prior art, LucidDreamer produced the best results but was plagued with inconsistencies such as 6 legs, 2 tails and 2 trunks for an elephant, while also showing unnatural colors. For the novel unseen animal ``Llama with octopus tentacles body” no prior method faithfully captures the text.\n\n__Q2. Fig.6 shows pose control does not affect generation__\n\nPlease note that Fig.6 shows the side and back view of the assets. Not using pose control results in a head forming on the backside of the elephant and hence __proves the efficacy__ of the pose control.\n\n__Q3. Quantitative evaluation using CLIP__\n\nPlease refer to Table.1 in Supplementary Sec. F where we compare CLIP similarity score with the best performing prior art: MVDream. \n\n__Q4. YouDream can only generate 16 categories of animals__\n\nThe 16 animal 3D pose library is meant to act as support for our multi-agent LLM and does not limit generation to only these categories. We demonstrate the same by generating fishes and insects using only these as support (see Fig.(b) rebuttal pdf). Our trained ControlNet can generate animals significantly out-of-domain, despite it being trained on a tetrapod animal database. This is evident from Fig. 1 (main paper)  where we show that YouDream can generate animals significantly out-of-domain of the Tetrapose ControlNet training data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xZslWNGuj9", "reviewer": "Reviewer_PBQU", "summary": "This work proposes a method to generate 3D models with pose control. It mainly contains two steps: 3D pose generation with LLM agents and text-to-3D generation with pose-conditioned ControlNet and viewpoint sampling. From the experiments, the method can generate novel 3D animals with controlled poses, while previous methods without pose conditions can not.", "questions": "What are the finder, observer, and modifier composed of, respectively? What modifications can the modifier make to the given 3D poses? For the text prompt “A giraffe with dragon wings,” are the poses automatically generated by the modifier?", "limitations": "None", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method enables the generation of 3D tetrapod animals with diverse poses.\n\n2. The method can generate novel animals that are out of the domain of training data.\n\n3. The pose-guided generation enables more fine-grained pose control for the text-to-3D generation.", "weaknesses": "1. The work allows more explicit pose control for generating tetrapod animals. However, original text-to-3D models can achieve this using only text, without limitations on object classes or the need for human labor to design poses.\n\n2. The poses appear to be user-designed, but I still doubt the difficulty of creating novel poses with the given library, like it's hard to create the tiger walking sequences (Fig. 21) given the library, are the poses generated from Multi-agent LLM or designed manually.\n\n3. The effectiveness of the LLM-based pose editor seems limited; how can a 2D-based model (LLM) effectively reason about the 3D poses? The given initial poses in Fig.3 are already plausible. What if more noisy poses are given?\n\n4. The generated model is of bad quality and not comparable with MVDream, although it has a 3D GT. I doubt whether adding the explicit pose control will make the mesh quality worse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_We thank the reviewer for their detailed questions which will help us improve our manuscript. If our response has adequately addressed their concerns and provided new insights, we would be grateful if they consider revising their score._\n\n__Q. Prior art can achieve explicit pose control using only text. YouDream is limited by object classes and human labor.__\n\nWe very strongly disagree with the reviewer’s opinion about prior models achieving generation of tetrapod animals using only text. We humbly request the reviewer to observe Fig. 1, 4, and 15 which highlight multiple examples where the prior text-to-3D models are unable to generate the desired outputs. Although models trained on 3D data such as MVDream can generate commonly seen tetrapod animals without artifacts, they lack naturalness (Fig.15 main paper) due to their limited training data. Despite being trained on 2D images, YouDream consistently generates realistic, geometrically consistent, and faithful-to-text assets unlike any previous method. Moreover, YouDream can easily enable the generation of novel unseen animals making it highly useful for the VFX industry. Since such unseen animals are not part of existing ML datasets, forcing models trained on these to create view-consistent images of such unseen animals is nearly impossible without an external control such as a 3D pose. For the same reasons, an LLM cannot generate poses for such unseen animals. Inadvertently this introduces the need for human effort to design poses for creative imaginary animals. For animals that are representable by prior methods, YouDream employs the multi-agent LLM for generating the pose.\n\n__Q. Is tiger walking sequence generated by Multi-agent LLM?__\n\nOur automatic pipeline can cater to animals well represented in existing datasets that LLMs or diffusion models have been trained on. Unseen animals or fine-grained pose designs as required to create a walking sequence cannot be created using an LLM based on purely textual description unless specifically trained to do so using a dedicated dataset. We generate the walking seq. manually only to illustrate fine-grained pose controllability. Although the Multi-agent LLM can create a generic walking pose, creating a sequence of poses remains as future work.\n\n__Q. How can a “2D-based model (LLM)” effectively reason about the 3D poses? What if noisy poses are given to the LLM?__\n\nPlease note that the LLM is purely text-based and operates in a multi-dimensional space. LLMs have previously demonstrated the ability to understand and reason about spatial relationships without any fine-tuning. Llava[1] shows that GPT can understand and describe spatial orientation in images using numerical bounding box coordinates in text prompts. LLM-GROP[2] shows GPT's capability for spatial understanding and object rearrangement in 3D space. LLMs also possess valuable relative knowledge of various classes, including animals and birds, useful for improving zero-shot classification[3]. We believe all these capabilities which evidence the logical and relational reasoning powers of LLMs help GPT understand and manipulate 3D poses.\n\nDirect pose generation by LLM without input-pose reference was mostly unsuccessful and required case-by-case prompt adjustments. Comparison between single-agent and multi-agent LLM results is shown in Fig. 9 of the main paper. These details will be added to the manuscript. Based on your suggestion, we added random noise to our keypoints within a 20% margin. The multi-agent LLM setup was able to remove noise from some keypoints, particularly on the limbs and tails, but struggled with the points on the head. Our library of 16 poses, designed to cover a wide range of animal shapes and poses, has proven effective in generating a vast range of animals presented in the paper and supplementary videos. Additionally, we demonstrated the LLM's ability to generate keypoints for a 'clownfish' and a 'four-legged tarantula' in Fig. (b) of the rebuttal PDF.\n\n__Q. The generated model is of bad quality and not comparable with MVDream, although it has a 3D GT.__\n\nWe disagree with this claim. There are primarily two reasons to do so:\n\na) Quality is a highly subjective aspect and opinions vary from person to person. For this reason, we do not make any objective claims about the overall quality of our method. Our user study presented in Sec. 4 is a representative opinion of multiple user preferences averaged over multiple assets.\n\nb) In the generative 3D domain, quality encompasses factors like naturalness, text alignment, and geometric artifacts. While we acknowledge that MVDream has better material quality due to training on human-designed 3D assets under various rendering conditions, this doesn't mean it has better overall quality. Our user study, shown in Fig. 5, includes preferences from 32 participants for assets generated using 22 diverse prompts, demonstrating the superiority of our method in terms of naturalness and T2I alignment.\n\n__Q. I doubt whether adding the explicit pose control will make the mesh quality worse.__\n\nAs noted by the reviewer, adding explicit pose control does not worsen the quality of the mesh.\n\n__Q. Details about the Multi-agent LLM and its scope__\n\nAll agents used are GPT-4o instances with well-defined system prompts which can be seen in the GPT_kpt_maker.py file. Although the Modifier LLM has not been explicitly instructed to make any specific kinds of modifications we have observed that it can shift, scale, and rotate keypoints w.r.t other keypoints. \nFor all assets in Fig. 1, the poses have been manually generated. This is noted as “artist's creative control” in the figure caption. We will clarify this further in the manuscript.\n\n[1] Liu et.al, Visual instruction tuning. NeurIPS’24.\n\n[2] Ding et.al, Task and motion planning with large language models for object rearrangement. IROS’23\n\n[3] Menon et.al, Visual Classification via Description from Large Language Models. ICLR’23", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jQ53J2csYT", "reviewer": "Reviewer_rnqi", "summary": "The paper presents YouDream, a framework for text-to-3D animal generation. The two keys to their methods are (1) a tetra-pose ControlNet that synthesizes animals given a text prompt and 2D tetrapod poses, and (2) a multi-agent LLM system that modifies 3D keypoint templates to generate different animal poses. Using the generated 3D animal poses as guidance, and the ControlNet for Score Distillation Sampling (SDS), YouDream can optimize Neural Radiance Fields (NeRFs) with better geometry and appearances, alleviating the multi-faces problem common in previous text-to-3D approach.\n\nQualitative comparisons and user studies in the paper suggest that YouDream achieves more favorable results compared to previous text-to-3D methods for 3D animal generation.", "questions": "Please clarify the following questions if possible:\n- L7-8: It might be more appropriate to say “generates 3D animals with controllable poses”. As shown in Figure 1, Figure 4, and Figure 15, other methods can also generate 3D animals, sometimes with even more appealing textual qualities (i.e., Figure 15 MVDream Giraffe). \n- There are flickerings in some of the supplementary videos across views. Since the underlying representation is a NeRF, I expect the 3D results to be view-consistent. It would be great if you could clarify if the flickerings are caused by compression or other issues. \n- L169-172 mentioned that TetraPose ControlNet works well for out-of-domain animals, which is somewhat backed by the llama octopus example (Figure 1). It would be interesting to see more results on non-tetrapods, such as snakes, fishes, insects, etc. This is a bit out of the scope of this manuscript, but it would be great to see to what extent the proposed multi-agent 3D pose generator and TetraPose ControlNet work.", "limitations": "The limitations are addressed appropriately in the paper. Please answer/clarify the above-raised questions/weaknesses if possible.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presented approach makes sense, and shows promising results. In particular, it has the following strengths:\n- An automatic system that produces 3D poses/skeletons for various animals. This is a very creative way to exploit the LLM. \n- The framework can produce 3D animal assets with more accurate 3D geometry, reducing the multi-face problem that plagues many SDS-based approaches.\n- YouDream offers controllability over the animal poses, making it a potentially useful tool for 3D artists.\n\nIn summary, the paper presents a promising framework that tackles an interesting problem (3D animal generation) that can benefit 3D artists and beyond, and the novelty lies in how YouDream exploits the LLM to handle diverse, varying animal skeletons.", "weaknesses": "YouDream has some weaknesses\n- Color saturation artifacts. This is a common problem among SDS-based approaches and could be fixed using implementation tricks from other work.\n- While greatly alleviating the multi-face problem, we can still observe artifacts (e.g., multiple pairs of ears in Supp. raccoon_standing), and extra limbs (Figure 8, tiger plush toy). The problem could be mitigated with a more detailed skeleton (i.e., more landmarks/keypoints).\n- Controllable, but not editable – YouDream has to re-run the optimization process to create the same 3D animal in a different pose. \n- Similar to many other text-to-3D methods, no quantitative/objective evaluations available. While the presented results look pretty good, we cannot assess/quantify the improvement over previous methods. This, however, is not a problem unique to YouDream.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_We sincerely thank the reviewer for their in-depth analysis of our work and for providing extremely helpful comments and questions which will greatly help us to improve our manuscript. We deeply appreciate the reviewer's acknowledgment of the impact of our work and their recognition of YouDream’s novelty and superior results. If our response has adequately addressed their concerns and provided new insights, we would be grateful if they consider revising their score._\n\n__Q. About alleviating remaining artifacts__\n\nFor the rebuttal, we have been able to improve quality by training using a higher dimensional NeRF. We used a low-dimensional NeRF due to experimental resource constraints. We plan to experiment with improving texture and color quality using variants of SDS and other tricks in the future.\n\nWe agree with the reviewer’s observation: a more detailed skeleton will reduce artifacts around the ears, fingers, etc. The extra limbs in Tiger Plush Toy are actually shadows, which could be mitigated by introducing a separate encoder for lighting. Currently, the effects of lighting are baked into the NeRF. \n\n__Q. Editability of YouDream__\n\nYouDream is not animatable as of now. As part of future work, we plan to take inspiration from DreamWaltz [1] which uses a parametric 3D model for representing humans: SMPL [2] to map the LBS weights onto the NeRF. In a similar fashion, we can achieve this using the parametric 3D model for animals: SMAL [3], but it will be limited to the small number of animals represented by SMAL.\n\n__Q. Quantitative/objective evaluations for text-to-3D methods__\n\nWe agree there is no definite quantitative evaluation protocol for such methods, despite that we found it important to report the CLIP scores in Sec. F. \n\n__Q. Rephrasing of line 7-8__\n\nIn this line, we are referring to imaginary animals that cannot be created without explicit pose control. We will update this sentence to “Our method is capable of generating novel imaginary animals which previous text-to-3D generative methods are unable to create.” to make it clearer.\n\n__Q. Flickering in turntable videos__\n\nWe were able to completely eliminate the flickering issues by increasing the sampling density of NeRF rendering without any re-training for all assets. Thank you for pointing this out.\n\n__Q. Generating severely out-of-domain assets such as fishes and insects__\n\nWe thank the reviewer for this excellent suggestion. We have generated \"a clownfish” and \"a four-legged tarantula” using our fully automatic pipeline and present the results in Fig.(b) of rebuttal pdf. Since the multi-agent LLM currently has been designed to generate animals with four limbs we choose to generate \"a four-legged tarantula”. For the \"clownfish” the LLM removed all limb keypoints except the keypoints for ‘thighs’ and ‘shoulders’.\n\n\n[1] Huang et.al, Dreamwaltz: Make a scene with complex 3d animatable avatars, NeurIPS’24.\n\n[2] Loper et.al, SMPL: A skinned multi-person linear model, ACM Trans. Graph. (SIGGRAPH Asia)‘23\n\n[3] Zuffi et.al, 3D Menagerie: Modeling the 3D Shape and Pose of Animals, CVPR’17", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a4VfVF6QT3", "reviewer": "Reviewer_4gaH", "summary": "1. This paper proposes to generate animals with pose control. \n2. The pose control is achieved by controlnet trained on pose-image pair data. \n3. The pose can be generated with LLM during inference.", "questions": "See the weakness above.", "limitations": "See the weakness above.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The pipeline consists many parts and some efforts are put to implement the whole pipeline.\n2. Experiments show robustness in animal generation compared to previous optimization-based methods.\n3. Introducing LLM into 3D generation is novel.", "weaknesses": "1. The description of how you implement LLM to generate poses in Sec. 3.2 is too vague:\n* Which LLM do you use? Did you fine-tuned the LLM on pose data?\n* How do you represent the pose in natural language to feed into the LLM? \n* Do you represent the pose with discrete tokens or continuous real number?  \n* How do you implement the 3 agents (obsever, modifier, finder)? \n\nMany details above are lack. These details are very important for the readers to understant how the method is implemented and evaluate the technical soundness of this work. I think the authors may provide these details in rebuttal and in the further revision of this manuscript.  Currently, I suggest a reject owing to the presentation. \n\n2. No results on the failure cases are reported. No details about the success rate of generation are reported. I suggest the authors to report a metric of the success rate of generation. Sometimes the generation results consist of much artifact and are considered as failure cases. As for success rate, I suggest the authors to present the portion of generation results that are not failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_We appreciate the reviewer for acknowledging the novelty of our work and the robustness of our method. We sincerely thank the reviewer for pointing out two very important points that should be included in the manuscript. We will make sure to update the same in the final version of the manuscript. If our response has adequately addressed the reviewer's concerns and provided new insights, we would be grateful if they consider revising their score._\n\n__Q. Clarifications of the multi-agent LLM setup__\n\nWe use GPT-4o out of the box. We mention this in Sec. E and will improve the language in Sec. 3.2 for better clarification. We have provided the code for the pose-generator setup in the Supplementary ZIP file GPT_kpt_maker.py as mentioned in Sec. E. It includes all details about the LLM prompts used for each agent. We store the pose keypoints as a JSON dictionary and add the contents of the file to the prompt in string format which is then tokenized by the OpenAI API. We also provide the bone sequence, which denotes the connections between the keypoints to the observer GPT.  The modifier GPT finally outputs the updated keypoints in the previously supplied JSON format. We will add these details along with the prompts in the manuscript to make it convenient for the reader. \n\n__Q. Discussion on failure cases of YouDream__\n\nWhile we mention the majority of the limitations of our method in Supplementary Sec. I, we can include the failure case examples – “a five-headed snake”, “an orange-colored crocodile”, and “an echidna”. For “a five-headed snake” the model was unable to make a coiled-up body due to pose control consisting only of neck_end and back_end points on the torso of the animal. For “an orange-colored crocodile” the model converged towards making a green-colored crocodile. And for “an echidna” the model was unable to create well-defined spines. As we could not generate these 3 out of the 45 different animals we tried, this suggests a success rate of 93.33%. We did not include this fact in the paper because success rate computation could be highly erroneous considering these two factors: \n\na) Various animals have quite similar body shapes and appearance, such as a zebra, a galloping horse, etc which are quite similar to a horse. Since we provide the generated asset for a horse, we chose to avoid sharing these similar kinds of generations. Such examples would automatically lift the success rate. \n\nb) For generating unseen animals we would need more creative ideas to test the boundaries of our method which could only happen when the open-source community uses our method and shares their observations. Within the limited creative ability of the authors, we were able to generate 9 out of the 10 prompts we tried.\n\nWe will add a discussion on these failure cases in our limitations section. Thank you for the helpful suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
_We sincerely thank the reviewers for their comments. Reviewers __127z__, __rnqi__, and __4gaH__ appreciated the __novelty__ of YouDream and the improvement in performance in terms of __geometric consistency__ and __robustness__, while reviewer __PBQU__ identified __YouDream’s out-of-domain generation capability__. Reviewer __rnqi__ recognized YouDream’s potential for __aiding 3D artists__ through pose controllable 3D generation and the use of LLM in pose creation._ 

Here we clarify the significance of YouDream further and answer a common question. 

YouDream aims to assist creators in two main ways:

__a) Fine-Grained User-Guided 3D Generation__: This grants the user more control to modify existing poses or to create poses for novel unseen animals. It is ideal for scenarios where the artist wishes to generate __imaginary animals__ or requires __precise pose__ adjustments. All poses for unseen animals depicted in Fig. 1 were manually created. The Tiger sequence was also manually designed, as such detailed pose changes cannot be described through text to an LLM which is not specifically fine-tuned to do so. We named the method YouDream to emphasize its ability to create anything a 3D artist can envision. We remind the reviewers that these novel innovative assets are impossible to create without YouDream as shown in Fig.1 of the main paper.

__b) Coarse-Grained LLM-Guided 3D Generation__: This is beneficial when users plan to generate poses for commonly seen animals without starting from scratch. It allows users to evaluate the model’s capabilities and is suitable for scenarios where manual pose editing is not preferred. Most of the poses for seen animals shown in the figures were generated by the multi-agent LLM.



__Details and scope of the Multi-agent LLM pose editor__

We have provided the complete code for the multi-agent LLM setup in GPT_kpt_maker.py included in the main submission Supplementary zip file. We use the GPT-4o API by OpenAI to implement all the agents. The keypoints are represented as a dictionary in JSON format and converted to a string to be appended to the text prompts of the observer and modifier LLMs. The observer LLM is instructed using the system prompt about details of the 3D coordinate space and relations among the various keypoints. The prompts for all the agents can also be seen in the same file. The bone sequence which represents the connections between the various keypoints is also provided as a list to the observer GPT’s prompt in order to reason about relative anatomy based on bone lengths. Finally, the multi-agent LLM outputs a keypoint dictionary in the same format as provided to it. The Multi-agent LLM is able to generate various animals by taking reference from a set of 16 3D animal poses. However, this setup can generate poses for animals that are __well out-of-domain__ of these 16 animals as shown in Fig.(b) of the rebuttal pdf. The Multi-agent LLM supports generating animals that can be represented using four limbs. Generating more than four-limbed animals such as insects using the LLM setup is a direction of future work. We will add these details to our manuscript to describe the Multi-agent LLM setup further.

Meta-review: The paper proposes a 3D generation pipeline that synthesizes 3D (un)real animals. The 3D pose of the generated animal can be controlled by a multi-agent LLM. 

After the rebuttal stage, the reviewers reported the final ratings of 5, 5, 6, 5. Reviewers PBQU and 4gaH did not update their final score, but shared the scores in the discussion threads.

The reviewers agree that the algorithm proposed in this paper is novel, the results presented are impressive, particularly those out-of-domain generation. The AC agrees with the reviewers and accepts the paper.

In the revision, the authors are encouraged to
1. add the qualitative/quantitative results, particularly the comparisons to GaussianDreamer.
2. add the discussion in the rebuttal stage to the paper or appendix.
Decision: Accept (poster)