Title: value: YouDream: Generating Anatomically Controllable Consistent Text-to-3D Animals
Authors: value: Sandeep Mishra
Oindrila Saha
Alan Bovik
Venue: NeurIPS 2024
Keywords: value: 3D Animal generation
Text-to-3D
Diffusion models

Abstract: value: 3D generation guided by text-to-image diffusion models enables the creation of visually compelling assets. However previous methods explore generation based on image or text. The boundaries of creativity are limited by what can be expressed through words or the images that can be sourced. We present YouDream, a method to generate high-quality anatomically controllable animals. YouDream is guided using a text-to-image diffusion model controlled by 2D views of a 3D pose prior. Our method is capable of generating novel imaginary animals that previous text-to-3D generative methods are unable to create. Additionally, our method can preserve anatomic consistency in the generated animals, an area where prior approaches often struggle. Moreover, we design a fully automated pipeline for generating commonly observed animals. To circumvent the need for human intervention to create a 3D pose, we propose a multi-agent LLM that adapts poses from a limited library of animal 3D poses to represent the desired animal. A user study conducted on the outcomes of YouDream demonstrates the preference of the animal models generated by our method over others. Visualizations and code are available at https://youdream3d.github.io/.


Review discussion:
1. The pipeline consists many parts and some efforts are put to implement the whole pipeline.
2. Experiments show robustness in animal generation compared to previous optimization-based methods.
3. Introducing LLM into 3D generation is novel.
1. The description of how you implement LLM to generate poses in Sec. 3.2 is too vague:
* Which LLM do you use? Did you fine-tuned the LLM on pose data?
* How do you represent the pose in natural language to feed into the LLM?
* Do you represent the pose with discrete tokens or continuous real number?
* How do you implement the 3 agents (obsever, modifier, finder)?
Many details above are lack. These details are very important for the readers to understant how the method is implemented and evaluate the technical soundness of this work. I think the authors may provide these details in rebuttal and in the further revision of this manuscript.  Currently, I suggest a reject owing to the presentation.
2. No results on the failure cases are reported. No details about the success rate of generation are reported. I suggest the authors to report a metric of the success rate of generation. Sometimes the generation results consist of much artifact and are considered as failure cases.
As for success rate, I suggest the authors to present the portion of generation results that are not failure cases.
See the weakness above.
See the weakness above.
No ethics review needed.
_We appreciate the reviewer for acknowledging the novelty of our work and the robustness of our method. We sincerely thank the reviewer for pointing out two very important points that should be included in the manuscript. We will make sure to update the same in the final version of the manuscript. If our response has adequately addressed the reviewer's concerns and provided new insights, we would be grateful if they consider revising their score._
__Q. Clarifications of the multi-agent LLM setup__
We use GPT-4o out of the box. We mention this in Sec. E and will improve the language in Sec. 3.2 for better clarification. We have provided the code for the pose-generator setup in the Supplementary ZIP file GPT_kpt_maker.py as mentioned in Sec. E. It includes all details about the LLM prompts used for each agent. We store the pose keypoints as a JSON dictionary and add the contents of the file to the prompt in string format which is then tokenized by the OpenAI API. We also provide the bone sequence, which denotes the connections between the keypoints to the observer GPT.  The modifier GPT finally outputs the updated keypoints in the previously supplied JSON format. We will add these details along with the prompts in the manuscript to make it convenient for the reader.
__Q. Discussion on failure cases of YouDream__
While we mention the majority of the limitations of our method in Supplementary Sec. I, we can include the failure case examples – “a five-headed snake”, “an orange-colored crocodile”, and “an echidna”. For “a five-headed snake” the model was unable to make a coiled-up body due to pose control consisting only of neck_end and back_end points on the torso of the animal. For “an orange-colored crocodile” the model converged towards making a green-colored crocodile. And for “an echidna” the model was unable to create well-defined spines. As we could not generate these 3 out of the 45 different animals we tried, this suggests a success rate of 93.33%. We did not include this fact in the paper because success rate computation could be highly erroneous considering these two factors:
a) Various animals have quite similar body shapes and appearance, such as a zebra, a galloping horse, etc which are quite similar to a horse. Since we provide the generated asset for a horse, we chose to avoid sharing these similar kinds of generations. Such examples would automatically lift the success rate.
b) For generating unseen animals we would need more creative ideas to test the boundaries of our method which could only happen when the open-source community uses our method and shares their observations. Within the limited creative ability of the authors, we were able to generate 9 out of the 10 prompts we tried.
We will add a discussion on these failure cases in our limitations section. Thank you for the helpful suggestions.
Thank you for your rebuttal. My concerns have been partially resolved and I decide to keep my score. I hope more details about how the LLM is used should be put in the main text in final version of this work, as it is very important in your work.
Thank you for your response and for the time you've taken to review our work. We appreciate your feedback and are committed to addressing your concerns in the final version of the manuscript. We understand the importance of providing more details about the LLM usage and have already incorporated these updates as discussed.
We understand that your rating is currently at Borderline Reject, and we respect your judgment.
However, we would be grateful if you could reconsider your score in light of the clarifications and additional details we've provided. Your feedback is invaluable, and we believe that with these revisions, our work has significantly improved.
We are keen to ensure our manuscript meets the highest standards, and your support would be crucial in achieving this. If there are any other aspects you'd like us to address or clarify further, we would be more than happy to do so.
Thank you once again for your thoughtful review and consideration.
The presented approach makes sense, and shows promising results.
In particular, it has the following strengths:
- An automatic system that produces 3D poses/skeletons for various animals. This is a very creative way to exploit the LLM.
- The framework can produce 3D animal assets with more accurate 3D geometry, reducing the multi-face problem that plagues many SDS-based approaches.
- YouDream offers controllability over the animal poses, making it a potentially useful tool for 3D artists.
In summary, the paper presents a promising framework that tackles an interesting problem (3D animal generation) that can benefit 3D artists and beyond, and the novelty lies in how YouDream exploits the LLM to handle diverse, varying animal skeletons.
YouDream has some weaknesses
- Color saturation artifacts. This is a common problem among SDS-based approaches and could be fixed using implementation tricks from other work.
- While greatly alleviating the multi-face problem, we can still observe artifacts (e.g., multiple pairs of ears in Supp. raccoon_standing), and extra limbs (Figure 8, tiger plush toy). The problem could be mitigated with a more detailed skeleton (i.e., more landmarks/keypoints).
- Controllable, but not editable – YouDream has to re-run the optimization process to create the same 3D animal in a different pose.
- Similar to many other text-to-3D methods, no quantitative/objective evaluations available.
While the presented results look pretty good, we cannot assess/quantify the improvement over previous methods. This, however, is not a problem unique to YouDream.
Please clarify the following questions if possible:
- L7-8: It might be more appropriate to say “generates 3D animals with controllable poses”. As shown in Figure 1, Figure 4, and Figure 15, other methods can also generate 3D animals, sometimes with even more appealing textual qualities (i.e., Figure 15 MVDream Giraffe).
- There are flickerings in some of the supplementary videos across views. Since the underlying representation is a NeRF, I expect the 3D results to be view-consistent. It would be great if you could clarify if the flickerings are caused by compression or other issues.
- L169-172 mentioned that TetraPose ControlNet works well for out-of-domain animals, which is somewhat backed by the llama octopus example (Figure 1). It would be interesting to see more results on non-tetrapods, such as snakes, fishes, insects, etc. This is a bit out of the scope of this manuscript, but it would be great to see to what extent the proposed multi-agent 3D pose generator and TetraPose ControlNet work.
The limitations are addressed appropriately in the paper. Please answer/clarify the above-raised questions/weaknesses if possible.
No ethics review needed.
_We sincerely thank the reviewer for their in-depth analysis of our work and for providing extremely helpful comments and questions which will greatly help us to improve our manuscript. We deeply appreciate the reviewer's acknowledgment of the impact of our work and their recognition of YouDream’s novelty and superior results. If our response has adequately addressed their concerns and provided new insights, we would be grateful if they consider revising their score._
__Q.
About alleviating remaining artifacts__
For the rebuttal, we have been able to improve quality by training using a higher dimensional NeRF. We used a low-dimensional NeRF due to experimental resource constraints. We plan to experiment with improving texture and color quality using variants of SDS and other tricks in the future.
We agree with the reviewer’s observation: a more detailed skeleton will reduce artifacts around the ears, fingers, etc. The extra limbs in Tiger Plush Toy are actually shadows, which could be mitigated by introducing a separate encoder for lighting. Currently, the effects of lighting are baked into the NeRF.
__Q. Editability of YouDream__
YouDream is not animatable as of now. As part of future work, we plan to take inspiration from DreamWaltz [1] which uses a parametric 3D model for representing humans: SMPL [2] to map the LBS weights onto the NeRF. In a similar fashion, we can achieve this using the parametric 3D model for animals: SMAL [3], but it will be limited to the small number of animals represented by SMAL.
__Q. Quantitative/objective evaluations for text-to-3D methods__
We agree there is no definite quantitative evaluation protocol for such methods, despite that we found it important to report the CLIP scores in Sec. F.
__Q. Rephrasing of line 7-8__
In this line, we are referring to imaginary animals that cannot be created without explicit pose control. We will update this sentence to “Our method is capable of generating novel imaginary animals which previous text-to-3D generative methods are unable to create.” to make it clearer.
__Q. Flickering in turntable videos__
We were able to completely eliminate the flickering issues by increasing the sampling density of NeRF rendering without any re-training for all assets. Thank you for pointing this out.
__Q. Generating severely out-of-domain assets such as fishes and insects__
We thank the reviewer for this excellent suggestion. We have generated "a clownfish” and "a four-legged tarantula” using our fully automatic pipeline and present the results in Fig.(b) of rebuttal pdf. Since the multi-agent LLM currently has been designed to generate animals with four limbs we choose to generate "a four-legged tarantula”. For the "clownfish” the LLM removed all limb keypoints except the keypoints for ‘thighs’ and ‘shoulders’.
[1] Huang et.al, Dreamwaltz: Make a scene with complex 3d animatable avatars, NeurIPS’24.
[2] Loper et.al, SMPL: A skinned multi-person linear model, ACM Trans. Graph. (SIGGRAPH Asia)‘23
[3] Zuffi et.al, 3D Menagerie: Modeling the 3D Shape and Pose of Animals, CVPR’17
Thanks for the comprehensive responses. My concerns are all addressed, and I would love to know other reviewers' opinions after reading the rebuttal. For now, my feedback regarding YouDream remains positive.
Thank you for your continued positive feedback on YouDream. We're glad our responses addressed your concerns comprehensively. Your suggestions were indeed highly valuable in helping us demonstrate YouDream's effectiveness more clearly. We appreciate your thoughtful engagement with our work and look forward to seeing other reviewers' perspectives as well.
Best,
Authors
1. The method enables the generation of 3D tetrapod animals with diverse poses.
2. The method can generate novel animals that are out of the domain of training data.
3. The pose-guided generation enables more fine-grained pose control for the text-to-3D generation.
1. The work allows more explicit pose control for generating tetrapod animals.
However, original text-to-3D models can achieve this using only text, without limitations on object classes or the need for human labor to design poses.
2. The poses appear to be user-designed, but I still doubt the difficulty of creating novel poses with the given library, like it's hard to create the tiger walking sequences (Fig. 21) given the library, are the poses generated from Multi-agent LLM or designed manually.
3. The effectiveness of the LLM-based pose editor seems limited; how can a 2D-based model (LLM) effectively reason about the 3D poses? The given initial poses in Fig.3 are already plausible. What if more noisy poses are given?
4. The generated model is of bad quality and not comparable with MVDream, although it has a 3D GT. I doubt whether adding the explicit pose control will make the mesh quality worse.
What are the finder, observer, and modifier composed of, respectively? What modifications can the modifier make to the given 3D poses? For the text prompt “A giraffe with dragon wings,” are the poses automatically generated by the modifier?
None
No ethics review needed.
_We thank the reviewer for their detailed questions which will help us improve our manuscript. If our response has adequately addressed their concerns and provided new insights, we would be grateful if they consider revising their score._
__Q. Prior art can achieve explicit pose control using only text. YouDream is limited by object classes and human labor.__
We very strongly disagree with the reviewer’s opinion about prior models achieving generation of tetrapod animals using only text. We humbly request the reviewer to observe Fig. 1, 4, and 15 which highlight multiple examples where the prior text-to-3D models are unable to generate the desired outputs.
Although models trained on 3D data such as MVDream can generate commonly seen tetrapod animals without artifacts, they lack naturalness (Fig.15 main paper) due to their limited training data.
Despite being trained on 2D images, YouDream consistently generates realistic, geometrically consistent, and faithful-to-text assets unlike any previous method.
Moreover, YouDream can easily enable the generation of novel unseen animals making it highly useful for the VFX industry. Since such unseen animals are not part of existing ML datasets, forcing models trained on these to create view-consistent images of such unseen animals is nearly impossible without an external control such as a 3D pose. For the same reasons, an LLM cannot generate poses for such unseen animals. Inadvertently this introduces the need for human effort to design poses for creative imaginary animals. For animals that are representable by prior methods, YouDream employs the multi-agent LLM for generating the pose.
__Q. Is tiger walking sequence generated by Multi-agent LLM?__
Our automatic pipeline can cater to animals well represented in existing datasets that LLMs or diffusion models have been trained on. Unseen animals or fine-grained pose designs as required to create a walking sequence cannot be created using an LLM based on purely textual description unless specifically trained to do so using a dedicated dataset. We generate the walking seq. manually only to illustrate fine-grained pose controllability.
Although the Multi-agent LLM can create a generic walking pose, creating a sequence of poses remains as future work.
__Q. How can a “2D-based model (LLM)” effectively reason about the 3D poses? What if noisy poses are given to the LLM?__
Please note that the LLM is purely text-based and operates in a multi-dimensional space. LLMs have previously demonstrated the ability to understand and reason about spatial relationships without any fine-tuning. Llava[1] shows that GPT can understand and describe spatial orientation in images using numerical bounding box coordinates in text prompts. LLM-GROP[2] shows GPT's capability for spatial understanding and object rearrangement in 3D space. LLMs also possess valuable relative knowledge of various classes, including animals and birds, useful for improving zero-shot classification[3]. We believe all these capabilities which evidence the logical and relational reasoning powers of LLMs help GPT understand and manipulate 3D poses.
Direct pose generation by LLM without input-pose reference was mostly unsuccessful and required case-by-case prompt adjustments. Comparison between single-agent and multi-agent LLM results is shown in Fig. 9 of the main paper. These details will be added to the manuscript. Based on your suggestion, we added random noise to our keypoints within a 20% margin. The multi-agent LLM setup was able to remove noise from some keypoints, particularly on the limbs and tails, but struggled with the points on the head. Our library of 16 poses, designed to cover a wide range of animal shapes and poses, has proven effective in generating a vast range of animals presented in the paper and supplementary videos.
Additionally, we demonstrated the LLM's ability to generate keypoints for a 'clownfish' and a 'four-legged tarantula' in Fig. (b) of the rebuttal PDF.
__Q. The generated model is of bad quality and not comparable with MVDream, although it has a 3D GT.__
We disagree with this claim. There are primarily two reasons to do so:
a) Quality is a highly subjective aspect and opinions vary from person to person.
For this reason, we do not make any objective claims about the overall quality of our method. Our user study presented in Sec. 4 is a representative opinion of multiple user preferences averaged over multiple assets.
b) In the generative 3D domain, quality encompasses factors like naturalness, text alignment, and geometric artifacts.
While we acknowledge that MVDream has better material quality due to training on human-designed 3D assets under various rendering conditions, this doesn't mean it has better overall quality. Our user study, shown in Fig. 5, includes preferences from 32 participants for assets generated using 22 diverse prompts, demonstrating the superiority of our method in terms of naturalness and T2I alignment.
__Q. I doubt whether adding the explicit pose control will make the mesh quality worse.__
As noted by the reviewer, adding explicit pose control does not worsen the quality of the mesh.
__Q. Details about the Multi-agent LLM and its scope__
All agents used are GPT-4o instances with well-defined system prompts which can be seen in the GPT_kpt_maker.py file.
Although the Modifier LLM has not been explicitly instructed to make any specific kinds of modifications we have observed that it can shift, scale, and rotate keypoints w.r.t other keypoints.
For all assets in Fig. 1, the poses have been manually generated. This is noted as “artist's creative control” in the figure caption. We will clarify this further in the manuscript.
[1] Liu et.al, Visual instruction tuning. NeurIPS’24.
[2] Ding et.al, Task and motion planning with large language models for object rearrangement. IROS’23
[3] Menon et.al, Visual Classification via Description from Large Language Models. ICLR’23
Dear R-PBQU,
Could you take a look at the authors' responses, and share your thoughts?
Thanks,
Your AC
The strengths of the proposed paper can be summarized as:
+ The paper proposes a multi-agent LLM to generate 3D pose of animals, which is novel and interesting for 3D generation.
+ The results are impressive, showcasing better performance than existing techniques.
The weakness of the proposed paper can be summarized as:
- Missing comparisons with GaussianDreamer, LucidDreamer, etc, which are more recent works than 3DFuse, Fantasia3D, and HiFA.
Meanwhile, it would also be better to compare with DreamFusion, Magic3D, and prolificdreamer to further demonstrate.
- The ablation in Fig. 6 is not positive. Actually: it seems that the initial shape provides more contributions, while the pose control doesn't affect much.
- It would also be good to show quantitative evaluations using CLIP-based metrics.
- The methods can only process 16 categories of animals.
Although it's a limitation of current datasets, it will still hurt the method's performance and bring the question of whether we need the technique specifically designed for these animals.
Please see weakness listed above.
Limitation has been discussed in the Appendix.
No ethics review needed.
_We appreciate the reviewers' thorough evaluation of our work, including recognition of our novel contributions and results. We thank them for their insightful suggestions to enhance our paper. If our response has adequately addressed their concerns and provided new insights, we would be grateful if they consider revising their score._
__Q1. Comparison with more prior art__
We thank the reviewer for the suggestion. We added comparisons with LucidDreamer, Stable DreamFusion, and ProlificDreamer in Fig.(a) of the rebuttal pdf and report the same inconsistencies. Among prior art, LucidDreamer produced the best results but was plagued with inconsistencies such as 6 legs, 2 tails and 2 trunks for an elephant, while also showing unnatural colors. For the novel unseen animal ``Llama with octopus tentacles body” no prior method faithfully captures the text.
__Q2. Fig.6 shows pose control does not affect generation__
Please note that Fig.6 shows the side and back view of the assets. Not using pose control results in a head forming on the backside of the elephant and hence __proves the efficacy__ of the pose control.
__Q3. Quantitative evaluation using CLIP__
Please refer to Table.1 in Supplementary Sec. F where we compare CLIP similarity score with the best performing prior art: MVDream.
__Q4. YouDream can only generate 16 categories of animals__
The 16 animal 3D pose library is meant to act as support for our multi-agent LLM and does not limit generation to only these categories. We demonstrate the same by generating fishes and insects using only these as support (see Fig.(b) rebuttal pdf). Our trained ControlNet can generate animals significantly out-of-domain, despite it being trained on a tetrapod animal database. This is evident from Fig. 1 (main paper)  where we show that YouDream can generate animals significantly out-of-domain of the Tetrapose ControlNet training data.
Thanks for the rebuttal from the authors.
Q1 -> I am not sure if the authors put the correct caption for results in Fig. (a). If it's correct, LucidDreamer is much better than the proposed methods.
Meanwhile, GaussianDreamer has not been compared, and no reason regarding this has been provided.
Q3 -> Only MVDream has been compared. No reason has been provided why the authors didn't compare with other methods, like LucidDreamer, GaussianDreamer, and so on.
I am learning towards the negative end after the rebuttal.
Thank you for taking the time to proactively participate in the author-reviewer discussion. We would like to clarify your concerns raised in the latest comment.
__Q1__ -> Since we are unable to provide turn-table videos (which would have been highly efficient in showcasing the anatomic inconsistencies) in the rebuttal, we would humbly request the reviewer to observe the generated outputs of LucidDreamer in Fig. (a). It can be noted that in the front-view, __the elephant's tusk is visible in between the front legs and it goes all the way down to the bottom of the legs.
While in the side-view the trunk is shown as raised in the air. In the front-view the elephant's front legs are shown as straight and touching the ground, but in the side-view, upon zooming you can identify that there is a bent leg and two straight legs on the front side of the animal. The generated asset also has two tails that are visible upon closer inspection in the side-view. We would also like to highlight that it is highly green.__ For the “llama with octopus tentacles body”, __LucidDreamer generates a full-body llama with a single tentacle around the head__, but with YouDream we were able to explicitly guide the generation of an animal with the llama head and tentacles as legs, where even the number of such tentacle-legs is controlled accurately using the 3d pose input.
We would also like to discuss other inconsistencies we observed in other assets we generated using LucidDreamer which we could not include due to space, such as “a giraffe, full body” only produced the head and neck of the giraffe, and “a three-headed dragon, full body” had a dragon with a single head with wings protruding from the head.
Due to time constraints of the rebuttal period, we were able to test 3 new methods – Stable Dreamfusion, ProlificDreamer, and LucidDreamer and were not able to test additional methods such as GaussianDreamer, Magic123.
However, since GaussianDreamer does not provide any explicit control/signal to avoid the generation of anatomical inconsistencies, such as multiple legs, heads, or tails, we believe it will contain similar issues as shown in LucidDreamer, HiFA, ProlificDreamer, etc. which are guided by text-to-image diffusion models.
We do agree that the visual fidelity of some of the prior methods, such as MVDream and LucidDreamer (uses SD 2.1 while YouDream uses SD1.5), is higher owing to certain factors such as 3D training data, higher NeRF dimensions, higher SD versions, and various other tricks shown in the prior art.
Hence, __in this paper, we focus on improving the anatomical consistencies which can be very easily alleviated by the pose conditioning offered by YouDream__. To showcase the impact of such tricks on visual fidelity we also added results using slightly higher dimension NeRF in Fig. (c) (due to resource constraints we could only increase the NeRF dimensions by 2x).
__Q2__ -> During the rebuttal period, we were able to produce 10 assets of LucidDreamer – “a tiger, full body”, “a giraffe, full body”, “a three-headed dragon, full body”, “a pangolin, full body”, “an elephant, full body”, “a llama with octopus tentacles body, full body”, “a giraffe with dragon wings, full body”, “a realistic mythical bird with two pairs of wings and two long thin lion-like tails, full body”, “a red male northern cardinal flying with wings spread out, full body”, and “a Tyrannosaurus rex, full body”. Only using these assets for a CLIP-score-based evaluation of LucidDreamer, MVDream, and YouDream, we obtained the following scores:
_LucidDreamer_ |  _MVDream_ |  _YouDream_ |
These numbers are using the same setting as described in the paper. Please note that in the paper we compare using 22 generated assets. We will extend this evaluation to all 22 assets for our final version.
We would also like to justify the reason behind not comparing CLIP-Score with prior text-to-image-based 3D generators and only comparing with MVDream, which was trained on 3D data. Since the core contribution of the paper is to enable the generation of anatomically consistent animals, we found that every text-to-image-based model produced some or the other geometrical artifacts resulting in inconsistent 3D animals.
Hence we only compare with MVDream which can recreate the geometric consistencies of natural animals (but fails for unseen animals) thanks to its 3D (multi-view consistent) pre-training.
- Thanks. I see the difference between LucidDreamer and YouDream now. LucidDreamer indeed has some issues as the authors describe.
However, the quality of YouDream is not as good as LucidDreamer. I would suggest the authors to further improve the performance.
- I still have a concern about the comparison with GaussianDreamer.
We thank you sincerely for accepting our request for re-checking and identifying the issues. The results in our main paper are indeed less sharp compared to LucidDreamer. On re-generating the assets with all settings the same, except increasing the NeRF dimension to 256x256x256 (__please note Fig.(c) of rebuttal PDF__ for the prompt “a tiger.”) we were able to achieve much sharper, more detailed (observe the face), cleaner, and flicker-free results. We will update the results of all our assets using the larger NeRF dimension in the final version of the manuscript.
We were able to run GaussianDreamer for only one asset since yesterday, which was “an elephant” and found similar problems with it, unfortunately, there is no way for us to share that result here.
However, we found another concurrent work __[1]__ that showcases the results of GaussianDreamer in __Fig. 11 (page 21, Supplementary section - DreamPolisher)__. They use the prompt: “A DSLR photo of a red panda”, where __GaussianDreamer fails to generate a tail for the 3D asset of the animal__, which is indeed very easily generatable using YouDream since our 3D pose includes a tail keypoint, and our tetrapose-ControlNet makes sure to include it in the generation process. We will also add the comparisons with both LucidDreamer and GaussianDreamer in our final version for better benchmarking. The same figure also showcases LucidDreamer's generated output which has the face of the animal visible in both views and hence has the multi-head problem.
We once again would like to thank the reviewer wholeheartedly for devoting their time proactively towards discussions and helping us improve the paper.
__[1]__ Lin, Y., Clark, R. and Torr, P., 2024. __Dreampolisher: Towards high-quality text-to-3d generation via geometric diffusion.__ arXiv preprint arXiv:2403.17237.
_We sincerely thank the reviewers for their comments. Reviewers __127z__, __rnqi__, and __4gaH__ appreciated the __novelty__ of YouDream and the improvement in performance in terms of __geometric consistency__ and __robustness__, while reviewer __PBQU__ identified __YouDream’s out-of-domain generation capability__. Reviewer __rnqi__ recognized YouDream’s potential for __aiding 3D artists__ through pose controllable 3D generation and the use of LLM in pose creation._
Here we clarify the significance of YouDream further and answer a common question.
YouDream aims to assist creators in two main ways:
__a) Fine-Grained User-Guided 3D Generation__: This grants the user more control to modify existing poses or to create poses for novel unseen animals. It is ideal for scenarios where the artist wishes to generate __imaginary animals__ or requires __precise pose__ adjustments. All poses for unseen animals depicted in Fig. 1 were manually created. The Tiger sequence was also manually designed, as such detailed pose changes cannot be described through text to an LLM which is not specifically fine-tuned to do so. We named the method YouDream to emphasize its ability to create anything a 3D artist can envision. We remind the reviewers that these novel innovative assets are impossible to create without YouDream as shown in Fig.1 of the main paper.
__b) Coarse-Grained LLM-Guided 3D Generation__: This is beneficial when users plan to generate poses for commonly seen animals without starting from scratch. It allows users to evaluate the model’s capabilities and is suitable for scenarios where manual pose editing is not preferred. Most of the poses for seen animals shown in the figures were generated by the multi-agent LLM.
__Details and scope of the Multi-agent LLM pose editor__
We have provided the complete code for the multi-agent LLM setup in GPT_kpt_maker.py included in the main submission Supplementary zip file. We use the GPT-4o API by OpenAI to implement all the agents. The keypoints are represented as a dictionary in JSON format and converted to a string to be appended to the text prompts of the observer and modifier LLMs. The observer LLM is instructed using the system prompt about details of the 3D coordinate space and relations among the various keypoints. The prompts for all the agents can also be seen in the same file. The bone sequence which represents the connections between the various keypoints is also provided as a list to the observer GPT’s prompt in order to reason about relative anatomy based on bone lengths.
Finally, the multi-agent LLM outputs a keypoint dictionary in the same format as provided to it. The Multi-agent LLM is able to generate various animals by taking reference from a set of 16 3D animal poses.
However, this setup can generate poses for animals that are __well out-of-domain__ of these 16 animals as shown in Fig.(b) of the rebuttal pdf. The Multi-agent LLM supports generating animals that can be represented using four limbs. Generating more than four-limbed animals such as insects using the LLM setup is a direction of future work. We will add these details to our manuscript to describe the Multi-agent LLM setup further.

Meta-review:
role: PC
data: comment: value: The paper proposes a 3D generation pipeline that synthesizes 3D (un)real animals. The 3D pose of the generated animal can be controlled by a multi-agent LLM. 

After the rebuttal stage, the reviewers reported the final ratings of 5, 5, 6, 5. Reviewers PBQU and 4gaH did not update their final score, but shared the scores in the discussion threads.

The reviewers agree that the algorithm proposed in this paper is novel, the results presented are impressive, particularly those out-of-domain generation. The AC agrees with the reviewers and accepts the paper.

In the revision, the authors are encouraged to
1. add the qualitative/quantitative results, particularly the comparisons to GaussianDreamer.
2. add the discussion in the rebuttal stage to the paper or appendix.

Decision: Accept (poster)