Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabrication

Authors: yunuo  chen
tianyi  xie
zeshun  zong
xuan  li
feng  gao
yin  yang
ying nian wu
chenfanfu  jiang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 05:58:28

Abstract: Existing diffusion-based text-to-3D generation methods primarily focus on producing visually realistic shapes and appearances, often neglecting the physical constraints necessary for downstream tasks. Generated models frequently fail to maintain balance when placed in physics-based simulations or 3D printed. This balance is crucial for satisfying user design intentions in interactive gaming, embodied AI, and robotics, where stable models are needed for reliable interaction. Additionally, stable models ensure that 3D-printed objects, such as figurines for home decoration, can stand on their own without requiring additional supports. To fill this gap, we introduce Atlas3D, an automatic and easy-to-implement method that enhances existing Score Distillation Sampling (SDS)-based text-to-3D tools. Atlas3D ensures the generation of self-supporting 3D models that adhere to physical laws of stability under gravity, contact, and friction. Our approach combines a novel differentiable simulation-based loss function with physically inspired regularization, serving as either a refinement or a post-processing module for existing frameworks. We verify Atlas3D's efficacy through extensive generation tasks and validate the resulting 3D models in both simulated and real-world environments.

Main paper: 
 introduction: 
 Generating high-quality 3D content is of great importance in modern visual computing. Realistic 3D models are highly sought after in computer graphics, while robust real 3D assets are gaining attention in training embodied AI. Nevertheless, the standability of 3D models -the ability to stand steadily without additional support -is often neglected. Real-world man-made objects such as action figures, toys, and furniture inherently possess some degree of geometric stability, allowing them to be safely placed on the ground. Although one usually takes such standability for granted, existing generative models fail to produce steady 3D assets due to their lack of physical perception; see Fig. 1 .Incorporating this stability expectation into 3D generation will significantly reduce the human effort required for tasks such as sorting out unqualified meshes, post-processing geometries, or adding external supports before actually using the 3D asset in any simulator or the real world. Furthermore, creating physically plausible 3D content will enhance the fidelity of simulations and policy training with these objects, potentially narrowing the sim-to-real gap and empowering embodied AI in robotic tasks. Towards this goal, we develop a 3D generation framework that can produce high-quality models adhering to basic physical laws, such as gravity, stability, and frictional contact.Several attempts have been made to incorporate physical constraints into 3D generation. Yang et al. utilized the spatial and physical sense of LLM to design floor plans and furniture arrangements [98] . PhyScene introduced physical guidance, such as collision and reachability constraints, to diffusion models to generate furniture layouts [97] . However, both works primarily consider straightforward spatial constraints, such as non-collision, and fail to incorporate more complex physics. Mezghanni et al. proposed a GAN-based network to generate physically-aware geometries by training a neural stability predictor using datasets labeled by Bullet [48] . Another GAN-based work by Wang et al. employed CFD software to compute vehicle drag coefficients, guiding the generation of streamlined vehicle meshes [85] . Such indirect incorporation of physical simulations, however, results in suboptimal efficiency and accuracy. Furthermore, due to the low expressibility of the backbone latent representation, the versatility of the generated results is very limited compared to current state-of-the-art diffusion-based models, as the results are typically confined to specific categories (e.g., furniture and vehicles). Most recently, Ni et al. bridged differentiable physical simulation with differentiable rendering to obtain virtual 3D reconstructions from real-world images that are physically plausible in simulators. Their work primarily focused on simple four-leg-supported objects such as tables and chairs [54] . Moreover, the evaluations of all aforementioned works are conducted in virtual simulators, leaving their performance in the real world untested. This limitation hinders potential downstream applications such as industrial manufacturing and robotic manipulation.Since the pioneering work DreamFusion [58] , Score Distillation Sampling (SDS) has demonstrated efficacy in elevating 2D content to 3D, inspiring numerous follow-up studies [8; 35; 46; 69; 88; 109] . These advancements have enhanced both the versatility of generated content and the quality of textures. However, none have addressed the crucial issue of physical stability. On the other hand, traditional computational fabrication has concentrated on employing topology and shape optimization to ensure that 3D printed objects can stand in a balanced state [59] . Directly integrating these methods with 3D generative AI as a postprocessing module is suboptimal. Shape optimization disregards the original input conditions of diffusion models, while topology optimization produces internal structures that defy intuitive physics, rendering them unsuitable for training embodied AI systems designed to emulate human-like reasoning about physical objects.Observing this gap, we introduce Atlas3D, a generation pipeline that produces physically plausible, self-supporting 3D models from text. Incorporating differentiable physics-based simulation into our process, we generate models that are both simulation-and fabrication-ready. That is, they can be directly utilized in physical simulators, or 3D-printed for real-world applications; see Fig. 1 and Fig. 2 . As our method is orthogonal to previous SDS-based techniques, which focus on non-physical qualities, it can be seamlessly integrated into many existing generation frameworks, functioning either as part of the refinement stage in a multi-stage method or as a post-processing step in a single-stage method. We demonstrate the efficacy of Atlas3D by comparing the stability of our models with those produced by existing methods. Validation examples reveal that our generated models can be deployed as virtual simulation assets. Their stability transfers directly to the real world, as evidenced by our 3D printed results, suggesting further applications in robot training. 
related work: 
 Diffusion-based 3D Generation Due to the abundance of information encoded in large image latent diffusion models (LDMs) [65] , extensive studies have used pre-trained LDMs to distill 3D content. One approach is to fine-tune LDMs to support novel view synthesis, with a separate multiview fusion step to produce 3D content [39; 68; 99; 38; 37; 89; 95; 42; 80; 25; 106; 41; 9; 83] . Another approach, which is more related to our paper, is using LDMs as likelihood discriminators. A differentiable renderer is connected to a 3D representation, and the LDMs guide the optimization of the representation parameters. [58] proposed Score Distillation Sampling (SDS). Efficiency has been improved by coarse-to-fine strategies [35; 60; 79; 8] and timestep scheduling [18; 100] . 3D priors are involved to improve multiview consistency [108; 66; 31; 1; 93; 36; 78] . Multiview diffusions can also be used to evaluate SDS [69; 87; 105; 94] . Other researchers have explored SDS variants or improvements [84; 77; 88; 24; 101; 16; 109; 96] . 3D LDMs that directly generate 3D representations are also explored, such as compositional scenes [17; 57] , point clouds [43; 81; 51] , SDFs [70; 107; 30] , occupancy fields [15; 72; 45; 12] and NeRFs [3; 11; 52; 6; 55; 5; 75; 23] .Physics-aware 3D Generation Most existing 3D generative models focus only on geometry or appearance modeling, with physics priors being underexplored. Time-independent physical constraints, such as penetrations, can be directly defined by penalties [97; 17; 82; 40; 102] . For time-dependent physical qualities, such as stability and comfort, data-driven quality checkers trained with offline simulators can be applied [10; 48; 4] . Offline simulators can also be used as validators to augment the training dataset [71] and update the design with reinforcement learning [85] , and as dynamics generators for generated 3D assets [91; 21; 62] . Another direction is to utilize differentiable simulations, which have be widely used in tasks like robotic control [74; 19; 61; 34; 76] or inverse problems [33; 20; 103; 53; 32; 73; 7; 104] . They can also be applied in 3D content generation to define physics-based losses to aid per-instance generation [54; 92] or model training [47] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations of the work are addressed in the "Discussion" section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
score distillation sampling: 
 SDS-based methods are shown to be effective in distilling 3D models from 2D images. They utilize a 3D representation such as implicit density field, implicit Signed Distance Field (SDF), or tetrahedral SDF [67] , a differentiable renderer like NeRF [50] , NeuS [86] , or Nvdiffrast [26] , and a pre-trained text-to-image model such as Stable Diffusion [64] serving as diffusion guidance. The generation process involves optimizing the parameters θ of the underlying 3D representation, where the 3D shape is differentiably rendered to 2D images z = g(θ) and compared against the real distribution from the diffusion model with text guidance y:EQUATIONwhere z t is the noisy image at noise level t, w(t) is a weighting function, and εϕ is the predicted noise. We refer readers to the Appendix for more technical details on SDS optimization.To incorporate physics into our framework, we propose predicting the dynamics of the generated 3D models using a differentiable simulator, where all objects are treated as rigid bodies. We follow the conventions in [2] to define the dynamical states of the simulation.The kinematics of a rigid body are described by its mass M and body-space inertia tensor I body , which remains constant. Assuming the center of mass of the body initially lies at the origin, the physical state Ψ of the body at time t (not to be confused with the noise level in diffusion) includes position T(t) and orientation R(t) (spatial information), and its linear and angular momentum P(t) and L(t) (velocity information). The rigid body equations of motion are given by EQUATIONwhere F(t) and τ (t) are the total force and torque exerted on the body, v(t) = P(t) M is the linear velocity, ω(t) = I(t) -1 L(t) is the angular velocity, I(t) = R(t)I body R(t) T is the world-space inertia tensor, and * denotes cross product of ω with the columns of R. The physical state at a later time can be derived via time integration: Ψ(t) = Ψ(0) + t 0 dΨ ds (s)ds, which can be solved by numerical methods. By optimizing the physical states together with the SDS loss, we can jointly refine both the 3D geometry and the physical attributes of the generated results.We provide a zoomed-in view of the local mesh topology change in Fig. 17 to demonstrate how our method optimizes the object's geometry to make it standable.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: The authors provide code to reproduce results in the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
atlas3d algorithm: 
 We introduce Atlas3D, a plug-and-play algorithm for generating 3D models from text. Focusing on man-made objects such as action figures and toys, which generally do not deform, Atlas3D treats generated models as rigid bodies and incorporates physics-based guidance into the generation process.As mentioned in § 3.2, we predict the dynamic behavior of generated models by rigid body simulations. While various explicit or implicit representations of 3D shapes can be chosen in a generation network, we opt for triangular meshes in our framework as they facilitate frictional contact modeling and simplify kinematics computation. Given a triangle surface mesh representation X(θ), where θ is the implicit parameter, we integrate X into a rigid body represented by the dynamic state Ψ(t) = [T(t), R(t), P(t), L(t)] T , where the world-space location x of any point X on the body is x(t) = R(t)X + T(t). Assuming the 3D model is initially placed upright3 on the ground with the bottom point touching the surface, we define standability as:EQUATIONStandability intuitively indicates an equilibrium state where all external forces acting on the object are balanced, and the physical state remains unchanged over time. However, perfectly placing an object straight on the ground without initial velocity is impractical in the real world. For example, when manually placing a cube on a flat table, the bottom face is unlikely to be perfectly parallel to the table surface. A stable 3D model should recover its initial state under mild perturbations, such as minor shaking. This state is known as stable equilibrium. Motivated by this, we augment standability with stable equilibrium Ψ(t) defined asEQUATIONwhere ϵ 0 represents mild perturbations to the initial physical state. We first describe how to incorporate the standability criterion (Eq. 3) into the optimization process of 3D generation, and then explain how to further augment it with stable equilibrium (Eq. 4). Additionally, we introduce geometry regularization to enhance the smoothness of generated meshes.We utilize a differentiable rigid-body simulator to obtain the physical state Ψ(t). Assuming that the 3D model will eventually reach its steady state, ∃T ∈ R large enough such that ∀t > T, Ψ(t) = Ψ(T ).Let S denote a differentiable simulation function. We approximate Ψ(T ) via simulation:EQUATION)Here T is the simulation end time, µ captures material parameters such as density and friction coefficient, as well as simulator parameters such as time step and damping. We adopt the semiimplicit Euler time-integrator in Warp [44] for simulation.Assuming the initial translation, rotation, and velocity are all zero, the difference between Ψ(T ) and Ψ(0) arises only from discrepancies in spatial location, as the velocity at the final steady state is also zero. Therefore, we propose a standability loss to penalize rotational deviation due to instability:EQUATIONwhere R ∈ R 3×3 represents the rotation matrix. We disregard the translation T, as real-world instability mostly leads to rotational deviation from the initial state, such as falling to one side, while most translations, like falling due to gravity, are irrelevant to standability.With a differentiable simulator, the standability loss can be backpropagated to mesh vertex coordinates and then to the implicit parameter θ as dLstand dθ = dLstand dX dX dθ . In theory, any differentiable simulator is compatible with our framework. Although the standability loss directly penalizes the nonstandability of a 3D object, it can be slow to compute, especially when T is large and many time steps are required, creating a huge computational graph. Consequently, both the simulation itself and the backpropagation of gradients through the simulation trajectory are time-consuming. Additionally, standability does not necessarily imply stable equilibrium, which is crucial for real-world 3D objects such as action figures and toys. Without this property, an object remains unstable even if standability is achieved, known as unstable equilibrium. Unstable equilibrium means that when a disturbance force is applied, the object moves away from its original position instead of recovering. Fig. 3 visualizes the difference between stable and unstable equilibrium. In the absence of perturbation, geometries like the upside-down triangle may remain standable in a simulator but are clearly unstable in the real world. Thus, we augment standability with stable equilibrium (Eq. 4). One straightforward way to incorporate this property is to introduce initial perturbation ϵ 0 into the simulator. However, this would require many more simulations with various perturbations and subsequent loss backpropagation, which is extremely time-consuming.Inspired by the concept of a potential well [13] , we augment our optimization objective with a robust and efficient stable equilibrium loss L stable . Specifically, for an object to be robustly standable, it needs to reside at a local minimum of potential energy-specifically, gravitational potential energy-so that if perturbed, gravity will act as a restoring force that returns the object to its original state. For a rigid body, gravitational potential energy is determined by the height of its center of mass. Thus, for any object in a stable equilibrium state, the center of mass would rise if it is slightly perturbed. This leads to our formulation of the stable equilibrium loss L stable . Let x com denote the position of the center of mass of the underlying geometry and H(x) denote the distance of the point x to the ground, assuming the object's pivot point is at z = 0. The stable equilibrium loss is defined as:EQUATIONwhere P ϕ v represents the rotation of ϕ radian about axis [v T , 0] T . Mathematically, a local minimum of gravitational potential energy is reached if ∃ϕ 0 such that ∀ϕ ∈ (0, ϕ 0 ), L stable = 0. In practice, we fix the perturbation scale ϕ and uniformly sample 20 perturbation directions v in xy-plane.While standability loss L stand and stable equilibrium loss L stable provide a well-defined objective for robust standing, they may lead to distorted optimized meshes due to the high-dimensional searching space of implicit parameter θ without constraint. To constrain the optimization space and obtain smooth meshes, we add a normal consistency term that favors smooth solutions:EQUATION)where T is the set of the triangle pairs sharing a common pair with n i , n j being their normals respectively. This term maximizes the cosine similarity between neighboring surface triangle normals, leading to smoother meshes. Considering the bottom surfaces of most robust standing objects are flat, we apply the Laplacian loss to a subset B of vertices with a height lower than a threshold h b :EQUATIONwhere δ i = (LV) i ∈ R 3 calculates the differential coordinates of vertex i with L being the Laplacian matrix of the mesh graph and V representing mesh vertices. Intuitively, this loss term attempts to minimize the distance between vertex i and the average position of adjacent vertices.With the physically-inspired loss terms derived above, we now describe how to incorporate them into the text-to-3D framework. SDS-based methods and their variants start optimization with a random initialization of the implicit parameters, which initially have no knowledge of the model's geometry. Adding physical constraints at this early stage would be ineffective. Therefore, we propose a two-stage training strategy: the coarse stage and the refine stage. In the coarse stage, we generate a rough shape of the model using a text prompt. We can adopt any SDS-based generation framework as our baseline model, offering various choices of implicit representation and differentiable renderers. In the refine stage, we optimize the geometry with our physical constraints included. For this, we use a tetrahedral SDF representation [67] and employ Deep Marching Tetrahedra (DmTet) to differentiably convert the coarse geometry from implicit density or SDF as necessary. We utilize Nvdiffrast [26] as the differentiable renderer and Stable Diffusion v2.1 [64] for guidance. We propose the following loss function for the joint optimization of texture, geometry, and stability: (10) In practice, we observe that adding standability loss once every 10 iterations is sufficient to ensure a significant reduction in loss without notably increasing computational overhead.L = λ SDS L SDS + λ stand L stand + λ stable L stable + λ normal L normal + λ b-lap L b-lap 
experiments: 
 In this section, we devise comprehensive experiments (both virtual and real-world) to demonstrate the efficacy of our method. We use a series of text prompts to generate 3D models that we expect to be self-supporting, and compare the generated results with baseline models. Our models are verified by simulation for stability and then fabricated using a 3D printer for real-world testing. We refer readers to the Appendix for more details about implementation, training, and experiment setup.Ours MVDreamFigure 5 : Atlas3D is also compatible with MVDream [69] , enhancing it with stable standability.Qualitative Comparison Using the same text prompts, we compare our generated models with previous methods. The quality of the results is assessed by their stability, which is verified by a forward simulation: we simulate the generated models in an upright initial position close to the ground for a sufficiently long time and record whether they fall. Using Magic3D [35] as the baseline model, we visualize the initial state and a later state in Fig. 4 . Our generated meshes remain stable throughout the simulation, while the baseline models fail under the same conditions due to a lack of consideration for physics.We highlight the main changes in mesh topology that enable the models to stand (see Fig. 4 ). These changes include modifications to both the overall shape and specific local geometries. More specifically, our physical adjustments alter the contact surface to gain more support from the ground and shift the center of mass to be slightly lower and more centered above the contact surface. These macroscopic and microscopic optimizations jointly increase the support to the models, thus ensuring their stability. Note that our method slightly modifies the texture of the generated results as we are jointly optimizing our physical adjustments with the SDS loss.Since we do not assume a specific baseline model in the first stage, we can vary the model used in the coarse stage to generate versatile, physically-aware 3D meshes based on different existing SDS generation models. We use MVDream [69] as another baseline model and compare the results side-by-side with ours in Fig. 5 . Our method improves the geometry of the mesh and ensures stability in simulation. The quality of the texture and the main part of the shape is determined by the underlying model used in the coarse stage, while we focus on improving the physical stability in the refinement stage. More qualitative comparisons with previous methods are provided in the Appendix.We perform ablation studies to demonstrate the necessity of our proposed losses. It can be observed in Fig. 6 (b), that without standability loss L stand , the model fails to stand. While the model can still stand without stable equilibrium loss L stable , as demonstrated in Fig. 6 (c), it is less stable under perturbation (see next section for details). The geometry regularization loss term L b-lap helps smooth the geometry and avoid spiky artifacts on the surface as shown in Fig. 6(d) . Additionally, we show that applying mesh regularization as a post-processing step, rather than integrating it with SDS loss in a joint optimization, can degrade text alignment as this neglects the semantics during the deformation process (see Fig. 6(e) ). Stability under Perturbation In the real world, placing an object on the ground always involves some noise, as both human and robot manipulation have imprecision in angles and directions. Hence, the standability of an object under small perturbations is crucial for improving the success rate of such tasks.To mimic this uncertainty in our framework, we evaluate the stability of our generated models under a small initial rotation. With a given precision ϕ max , we rotate the generated mesh at random angles ϕ y , ϕ x ∈ (-ϕ max , ϕ max ) in both the y and x axes, respectively (with the z axis being the up direction). The mesh is then placed close to the ground and tested in a simulation to see if it can still stand. We choose 13 different values of the maximum perturbation angle ϕ max and perform 100 random tests with each angle on 6 of our generated models. We report the success rate in Fig. 7 . We define a successful test as: after a sufficiently long time period, the maximum height of the model stays within 3% of the initial maximum height.Due to the presence of physical constraints, our generated models can withstand small initial perturbations, while the baseline models fail to stand when placed straight up (without rotation), let alone with perturbations. Furthermore, introducing the stable equilibrium loss consistently increases the success rate of standing under different scales of perturbations, as shown in Table . 1.w/ Atlas3D w/o Atlas3D Quantitative Evaluation We examine the versatility of our method. We randomly select 150 prompts from [58] and manually exclude 43 prompts deemed unfeasible (for instance, it does not make sense to require "a swan and its cygnets swimming in a pond" to be standable), leaving a total of 107 prompts. We use the two-stage Magic3D [35] as the baseline model and compare our optimized mesh with the results from the refine stage of the baseline under the same settings (e.g., iterations, loss weights).To evaluate the standability of the baseline method and our method, we run the rigid body simulation in Warp with simulation end time T = 2.0 at which almost all objects have reached the steady state. We propose Time-Averaged Rotation Deviation Loss (TRD) defined asEQUATIONto assess the standability, as a representation of the average tilting of the upward direction ẑ (of the object) over time. We approximate the integral (Eq. 11) with discrete quadrature ∆t = 0.02.Results are plotted in Fig. 9 4 . The mean TRD score is reduced by more than six times compared with the baseline method. As shown in Table . 2, we calculate the average CLIP score [63] of 107 generated shapes for both our proposed method and baseline. Furthermore, Elo (GPT-4o) [90] scores are presented. Both metrics illustrate that our method not only implements physics-based stability but also maintains the fidelity of the generated 3D shapes in terms of content alignment and overall shape quality. More details are provided in the Appendix. One major advantage of incorporating physics-based simulation into the optimization pipeline is that it bridges the gap between the generated model and the real world. Our method ensures the direct usability of the model for fabrication, with success primarily dependent on the accuracy of both the simulator and the manufacturing machine.We test the readiness of our generated meshes for real-world application by producing eight figures using a 3D-printing device (Zortrax M200 with Z-ABS filament material). For reference, we also print the corresponding baseline meshes generated without physical constraints. Our physically constrained figures can steadily support themselves when gently placed on an even surface, while the baseline figures either fail to stand at all, or require extensive adjustments and fall easily with little perturbation (see Fig. 2 ).We conduct user studies with these printed figures to assess their stability under different types of human manipulation. Ten users were asked to place the figures upright on a table, with five trials for each figure, resulting in a total of 800 trials. Fabricated figures generated from the baseline has a success rate of 7%, while figures generated from our method has an overall success rate of 92.25%.Itemized results are provided in the Appendix. Our method significantly increases the physical stability of the models under varying human efforts.Validation with a Robot Arm To showcase the compatibility of our framework with robotic applications, we test our fabricated figures with a teleoperated LewanSoul LeArm robot arm outfitted with a two-finger parallel gripper (see Fig. 10 ). The gripper is set to initially grasp a figure above the ground. It slowly moves downward, and is then gently released to place the figure on the ground. Four trials were performed for each figure, yielding 64 trials in total. For the baseline method, 6.25% trials resulted in successful standing figures, while ours has a success rate of 90.6%. Experimental data are provided in the Appendix and supplemental video. 
conclusion: 
 We present Atlas3D, a physically constrained SDS-based framework that generates self-supporting 3D models from text prompts. Our framework can learn standability through a differentiable physicsbased simulator and other physics-inspired loss functions. The generated 3D models can be directly imported into a physics simulator and are ready to be manufactured and deployed in the real world. Our method has wide potential for generation tasks, as it can be easily integrated into many existing pipelines and improve the physical plausibility of their generated results.Limitations and Future Work Our physical adjustments are optimized over all mesh vertices, resulting in a large degree of freedom in optimization. This may lead to undesired distorted meshes [49] . Future works may consider adding a latent embedding or skeleton rigging to limit the variety of mesh deformation. Our framework focuses on SDS-based methods as a backbone. It would be interesting to further generalize our physical constraints to other non-SDS or non-diffusion based methods [22; 49; 28] . Finally, we only consider text-to-3D tasks in this work. An exciting extension is to generalize our work to image-to-3D tasks [60; 39] .Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The paper specifies all the training and test details in the "Experiments" section.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The paper reports appropriate information about the statistical aspect in the "Experiments" section.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: There is no new societal impact compared to previous generative models on similar topics, as the improvement in quality does not introduce additional uncertainty. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All existing assets used are clearly cited or credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 Score Distillation Sampling Score Distillation Sampling (SDS) has proven to be an effective method for distilling 3D models from 2D images. SDS-based methods leverage a combination of 3D representations, differentiable rendering, and pre-trained text-to-image diffusion models to optimize 3D shapes with high fidelity and realism.The generation process in SDS methods involves optimizing the parameters θ of the 3D representation. At each iteration, the 3D geometry is differentiably rendered into a 2D image z = g(θ). This image is then compared with the distribution of real images as modeled by the diffusion model. More specifically, the input image z is first noised with a random noise ϵ at a specified noise level t. The diffusion model then predicts the noise εϕ with text guidance y, and this prediction is compared with ϵ, resulting in the following loss:EQUATION)where w(t) is a weighting function modulating the influence of different noise levels. The expectation is taken over the noise level t and noise term ϵ, ensuring that the generated 3D shape aligns with the text-guided distribution of images.The gradient with respect to the optimization parameter θ is then backpropagated as follows:EQUATIONwhere the U-Net Jacobian term∂ε ϕ∂zt is omitted for efficient optimization. For further technical details on SDS optimization, we refer readers to [58; 35; 46] .We implement our pipeline in PyTorch with Adam optimizer. For differentiable simulation, we adopt the semi-implicit Euler simulator in Warp [44] , where gradients of physical states are computed via auto-differentiation and backpropagated to the parameters of 3D representations [44; 56] . We use a two-stage training strategy and leave the choice of the first stage open for various SDS-based methods as baselines. For a two-stage baseline method, we implement our physics-inspired losses as submodules that can be seamlessly integrated into the refinement stage. For a one-stage method, our approach can be used as a standalone refinement stage to improve the physical quality of the baseline. For baseline methods that are not publicly available, we use the reimplementation from threestudio [14] .We train our models using a single NVIDIA RTX 3090 GPU. During our refinement stage, we implement a skipping strategy, incorporating standability loss only once every 10 iterations. In our quantitative evaluation of a batch of prompts, we observed an average refinement time of 36 minutes for each training step, with a default setting of 5,000 iterations.For the rigid body simulator in Warp, we set dt = 10 -3 s. Contact stiffness and damping are set to 10 3 and 2.0; friction coefficient is set to 0.5; stiffness of friction force is set to 10 3 . Density of the 3D objects is set to 10 3 .In our experiments, we use the following default weights for the loss terms: {λ SDS = 1, λ normal = 10 4 , λ stand = 10 5 , λ stable = 10 5 , λ b-lap = 10 7 }. For some examples, we tune these weights within the following ranges: {λ stand = 10 5 ∼ 5 × 10 5 , λ stable = 10 5 ∼ 5 × 10 5 , λ b-lap = 10 6 ∼ 10 7 }. Our heuristic intuition is to keep the SDS and physical loss terms roughly on the same scale. For the regularization terms, we scale them to around 1/1000 to 1/100 of the SDS and physical loss terms.While directly post-processing 3D generated models is a straightforward and effective approach to achieving physical stability, it may result in undesirable outcomes such as misalignment with the text prompt, as it overlooks semantics.One simple post-processing method is to cut the mesh by a flat plane slightly higher than the lowest vertex. However, this method will fail when the projection of the center of mass lies outside the contact region, as shown in Fig. 11 . Additionally, determining the cutting height is another parameter that must be manually set or optimized, and this may also degrade the overall appearance. Another post-processing method, make-it-stand [59] , offers an effective way to relocate the center of mass to achieve standability. However, it assumes that the supporting surface is fixed during optimization, which can lead to distorted results due to the imperfect quality of text-to-3D generated models. We provide two examples in Fig. 12 . For the goose example generated by Magic3D, one leg is shorter than the other. Make-it-stand only treats one foot as the supporting surface and ignores the other due to its post-processing nature, whereas our joint-optimization pipeline enables stable standing with two legs on the ground. A similar issue also happens to the kangaroo example. More importantly, the text alignment degrades as semantics are overlooked during optimization. In contrast, our proposed joint optimization method preserves the text alignment and dynamically adjusts the center of mass as well as the supporting surface configuration.Make-it-StandFigure 12 : Comparison with make-it-stand. 5In Fig. 6 (e) in the main text, we also apply our proposed losses in a post-processing manner. Similarly, while it is able to optimize the 3D models to make it standable, the text alignment is compromised. Overall, compared to the post-processing method, joint optimization is a more robust way which better balances text alignment and physical constraints.[…] a standing goose Ours MVDream w/o Atlas3D For CLIP score calculation, we specifically employ openai/clip-vit-large-patch14-336 as the check point of CLIP model. We rendered images from various angles (0-360 degree, 3 degree as the interval). For Elo (GPT-4o) benchmark, we made use of the newly released GPT-4o model instead of the one in [90] due to capacity limitation. Additionally, we reduce the number of views to 6 because of token length limitation. We report the detailed results of our user studies in Table . 4. Justification: The main claims are meticulously demonstrated in the "Experiments" section.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.We define the upright pose by setting the upward axis to coincide with the upward axis from the pre-trained text-to-3D model, as the upright direction is semantics-driven and generative models inherently learn these semantics from the training data.Results are ranked by TRD scores of the baseline approach.Although the output mesh of make-it-stand has the same resolution as the input mesh, the ordering of vertices and surfaces are shuffled in their implementation. As a result, we are unable to re-attach the original textures onto the output mesh. We are therefore unable to run metrics like CLIP in the paper to compare text alignment. Nevertheless, it should be visually clear that our results do have better text alignment.We define as upward pose for the kangaroo figure as its tail touching the ground, as a real does. 


Decision: This paper received divergent initial opinions with three reviewers in support of acceptance and one negative reviewer.  The main concerns by the negative reviewer revolved around missing ablations to demonstrate the impact of loss terms, and a lack of clarity about relation to robotics and fabrication.  The rebuttal responded to reviewer concerns, and the initially negative reviewer raised their opinion to accept. 
 The AC does not find a basis to overrule the reviewer consensus in favor of acceptance.  The AC thus recommends acceptance and strongly encourages the authors to incorporate clarifications and revisions to improve the final manuscript.

Reviews: review: paper_summary: paper_summary: This paper addresses the problem of generating 3D models from text that are visually appealing but often physically unstable in simulations or when 3D printed. The authors incorporates a differentiable simulation-based loss function and physically inspired regularization to ensure generated models are stable under perturbation. The proposed method involves a two-stage training process where a coarse model is first generated from text prompts and then refined with physical constraints to ensure stability. Experiments show that models created with Atlas3D maintain stability better than those from existing methods. The optimized models are also 3D fabricated to verify the stability in real world.
strength_weakness: strength_weakness: The paper is strong overall, and I did not identify any significant weaknesses. However, to further support future research in this area, I would suggest releasing the code. Providing the implementation would greatly benefit the community, enabling others to replicate the results and build upon this work.
questions: questions: I don't have any specific questions at this time. The paper is clear and well-executed, addressing an important issue effectively. Implementation is also described in detail.
limitations: limitations: I am curious about the failure modes of Atlas3D. Could you provide more details on the scenarios or conditions under which the method might not perform as expected? Understanding these limitations would be helpful for future research and applications. Additionally, addressing these potential failure modes could further strengthen the robustness of your approach.
paper_summary: paper_summary: The paper introduces a differentiable simulation-based loss to refine the existing SDS(Score Distillation Sampling)-based text-to-3D frameworks. Concretely, it relies on a differentiable simulator Warp to provide gradients for keeping the rotation of the generated mesh unchanged after a period of time. Besides, the authors also introduce a stable equilibrium loss, which favors a flat contact surface. Experiments show that the proposed method can generate self-supporting 3D models given text prompts, verified by simulation and real-world results with 3D-printed objects.
strength_weakness: strength_weakness: 1. The paper lacks enough baselines to show that the task the paper tackles is non-trivial. For example, the claim that "Directly integrating these methods with 3D generative AI as a postprocessing module is suboptimal" (L55-L56) is not verified by the experiments. If a self-supporting mesh is needed, the simplest baseline can be cutting the mesh by a flat plane moderately higher than the lowest vertex, as a postprocessing step.
2. The physical constraints used in this work are relatively limited. The standability loss only keeps the rotation unchanged. It can not recover some common cases like "a standing horse" (e.g., Napoleon Crossing the Alps), although those cases may be beyond "self-supporting" defined in this paper. It seems that the simulator has not been fully leveraged, since a simple post-processing baseline (e.g., flatting surfaces that contact with the ground) may also achieve good standability and stability.
3. The current method is only applied to SDS-based methods only (already pointed out by the authors in the limitations)
questions: questions: 1. Can the authors provide more baselines, especially any existing post-processing method with the help of differentiable simulators, to enhance their technical contributions? One example is given in the *Weaknesses* section.
2. Can the authors give a clear definition of "self-supporting"? It is a little ambiguous, since one can replace the initial pose with the pose after simulation so that Eq. (3) about standability can be satisfied, as long as the limit exists.
limitations: limitations: The paper has addressed its limitations.
paper_summary: paper_summary: This paper introduces a method to make 3D assets produced by SDS-based generative models stand on their own. On top of the SDS loss, it proposes a “standability" loss that encourages the rigid-body simulation result to be rotation-free, and a “stable” loss to encourage the generated shape to be a “local minimum” of height for the center of mass. The evaluation is conducted both in simulation and 3D-printed shapes in the real world.
strength_weakness: strength_weakness: - The proposed method involves solving an optimization that involves multiple loss terms. No ablation study is provided to prove the necessity of each term. In particular, are both standability loss and stable loss required? How are their influence on the optimization results respectively? Are both normal smoothing Eq (8) and Laplacian smoothing Eq (9) required and why is Laplacian smoothing only applied to the bottom of the shape?
- The traditional balancing method in computational fabrication [59] is only discussed but not actually compared in experiments.
- What are the loss weights for each term in Eq (10)? These numbers may be tricky to set, and thus make the paper hard to reproduce.
- How is \(T\) set in Eq (5)? How do different choices of \(T\) influence the results?
- The success rate of the baseline method is not reported in Fig. 6 for the audience to understand the improvement.
questions: questions: See weaknesses. These are important questions to answer for a well-rounded paper.
limitations: limitations: Seems adequate.
paper_summary: paper_summary: This paper proposes to combine physical standability with existing diffusion-based text-to-3D generators in order to synthesize objects that not only follow the text description but also can stand on their own without falling. To this end, the paper proposes a physical loss function and uses it with score distillation sampling methods that generate 3D objects from texts. The paper shows that their generated model show better standability both in simulation and after physical realization.
strength_weakness: strength_weakness: First, there are good methods, such as 'make it stand' that could be used as post-processing tools for making text-based generated models stand. Why should we go through the paper's approach in the light of these methods? 
Some important ablations are missing. For example,
- Separating of how much each loss is contributing to standability. For example, what if we use Magic 3D outputs and smoothen the geometry? 
- What are the pros and cons of using the physical loss as a post-processing or jointly with SDS? 


The claims are more broad than the actual contributions. 
- Take "fabrication-ready" in line 62. The paper is concerned with stabdability and it should adapt the text (and even the title). In 3D printing, "self supporting" means that there is no need to support materials. The paper doesn't cover this. 
- The relationship to robotics is not well-founded except an example that uses robot grippers to do the standability tests, which is not really 
 related to benefits to robotics. 

The paper is heavily relied on score distillation sampling methods but does not explain them very well. I understand that the paper refers to the original papers in this area but it is worth including a paragraph about SDS. 

I look forward to see a justification in the rebuttal.
questions: questions: - The paper mentions too many steps of integration and back propagation in Section 4.1.1. Does it still do this? If not, where exactly the differentiable physics simulation is used? 
- What is the default intimal physical state? 
- How does the paper figures out the upright position? 
- The paper mentions mesh topology (Line 246) but doesn't show it. A zoom in mesh is necessary for this. 
- What are the mesh sizes and dimensionalities of printed objects? 
- Are Figure 7 results printed? 
- x_{com} is not defined in Eq. 7.
limitations: limitations: A main limitation is the mesh size. The paper has mentioned it. This is important in the light of 3D printing that require sometimes huge resolutions.

Comments: comment: Thank you for the response. I do not have further questions.
First of all, thank you for the answers in the rebuttal.
For Q1, I appreciate that the authors provide comparison on certain, perhaps representative, cases qualitatively. However, it is better to compare with other methods quantitatively, which will make the paper more convincing. I would like to keep my rating.
Thank you for your explanations. I changed my score and agree with accepting the paper.