Title: Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabrication
Venue: NeurIPS 2024
Accepted: yes
Keywords: Text-to-3D Generation

Score Distillation Sampling

Physics-based Simulation
Abstract: Existing diffusion-based text-to-3D generation methods primarily focus on producing visually realistic shapes and appearances, often neglecting the physical constraints necessary for downstream tasks. Generated models frequently fail to maintain balance when placed in physics-based simulations or 3D printed. This balance is crucial for satisfying user design intentions in interactive gaming, embodied AI, and robotics, where stable models are needed for reliable interaction. Additionally, stable models ensure that 3D-printed objects, such as figurines for home decoration, can stand on their own without requiring additional supports. To fill this gap, we introduce Atlas3D, an automatic and easy-to-implement method that enhances existing Score Distillation Sampling (SDS)-based text-to-3D tools. Atlas3D ensures the generation of self-supporting 3D models that adhere to physical laws of stability under gravity, contact, and friction. Our approach combines a novel differentiable simulation-based loss function with physically inspired regularization, serving as either a refinement or a post-processing module for existing frameworks. We verify Atlas3D's efficacy through extensive generation tasks and validate the resulting 3D models in both simulated and real-world environments.

Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabrication

Abstract

Existing diffusion-based text-to-3D generation methods primarily focus on producing visually realistic shapes and appearances, often neglecting the physical constraints necessary for downstream tasks. Generated models frequently fail to maintain balance when placed in physics-based simulations or 3D-printed. This balance is crucial for satisfying user design intentions in interactive gaming, embodied AI, and robotics, where stable models are needed for reliable interaction. Additionally, stable models ensure that 3D-printed objects, such as figurines for home decoration, can stand on their own without requiring additional support. To fill this gap, we introduce Atlas3D, an automatic and easy-to-implement method that enhances existing Score Distillation Sampling (SDS)-based text-to-3D tools. Atlas3D ensures the generation of self-supporting 3D models that adhere to physical laws of stability under gravity, contact, and friction. Our approach combines a novel differentiable simulation-based loss function with physically inspired regularization, serving as either a refinement or a post-processing module for existing frameworks. We verify Atlas3D's efficacy through extensive generation tasks and validate the resulting 3D models in both simulated and real-world environments.

1 Introduction

Generating high-quality 3D content is of great importance in modern visual computing. Realistic 3D models are highly sought after in computer graphics, while robust real 3D assets are gaining attention in training embodied AI. Nevertheless, the standability of 3D models - the ability to stand steadily without additional support - is often neglected. Real-world man-made objects such as action figures, toys, and furniture inherently possess some degree of geometric stability, allowing them to be safely placed on the ground. Although one usually takes such standability for granted, existing generative models fail to produce steady 3D assets due to their lack of physical perception; see Fig. 1.

Incorporating this stability expectation into 3D generation will significantly reduce the human effort required for tasks such as sorting out unqualified meshes, post-processing geometries, or adding external supports before actually using the 3D asset in any simulator or the real world. Furthermore, creating physically plausible 3D content will enhance the fidelity of simulations and policy training with these objects, potentially narrowing the sim-to-real gap and empowering embodied AI in robotic tasks. Towards this goal, we develop a 3D generation framework that can produce high-quality models adhering to basic physical laws, such as gravity, stability, and frictional contact.

Several attempts have been made to incorporate physical constraints into 3D generation. Yang et al. utilized the spatial and physical sense of LLM to design floor plans and furniture arrangements [98].

(a) w/ Atlas3D

(b) w/o Atlas3D

PhyScene introduced physical guidance, such as collision and reachability constraints, to diffusion models to generate furniture layouts [97]. However, both works primarily consider straightforward spatial constraints, such as non-collision, and fail to incorporate more complex physics. Mezghanni et al. proposed a GAN-based network to generate physically-aware geometries by training a neural stability predictor using datasets labeled by Bullet [48]. Another GAN-based work by Wang et al. employed CFD software to compute vehicle drag coefficients, guiding the generation of streamlined vehicle meshes [85]. Such indirect incorporation of physical simulations, however, results in suboptimal efficiency and accuracy. Furthermore, due to the low expressibility of the backbone latent representation, the versatility of the generated results is very limited compared to current state-of-the-art diffusion-based models, as the results are typically confined to specific categories (e.g., furniture and vehicles). Most recently, Ni et al. bridged differentiable physical simulation with differentiable rendering to obtain virtual 3D reconstructions from real-world images that are physically plausible in simulators. Their work primarily focused on simple four-leg-supported objects such as tables and chairs [54]. Moreover, the evaluations of all aforementioned works are conducted in virtual simulators, leaving their performance in the real world untested. This limitation hinders potential downstream applications such as industrial manufacturing and robotic manipulation.

Since the pioneering work DreamFusion [58], Score Distillation Sampling (SDS) has demonstrated efficacy in elevating 2D content to 3D, inspiring numerous follow-up studies [8; 35; 46; 69; 88; 109]. These advancements have enhanced both the versatility of generated content and the quality of textures. However, none have addressed the crucial issue of physical stability. On the other hand, traditional computational fabrication has concentrated on employing topology and shape optimization to ensure that 3D printed objects can stand in a balanced state [59]. Directly integrating these methods with 3D generative AI as a postprocessing module is suboptimal. Shape optimization disregards the original input conditions of diffusion models, while topology optimization produces internal structures that defy intuitive physics, rendering them unsuitable for training embodied AI systems designed to emulate human-like reasoning about physical objects.

Observing this gap, we introduce Atlas3D, a generation pipeline that produces physically plausible, self-supporting 3D models from text. Incorporating differentiable physics-based simulation into our process, we generate models that are both simulation- and fabrication-ready. That is, they can be directly utilized in physical simulators, or 3D-printed for real-world applications; see Fig. 1 and Fig. 2. As our method is orthogonal to previous SDS-based techniques, which focus on non-physical qualities, it can be seamlessly integrated into many existing generation frameworks, functioning either as part of the refinement stage in a multi-stage method or as a post-processing step in a single-stage method. We demonstrate the efficacy of Atlas3D by comparing the stability of our models with those

produced by existing methods. Validation examples reveal that our generated models can be deployed as virtual simulation assets. Their stability transfers directly to the real world, as evidenced by our 3D printed results, suggesting further applications in robot training.

2 Related Work

Diffusion-based 3D Generation Due to the abundance of information encoded in large image latent diffusion models (LDMs) [65], extensive studies have used pre-trained LDMs to distill 3D content. One approach is to fine-tune LDMs to support novel view synthesis, with a separate multiview fusion step to produce 3D content [39; 68; 99; 38; 37; 89; 95; 42; 80; 25; 106; 41; 9; 83]. Another approach, which is more related to our paper, is using LDMs as likelihood discriminators. A differentiable renderer is connected to a 3D representation, and the LDMs guide the optimization of the representation parameters. [58] proposed Score Distillation Sampling (SDS). Efficiency has been improved by coarse-to-fine strategies [35; 60; 79; 8] and timestep scheduling [18; 100]. 3D priors are involved to improve multiview consistency [108; 66; 31; 1; 93; 36; 78]. Multiview diffusions can also be used to evaluate SDS [69; 87; 105; 94]. Other researchers have explored SDS variants or improvements [84; 77; 88; 24; 101; 16; 109; 96]. 3D LDMs that directly generate 3D representations are also explored, such as compositional scenes [17; 57], point clouds [43; 81; 51], SDFs [70; 107; 30], occupancy fields [15; 72; 45; 12] and NeRFs [3; 11; 52; 6; 55; 5; 75; 23].

Physics-aware 3D Generation Most existing 3D generative models focus only on geometry or appearance modeling, with physics priors being underexplored. Time-independent physical constraints, such as penetrations, can be directly defined by penalties [97; 17; 82; 40; 102]. For time-dependent physical qualities, such as stability and comfort, data-driven quality checkers trained with offline simulators can be applied [10; 48; 4]. Offline simulators can also be used as validators to augment the training dataset [71] and update the design with reinforcement learning [85], and as dynamics generators for generated 3D assets [91; 21; 62]. Another direction is to utilize differentiable simulations, which have be widely used in tasks like robotic control [74; 19; 61; 34; 76] or inverse problems [33; 20; 103; 53; 32; 73; 7; 104]. They can also be applied in 3D content generation to define physics-based losses to aid per-instance generation [54; 92] or model training [47].

3 Background

3.1 Score Distillation Sampling

SDS-based methods are shown to be effective in distilling 3D models from 2D images. They utilize a 3D representation such as implicit density field, implicit Signed Distance Field (SDF), or tetrahedral SDF [67], a differentiable renderer like NeRF [50], NeuS [86], or Nvdiffrast [26], and a pre-trained text-to-image model such as Stable Diffusion [64] serving as diffusion guidance. The generation process involves optimizing the parameters θ of the underlying 3D representation, where the 3D shape is differentiably rendered to 2D images z = g ( θ ) and compared against the real distribution from the diffusion model with text guidance y :

where z t is the noisy image at noise level t , w ( t ) is a weighting function, and ˆ ϵ ϕ is the predicted noise. We refer readers to the Appendix for more technical details on SDS optimization.

3.2 Rigid Body Dynamics

To incorporate physics into our framework, we propose predicting the dynamics of the generated 3D models using a differentiable simulator, where all objects are treated as rigid bodies. We follow the conventions in [2] to define the dynamical states of the simulation.

The kinematics of a rigid body are described by its mass M and body-space inertia tensor I body, which remains constant. Assuming the center of mass of the body initially lies at the origin, the physical state Ψ of the body at time t (not to be confused with the noise level in diffusion) includes position T ( t ) and orientation R ( t ) (spatial information), and its linear and angular momentum P ( t ) and L ( t ) (velocity information). The rigid body equations of motion are given by

where F ( t ) and τ ( t ) are the total force and torque exerted on the body, v ( t ) = P ( t ) M is the linear velocity, ω ( t ) = I ( t ) -1 L ( t ) is the angular velocity, I ( t ) = R ( t ) I body R ( t ) T is the world-space inertia tensor, and ∗ denotes cross product of ω with the columns of R . The physical state at a later time can be derived via time inte-

gration: Ψ( t ) = Ψ(0) + ∫ t 0 d Ψ ds ( s ) ds, which can be solved by numerical methods. By optimizing the physical states together with the SDS loss, we can jointly refine both the 3D geometry and the physical attributes of the generated results.

4 Atlas3D Algorithm

We introduce Atlas3D, a plug-and-play algorithm for generating 3D models from text. Focusing on man-made objects such as action figures and toys, which generally do not deform, Atlas3D treats generated models as rigid bodies and incorporates physics-based guidance into the generation process.

4.1 Physics Incorporation

As mentioned in § 3.2, we predict the dynamic behavior of generated models by rigid body simulations. While various explicit or implicit representations of 3D shapes can be chosen in a generation network, we opt for triangular meshes in our framework as they facilitate frictional contact modeling and simplify kinematics computation. Given a triangle surface mesh representation X ( θ ) , where θ is the implicit parameter, we integrate X into a rigid body represented by the dynamic state Ψ( t ) = [ T ( t ) , R ( t ) , P ( t ) , L ( t )] T , where the world-space location x of any point X on the body is x ( t ) = R ( t ) X + T ( t ) . Assuming the 3D model is initially placed upright 3 on the ground with the bottom point touching the surface, we define standability as:

Standability intuitively indicates an equilibrium state where all external forces acting on the object are balanced, and the physical state remains unchanged over time. However, perfectly placing an object straight on the ground without initial velocity is impractical in the real world. For example, when manually placing a cube on a flat table, the bottom face is unlikely to be perfectly parallel to the table surface. A stable 3D model should recover its initial state under mild perturbations, such as minor shaking. This state is known as stable equilibrium. Motivated by this, we augment standability with stable equilibrium ˜ Ψ( t ) defined as

where ϵ 0 represents mild perturbations to the initial physical state. We first describe how to incorporate the standability criterion (Eq. 3) into the optimization process of 3D generation, and then explain how to further augment it with stable equilibrium (Eq. 4). Additionally, we introduce geometry regularization to enhance the smoothness of generated meshes.

4.1.1 Standability through Differentiable Simulation

We utilize a differentiable rigid-body simulator to obtain the physical state Ψ( t ) . Assuming that the 3D model will eventually reach its steady state, ∃ T ∈ R large enough such that ∀ t > T, Ψ( t ) = Ψ( T ) . Let S denote a differentiable simulation function. We approximate Ψ( T ) via simulation:

Here T is the simulation end time, µ captures material parameters such as density and friction coefficient, as well as simulator parameters such as time step and damping. We adopt the semiimplicit Euler time-integrator in Warp [44] for simulation.

Assuming the initial translation, rotation, and velocity are all zero, the difference between Ψ( T ) and Ψ(0) arises only from discrepancies in spatial location, as the velocity at the final steady state is also zero. Therefore, we propose a standability loss to penalize rotational deviation due to instability:

where R ∈ R 3 × 3 represents the rotation matrix. We disregard the translation T , as real-world instability mostly leads to rotational deviation from the initial state, such as falling to one side, while most translations, like falling due to gravity, are irrelevant to standability.

With a differentiable simulator, the standability loss can be backpropagated to mesh vertex coordinates and then to the implicit parameter θ as d L stand dθ = d L stand d X d X dθ . In theory, any differentiable simulator is compatible with our framework.

4.1.2 Stable Equilibrium

Although the standability loss directly penalizes the nonstandability of a 3D object, it can be slow to compute, especially when T is large and many time steps are required, creating a huge computational graph. Consequently, both the simulation itself and the backpropagation of gradients through the simulation trajectory are time-consuming. Additionally, standability does not necessarily imply stable equilibrium, which is crucial for real-world 3D objects such as action figures and toys. Without this property, an object remains unstable even if standability is achieved, known as unstable equilibrium. Unstable equilibrium means that when a disturbance force is applied, the object moves away from its original position instead of recovering. Fig. 3 visualizes the difference between stable and unstable equilibrium. In the absence of perturbation, geometries like the upside-down triangle may remain standable in a simulator but are clearly unstable in the real world. Thus, we augment standability with stable equilibrium (Eq. 4). One straightforward way to incorporate this property is to introduce initial perturbation ϵ 0 into the simulator. However, this would require many more simulations with various perturbations and subsequent loss backpropagation, which is extremely time-consuming.

(a)

H(x

H(xcom)

(b)

Inspired by the concept of a potential well [13], we augment our optimization objective with a robust and efficient stable equilibrium loss L stable. Specifically, for an object to be robustly standable, it needs to reside at a local minimum of potential energy-specifically, gravitational potential energy-so that if perturbed, gravity will act as a restoring force that returns the object to its original state. For a rigid body, gravitational potential energy is determined by the height of its center of mass. Thus, for any object in a stable equilibrium state, the center of mass would rise if it is slightly perturbed. This leads to our formulation of the stable equilibrium loss L stable. Let x com denote the position of the center of mass of the underlying geometry and H ( x ) denote the distance of the point x to the ground, assuming the object's pivot point is at z = 0 . The stable equilibrium loss is defined as:

where P ϕ v represents the rotation of ϕ radian about axis [ v T , 0] T . Mathematically, a local minimum of gravitational potential energy is reached if ∃ ϕ 0 such that ∀ ϕ ∈ (0 , ϕ 0 ) , L stable = 0 . In practice, we fix the perturbation scale ϕ and uniformly sample 20 perturbation directions v in xy -plane.

4.2 Additional Regularization

While standability loss L stand and stable equilibrium loss L stable provide a well-defined objective for robust standing, they may lead to distorted optimized meshes due to the high-dimensional searching

[…] a standing rooster

[…]a broccoli

[…] front view of a snowboarding man

Magic3D

Ours

space of implicit parameter θ without constraint. To constrain the optimization space and obtain smooth meshes, we add a normal consistency term that favors smooth solutions:

where T is the set of the triangle pairs sharing a common pair with n i , n j being their normals respectively. This term maximizes the cosine similarity between neighboring surface triangle normals, leading to smoother meshes. Considering the bottom surfaces of most robust standing objects are flat, we apply the Laplacian loss to a subset B of vertices with a height lower than a threshold h b :

where δ i = ( LV ) i ∈ R 3 calculates the differential coordinates of vertex i with L being the Laplacian matrix of the mesh graph and V representing mesh vertices. Intuitively, this loss term attempts to minimize the distance between vertex i and the average position of adjacent vertices.

4.3 Method Overview

With the physically-inspired loss terms derived above, we now describe how to incorporate them into the text-to-3D framework. SDS-based methods and their variants start optimization with a random initialization of the implicit parameters, which initially have no knowledge of the model's geometry. Adding physical constraints at this early stage would be ineffective. Therefore, we propose a two-stage training strategy: the coarse stage and the refine stage. In the coarse stage, we generate a rough shape of the model using a text prompt. We can adopt any SDS-based generation framework as our baseline model, offering various choices of implicit representation and differentiable renderers. In the refine stage, we optimize the geometry with our physical constraints included. For this, we use a tetrahedral SDF representation [67] and employ Deep Marching Tetrahedra (DmTet) to differentiably convert the coarse geometry from implicit density or SDF as necessary. We utilize Nvdiffrast [26] as the differentiable renderer and Stable Diffusion v2.1 [64] for guidance. We propose the following loss function for the joint optimization of texture, geometry, and stability:

In practice, we observe that adding standability loss once every 10 iterations is sufficient to ensure a significant reduction in loss without notably increasing computational overhead.

5 Experiments

In this section, we devise comprehensive experiments (both virtual and real-world) to demonstrate the efficacy of our method. We use a series of text prompts to generate 3D models that we expect to be self-supporting, and compare the generated results with baseline models. Our models are verified by simulation for stability and then fabricated using a 3D printer for real-world testing. We refer readers to the Appendix for more details about implementation, training, and experiment setup.

An astronaut

riding a horse

Ours

MVDream

holding up a

celestial globe

on his

shoulders

A running Harley

motorcycle

Ours

MVDream

A floor lamp

5.1 Simulation Verification

Qualitative Comparison Using the same text prompts, we compare our generated models with previous methods. The quality of the results is assessed by their stability, which is verified by a forward simulation: we simulate the generated models in an upright initial position close to the ground for a sufficiently long time and record whether they fall. Using Magic3D [35] as the baseline model, we visualize the initial state and a later state in Fig. 4. Our generated meshes remain stable throughout the simulation, while the baseline models fail under the same conditions due to a lack of consideration for physics.

We highlight the main changes in mesh topology that enable the models to stand (see Fig. 4). These changes include modifications to both the overall shape and specific local geometries. More specifically, our physical adjustments alter the contact surface to gain more support from the ground and shift the center of mass to be slightly lower and more centered above the contact surface. These macroscopic and microscopic optimizations jointly increase the support to the models, thus ensuring their stability. Note that our method slightly modifies the texture of the generated results as we are jointly optimizing our physical adjustments with the SDS loss.

Since we do not assume a specific baseline model in the first stage, we can vary the model used in the coarse stage to generate versatile, physically-aware 3D meshes based on different existing SDS generation models. We use MVDream [69] as another baseline model and compare the results side-by-side with ours in Fig. 5. Our method improves the geometry of the mesh and ensures stability in simulation. The quality of the texture and the main part of the shape is determined by the underlying model used in the coarse stage, while we focus on improving the physical stability in the refinement stage. More qualitative comparisons with previous methods are provided in the Appendix.

Ablation Study We perform ablation studies to demonstrate the necessity of our proposed losses. It can be observed in Fig. 6(b), that without standability loss L stand, the model fails to stand. While the model can still stand without stable equilibrium loss L stable, as demonstrated in Fig. 6(c), it is less stable under perturbation (see next section for details). The geometry regularization loss term L b-lap helps smooth the geometry and avoid spiky artifacts on the surface as shown in Fig. 6(d). Additionally, we show that applying mesh regularization as a post-processing step, rather than integrating it with SDS loss in a joint optimization, can degrade text alignment as this neglects the semantics during the deformation process (see Fig. 6(e)).

(e) w/o SDS

(d) w/o b-lap

(b) w/o stand

(a) Ours

(c) w/o stable

Stability under Perturbation In the real world, placing an object on the ground always involves some noise, as both human and robot manipulation have imprecision in angles and directions. Hence, the standability of an object under small perturbations is crucial for improving the success rate of such tasks.

To mimic this uncertainty in our framework, we evaluate the stability of our generated models under a small initial rotation. With a given precision ϕ max , we rotate the generated mesh at random angles ϕ y , ϕ x ∈ ( -ϕ max , ϕ max ) in both the y and x axes, respectively (with the z axis being the up direction). The mesh is then placed close to the ground and tested in a simulation to see if it can still stand. We choose 13 different values of the maximum perturbation angle ϕ max and perform 100 random tests with each angle on 6 of our generated models. We report the success rate in Fig. 7. We define a successful test as: after a sufficiently long time period, the maximum height of the model stays within 3% of the initial maximum height.

Perturbation Angle (rad)

Success Rate

0.00

0.25

0.50

0.75

1.00

0.00

0.05

0.10

0.15

0.20

ultraman

astronaut-horse

goose

mannequin

snowboarding

Due to the presence of physical constraints, our generated models can withstand small initial perturbations, while the baseline models fail to stand when placed straight up (without rotation), let alone with perturbations. Furthermore, introducing the stable equilibrium loss consistently increases the success rate of standing under different scales of perturbations, as shown in Table. 1.

Standability on Different Platforms Our pipeline can be generalized to learn standability on various platforms, not just flat ground, by incorporating them as boundary conditions in the simulator. To demonstrate this, we use a 10-degree inclined plane and a sphere, then train our 3D model separately to stand still on each. Modeling frictional contact is crucial for achieving stability in such scenarios. As shown in Figure Fig. 8, our optimized mesh stands stably on both the incline and the sphere with the help of static friction, whereas the baseline model fails as expected.

Simulator Cross-validation While we base our method on a differentiable rigid-body simulator with a semi-implicit Euler time-integrator, our pipeline is compatible with any other physics-based simulator as the backbone, with differentiability required for the training stage. To verify the reliability of models generated with our simulator, we include an external simulator in the testing stage to verify the correctness of the simulated dynamics. We choose the Incremental Potential Contact (IPC) method [27; 29], which has been proven accurate for frictional contact. We validate the correctness of every single generated model and visualize the simulation results in Fig. 1.

w/ Atlas3D

w/o Atlas3D

Brocolli

Snowboarding

Ultraman

w/Atlas3D

穴

w/oAtlas3D

KA

Quantitative Evaluation We examine the versatility of our method. We randomly select 150 prompts from [58] and manually exclude 43 prompts deemed unfeasible (for instance, it does not make sense to require 'a swan and its cygnets swimming in a pond' to be standable), leaving a total of 107 prompts. We use the two-stage Magic3D [35] as the baseline model and compare our optimized mesh with the results from the refine stage of the baseline under the same settings (e.g., iterations, loss weights).

To evaluate the standability of the baseline method and our method, we run the rigid body simulation in Warp with simulation end time T = 2 . 0 at which almost all objects have reached the steady state. We propose Time-Averaged Rotation Deviation Loss (TRD) defined as

0.00

0.25

0.50

0.75

1.00

1.25

w/o Atlas3D

w/ Atlas3D

to assess the standability, as a representation of the average tilting of the upward direction ˆ z (of the object) over time. We approximate the integral (Eq. 11) with discrete quadrature ∆ t = 0 . 02 . Results are plotted in Fig. 9 4 . The mean TRD score is reduced by more than six times compared with the baseline method. As shown in Table. 2, we calculate the average CLIP score [63] of 107 generated shapes for both our proposed method and baseline. Furthermore, Elo (GPT-4o) [90] scores are presented. Both metrics illustrate that our method not only implements physics-based stability but also maintains the fidelity of the generated 3D shapes in terms of content alignment and overall shape quality. More details are provided in the Appendix.

5.2 Real-world Validation

One major advantage of incorporating physics-based simulation into the optimization pipeline is that it bridges the gap between the generated model and the real world. Our method ensures the direct usability of the model for fabrication, with success primarily dependent on the accuracy of both the simulator and the manufacturing machine.

3D Printing and User Studies We test the readiness

of our generated meshes for real-world application by producing eight figures using a 3D-printing device (Zortrax M200 with Z-ABS filament material). For reference, we also print the corresponding baseline meshes generated without physical constraints. Our physically constrained figures can steadily support themselves when gently placed on an even surface, while the baseline figures either fail to stand at all, or require extensive adjustments and fall easily with little perturbation (see Fig. 2).

We conduct user studies with these printed figures to assess their stability under different types of human manipulation. Ten users were asked to place the figures upright on a table, with five trials for each figure, resulting in a total of 800 trials. Fabricated figures generated from the baseline has a success rate of 7%, while figures generated from our method has an overall success rate of 92.25%.

Itemized results are provided in the Appendix. Our method significantly increases the physical stability of the models under varying human efforts.

Validation with a Robot Arm To showcase the compatibility of our framework with robotic applications, we test our fabricated figures with a teleoperated LewanSoul LeArm robot arm outfitted with a two-finger parallel gripper (see Fig. 10). The gripper is set to initially grasp a figure above the ground. It slowly moves downward, and is then gently released to place the figure on the ground. Four trials were performed for each figure, yielding 64 trials in total. For the baseline method, 6.25% trials resulted in successful standing figures, while ours has a success rate of 90.6%. Experimental data are provided in the Appendix and supplemental video.

6 Conclusion

We present Atlas3D, a physically constrained SDS-based framework that generates self-supporting 3D models from text prompts. Our framework can learn standability through a differentiable physicsbased simulator and other physics-inspired loss functions. The generated 3D models can be directly imported into a physics simulator and are ready to be manufactured and deployed in the real world. Our method has wide potential for generation tasks, as it can be easily integrated into many existing pipelines and improve the physical plausibility of their generated results.

Limitations and Future Work Our physical adjustments are optimized over all mesh vertices, resulting in a large degree of freedom in optimization. This may lead to undesired distorted meshes [49]. Future works may consider adding a latent embedding or skeleton rigging to limit the variety of mesh deformation. Our framework focuses on SDS-based methods as a backbone. It would be interesting to further generalize our physical constraints to other non-SDS or non-diffusion based methods [22; 49; 28]. Finally, we only consider text-to-3D tasks in this work. An exciting extension is to generalize our work to image-to-3D tasks [60; 39].

References

Titas Anciukeviˇ cius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1260812618, 2023.

David Baraff. Physically based modeling: Rigid body simulation. SIGGRAPH Course Notes, ACM SIGGRAPH , 2(1):2-1, 2001.

Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, et al. Gaudi: A neural architect for immersive 3d scene generation. Advances in Neural Information Processing Systems , 35:25102-25116, 2022.

Bryce Blinn, Alexander Ding, R Kenny Jones, Manolis Savva, Srinath Sridhar, and Daniel Ritchie. Learning body-aware 3d shape generative models. arXiv preprint arXiv:2112.07022 , 2021.

Ziang Cao, Fangzhou Hong, Tong Wu, Liang Pan, and Ziwei Liu. Large-vocabulary 3d diffusion model with transformer. arXiv preprint arXiv:2309.07920 , 2023.

Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2416-2425, 2023.

Hsiao-yu Chen, Edith Tretschk, Tuur Stuyck, Petr Kadlecek, Ladislav Kavan, Etienne Vouga, and Christoph Lassner. Virtual elastic objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15827-15837, 2022.

Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22246-22256, 2023.

Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, and Huaping Liu. V3d: Video diffusion models are effective 3d generators. arXiv preprint arXiv:2403.06738 , 2024.

Yuan Dong, Qi Zuo, Xiaodong Gu, Weihao Yuan, Zhengyi Zhao, Zilong Dong, Liefeng Bo, and Qixing Huang. Gpld3d: Latent diffusion of 3d shape generative models by edupont2022datanforcing geometric and physical priors. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024) . IEEE/CVF, 2024.

Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. arXiv preprint arXiv:2201.12204 , 2022.

Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14300-14310, 2023.

David J Griffiths and Darrell F Schroeter. Introduction to quantum mechanics . Cambridge university press, 2018.

Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia-Hao Chen, Zi-Xin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A unified framework for 3d content generation. https://github.com/threestudio-project/threestudio , 2023.

Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas O˘ guz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.05371 , 2023.

Susung Hong, Donghoon Ahn, and Seungryong Kim. Debiasing scores and prompts of 2d diffusion for view-consistent text-to-3d generation. Advances in Neural Information Processing Systems , 36, 2024.

Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song-Chun Zhu. Diffusion-based generation, optimization, and planning in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16750-16761, 2023.

Yukun Huang, Jianan Wang, Yukai Shi, Boshi Tang, Xianbiao Qi, and Lei Zhang. Dreamtime: An improved optimization strategy for diffusion-guided 3d generation. In The Twelfth International Conference on Learning Representations , 2023.

Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. arXiv preprint arXiv:2104.03311 , 2021.

Zizhou Huang, Davi Colli Tozoni, Arvi Gjoka, Zachary Ferguson, Teseo Schneider, Daniele Panozzo, and Denis Zorin. Differentiable solver for time-dependent deformation problems with contact. ACM Transactions on Graphics , 2022.

Ying Jiang, Chang Yu, Tianyi Xie, Xuan Li, Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, et al. Vr-gs: A physical dynamics-aware interactive gaussian splatting system in virtual reality. arXiv preprint arXiv:2401.16663 , 2024.

Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463 , 2023.

Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy J Mitra. Holodiffusion: Training a 3d diffusion model using 2d images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 18423-18433, 2023.

Oren Katzir, Or Patashnik, Daniel Cohen-Or, and Dani Lischinski. Noise-free score distillation. In The Twelfth International Conference on Learning Representations , 2024.

Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, and Andrew J Davison. Eschernet: A generative model for scalable view synthesis. arXiv preprint arXiv:2402.03908 , 2024.

Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics (ToG) , 39(6):114, 2020.

Lei Lan, Danny M Kaufman, Minchen Li, Chenfanfu Jiang, and Yin Yang. Affine body dynamics: Fast, stable & intersection-free simulation of stiff materials. arXiv preprint arXiv:2201.10022 , 2022.

Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311.06214 , 2023.

Minchen Li, Zachary Ferguson, Teseo Schneider, Timothy R Langlois, Denis Zorin, Daniele Panozzo, Chenfanfu Jiang, and Danny M Kaufman. Incremental potential contact: intersection-and inversion-free, large-deformation dynamics. ACM Trans. Graph. , 39(4):49, 2020.

Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdf: Text-to-shape via voxelized diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1264212651, 2023.

Weiyu Li, Rui Chen, Xuelin Chen, and Ping Tan. Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d. arXiv preprint arXiv:2310.02596 , 2023.

Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for geometry-agnostic system identification. In The Eleventh International Conference on Learning Representations , 2022.

Yifei Li, Hsiao-yu Chen, Egor Larionov, Nikolaos Sarafianos, Wojciech Matusik, and Tuur Stuyck. Diffavatar: Simulation-ready garment optimization with differentiable simulation. arXiv preprint arXiv:2311.12194 , 2023.

Yifei Li, Tao Du, Kui Wu, Jie Xu, and Wojciech Matusik. Diffcloth: Differentiable cloth simulation with dry frictional contact. ACM Transactions on Graphics (TOG) , 42(1):1-20, 2022.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 300-309, 2023.

Yuanze Lin, Ronald Clark, and Philip Torr. Dreampolisher: Towards high-quality text-to-3d generation via geometric diffusion. arXiv preprint arXiv:2403.17237 , 2024.

Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:2311.07885 , 2023.

Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-23-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems , 36, 2024.

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9298-9309, 2023.

Xueyi Liu, Bin Wang, He Wang, and Li Yi. Few-shot physically-aware articulated mesh generation via hierarchical deformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 854-864, 2023.

Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453 , 2023.

Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008 , 2023.

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2837-2845, 2021.

Miles Macklin. Warp: A high-performance python framework for gpu simulation and graphics. https: //github.com/nvidia/warp , March 2022. NVIDIA GPU Technology Conference (GTC).

Antoine Mercier, Ramin Nakhli, Mahesh Reddy, Rajeev Yasarla, Hong Cai, Fatih Porikli, and Guillaume Berger. Hexagen3d: Stablediffusion is just one step away from fast and diverse text-to-3d generation. 2024.

Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shapeguided generation of 3d shapes and textures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12663-12673, 2023.

Mariem Mezghanni, Théo Bodrito, Malika Boulkenafed, and Maks Ovsjanikov. Physical simulation layer for accurate 3d modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13514-13523, 2022.

Mariem Mezghanni, Malika Boulkenafed, Andre Lieutier, and Maks Ovsjanikov. Physically-aware generative network for 3d shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9330-9341, 2021.

Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13492-13502, 2022.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Shentong Mo, Enze Xie, Ruihang Chu, Lanqing Hong, Matthias Niessner, and Zhenguo Li. Dit3d: Exploring plain diffusion transformers for 3d shape generation. Advances in Neural Information Processing Systems , 36, 2024.

Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4328-4338, 2023.

J Krishna Murthy, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jérôme Parent-Lévesque, Kevin Xie, Kenny Erleben, et al. gradsim: Differentiable simulation for system identification and visuomotor control. In International conference on learning representations , 2020.

Junfeng Ni, Yixin Chen, Bohan Jing, Nan Jiang, Bin Wang, Bo Dai, Yixin Zhu, Song-Chun Zhu, and Siyuan Huang. Phyrecon: Physically plausible neural scene reconstruction. arXiv preprint arXiv:2404.16666 , 2024.

Evangelos Ntavelis, Aliaksandr Siarohin, Kyle Olszewski, Chaoyang Wang, Luc V Gool, and Sergey Tulyakov. Autodecoding latent 3d diffusion models. Advances in Neural Information Processing Systems , 36:67021-67047, 2023.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Ryan Po and Gordon Wetzstein. Compositional 3d scene generation using locally conditioned diffusion. arXiv preprint arXiv:2303.12218 , 2023.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022.

Romain Prévost, Emily Whiting, Sylvain Lefebvre, and Olga Sorkine-Hornung. Make it stand: balancing shapes for 3d fabrication. ACM Transactions on Graphics (TOG) , 32(4):1-10, 2013.

Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In The Twelfth International Conference on Learning Representations (ICLR) , 2024.

Yi-Ling Qiao, Junbang Liang, Vladlen Koltun, and Ming C Lin. Efficient differentiable simulation of articulated bodies. In International Conference on Machine Learning , pages 8661-8671. PMLR, 2021.

Ri-Zhao Qiu, Ge Yang, Weijia Zeng, and Xiaolong Wang. Feature splatting: Language-driven physicsbased scene synthesis and editing. arXiv preprint arXiv:2404.01223 , 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.

Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Hyeonsu Kim, Jaehoon Ko, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint arXiv:2303.07937 , 2023.

Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems , 34:6087-6101, 2021.

Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model, 2023.

Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.16512 , 2023.

Jaehyeok Shim, Changwoo Kang, and Kyungdon Joo. Diffusion-based signed distance fields for 3d shape generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20887-20897, 2023.

Dule Shu, James Cunningham, Gary Stump, Simon W Miller, Michael A Yukish, Timothy W Simpson, and Conrad S Tucker. 3d design using generative adversarial networks and physics-based validation. Journal of Mechanical Design , 142(7):071701, 2020.

J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20875-20886, 2023.

Michael Strecke and Joerg Stueckler. Diffsdfsim: Differentiable rigid-body dynamics with implicit shapes. In 2021 international conference on 3D Vision (3DV) , pages 96-105. IEEE, 2021.

Priya Sundaresan, Rika Antonova, and Jeannette Bohgl. Diffcloud: Real-to-sim from point clouds with differentiable simulation and rendering of deformable objects. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 10828-10835. IEEE, 2022.

Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi. Viewset diffusion:(0-) imageconditioned 3d generative models from 2d data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8863-8873, 2023.

Tetsuya Takahashi, Junbang Liang, Yi-Ling Qiao, and Ming C Lin. Differentiable fluids with solid coupling for learning and control. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 6138-6146, 2021.

Boshi Tang, Jianan Wang, Zhiyong Wu, and Lei Zhang. Stable score distillation for high-quality 3d generation. arXiv preprint arXiv:2312.09305 , 2023.

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 , 2023.

Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22819-22829, 2023.

Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems , 36:13631389, 2023.

Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, Karsten Kreis, et al. Lion: Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems , 35:10021-10039, 2022.

Alexander Vilesov, Pradyumna Chari, and Achuta Kadambi. Cg3d: Compositional generation for text-to-3d via gaussian splatting. arXiv preprint arXiv:2311.17907 , 2023.

Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint arXiv:2403.12008 , 2024.

Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12619-12629, 2023.

Jianren Wang and Yihui He. Physics-aware 3d mesh synthesis. In 2019 International Conference on 3D Vision (3DV) , pages 502-512. IEEE, 2019.

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 , 2021.

Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201 , 2023.

Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems , 36, 2024.

Haohan Weng, Tianyu Yang, Jianan Wang, Yu Li, Tong Zhang, C. L. Philip Chen, and Lei Zhang. Consistent123: Improve consistency for one image to 3d object synthesis, 2023.

Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v (ision) is a human-aligned evaluator for text-to-3d generation. arXiv preprint arXiv:2401.04092 , 2024.

Tianyi Xie, Zeshun Zong, Yuxin Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. arXiv preprint arXiv:2311.12198 , 2023.

Qingshan Xu, Jiao Liu, Melvin Wong, Caishun Chen, and Yew-Soon Ong. Precise-physics driven text-to-3d generation. arXiv preprint arXiv:2403.12438 , 2024.

Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:2311.09217 , 2023.

Fan Yang, Jianfeng Zhang, Yichun Shi, Bowen Chen, Chenxu Zhang, Huichao Zhang, Xiaofeng Yang, Jiashi Feng, and Guosheng Lin. Magic-boost: Boost 3d generation with mutli-view conditioned diffusion. arXiv preprint arXiv:2404.06429 , 2024.

Jiayu Yang, Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong Li. Consistnet: Enforcing 3d consistency for multi-view images diffusion. arXiv , 2023.

Xiaofeng Yang, Yiwen Chen, Cheng Chen, Chi Zhang, Yi Xu, Xulei Yang, Fayao Liu, and Guosheng Lin. Learn to optimize denoising scores for 3d generation: A unified and improved diffusion prior on nerf and 3d gaussian splatting. arXiv preprint arXiv:2312.04820 , 2023.

Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. Physcene: Physically interactable 3d scene synthesis for embodied ai. arXiv preprint arXiv:2404.09465 , 2024.

Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided generation of 3d embodied ai environments. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024) , volume 30, pages 20-25. IEEE/CVF, 2024.

Jianglong Ye, Peng Wang, Kejie Li, Yichun Shi, and Heng Wang. Consistent-1-to-3: Consistent image to 3d view synthesis via geometry-aware diffusion models. arXiv preprint arXiv:2310.03020 , 2023.

Xuanyu Yi, Zike Wu, Qingshan Xu, Pan Zhou, Joo-Hwee Lim, and Hanwang Zhang. Diffusion time-step curriculum for one image to 3d generation. arXiv preprint arXiv:2404.04562 , 2024.

Xin Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Song-Hai Zhang, and XIAOJUAN QI. Text-to-3d with classifier score distillation. In The Twelfth International Conference on Learning Representations , 2024.

Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16010-16021, 2023.

Shutong Zhang, Yi-Ling Qiao, Guanglei Zhu, Eric Heiden, Dylan Turpin, Jingzhou Liu, Ming Lin, Miles Macklin, and Animesh Garg. Handypriors: Physically consistent perception of hand-object interactions with differentiable priors. arXiv preprint arXiv:2311.16552 , 2023.

Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T Freeman. Physdreamer: Physics-based interaction with 3d objects via video generation. arXiv preprint arXiv:2404.13026 , 2024.

Minda Zhao, Chaoyi Zhao, Xinyue Liang, Lincheng Li, Zeng Zhao, Zhipeng Hu, Changjie Fan, and Xin Yu. Efficientdreamer: High-fidelity and robust 3d creation via orthogonal-view diffusion prior. arXiv preprint arXiv:2308.13223 , 2023.

Chuanxia Zheng and Andrea Vedaldi. Free3d: Consistent novel view synthesis without 3d representation. arXiv , 2023.

Xin-Yang Zheng, Hao Pan, Peng-Shuai Wang, Xin Tong, Yang Liu, and Heung-Yeung Shum. Locally attentional sdf diffusion for controllable 3d shape generation. ACMTransactions on Graphics (SIGGRAPH) , 42(4), 2023.

Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. In CVPR , 2023.

Joseph Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d with advanced diffusion guidance. arXiv preprint arXiv:2305.18766 , 2023.

A Appendix

A.1 Score Distillation Sampling

Score Distillation Sampling (SDS) has proven to be an effective method for distilling 3D models from 2D images. SDS-based methods leverage a combination of 3D representations, differentiable rendering, and pre-trained text-to-image diffusion models to optimize 3D shapes with high fidelity and realism.

The generation process in SDS methods involves optimizing the parameters θ of the 3D representation. At each iteration, the 3D geometry is differentiably rendered into a 2D image z = g ( θ ) . This image is then compared with the distribution of real images as modeled by the diffusion model. More specifically, the input image z is first noised with a random noise ϵ at a specified noise level t . The diffusion model then predicts the noise ˆ ϵ ϕ with text guidance y , and this prediction is compared with ϵ , resulting in the following loss:

where w ( t ) is a weighting function modulating the influence of different noise levels. The expectation is taken over the noise level t and noise term ϵ , ensuring that the generated 3D shape aligns with the text-guided distribution of images.

The gradient with respect to the optimization parameter θ is then backpropagated as follows:

where the U-Net Jacobian term ∂ ˆ ϵ ϕ ∂ z t is omitted for efficient optimization.

For further technical details on SDS optimization, we refer readers to [58; 35; 46].

A.2 Implementation and Training Details

We implement our pipeline in PyTorch with Adam optimizer. For differentiable simulation, we adopt the semi-implicit Euler simulator in Warp [44], where gradients of physical states are computed via auto-differentiation and backpropagated to the parameters of 3D representations [44; 56]. We use a two-stage training strategy and leave the choice of the first stage open for various SDS-based methods as baselines. For a two-stage baseline method, we implement our physics-inspired losses as submodules that can be seamlessly integrated into the refinement stage. For a one-stage method, our approach can be used as a standalone refinement stage to improve the physical quality of the baseline. For baseline methods that are not publicly available, we use the reimplementation from threestudio [14].

We train our models using a single NVIDIA RTX 3090 GPU. During our refinement stage, we implement a skipping strategy, incorporating standability loss only once every 10 iterations. In our quantitative evaluation of a batch of prompts, we observed an average refinement time of 36 minutes for each training step, with a default setting of 5,000 iterations.

For the rigid body simulator in Warp, we set dt = 10 -3 s. Contact stiffness and damping are set to 10 3 and 2 . 0 ; friction coefficient is set to 0 . 5 ; stiffness of friction force is set to 10 3 . Density of the 3D objects is set to 10 3 .

In our experiments, we use the following default weights for the loss terms: { λ SDS = 1 , λ normal = 10 4 , λ stand = 10 5 , λ stable = 10 5 , λ b-lap = 10 7 } . For some examples, we tune these weights within the following ranges: { λ stand = 10 5 ∼ 5 × 10 5 , λ stable = 10 5 ∼ 5 × 10 5 , λ b-lap = 10 6 ∼ 10 7 } . Our heuristic intuition is to keep the SDS and physical loss terms roughly on the same scale. For the regularization terms, we scale them to around 1 / 1000 to 1 / 100 of the SDS and physical loss terms.

A.3 Comparison with Post-processing Methods

While directly post-processing 3D generated models is a straightforward and effective approach to achieving physical stability, it may result in undesirable outcomes such as misalignment with the text prompt, as it overlooks semantics.

One simple post-processing method is to cut the mesh by a flat plane slightly higher than the lowest vertex. However, this method will fail when the projection of the center of mass lies outside the contact region, as shown in Fig. 11. Additionally, determining the cutting height is another parameter that must be manually set or optimized, and this may also degrade the overall appearance.

Ours

z = 0.05

z = 0.10

z = 0.20

z = 0.15

Another post-processing method, make-it-stand [59], offers an effective way to relocate the center of mass to achieve standability. However, it assumes that the supporting surface is fixed during optimization, which can lead to distorted results due to the imperfect quality of text-to-3D generated models. We provide two examples in Fig. 12. For the goose example generated by Magic3D, one leg is shorter than the other. Make-it-stand only treats one foot as the supporting surface and ignores the other due to its post-processing nature, whereas our joint-optimization pipeline enables stable standing with two legs on the ground. A similar issue also happens to the kangaroo example. More importantly, the text alignment degrades as semantics are overlooked during optimization. In contrast, our proposed joint optimization method preserves the text alignment and dynamically adjusts the center of mass as well as the supporting surface configuration.

Ours

Ours

Make-it-Stand

Make-it-Stand

In Fig. 6(e) in the main text, we also apply our proposed losses in a post-processing manner. Similarly, while it is able to optimize the 3D models to make it standable, the text alignment is compromised. Overall, compared to the post-processing method, joint optimization is a more robust way which better balances text alignment and physical constraints.

A.4 Additional Results

A.4.1 Qualitative Comparison with Magic3D Baseline

[…] a

standing

goose

Ours

Magic3D

[…] a

standing

pigeon

[…] a

jumping

kangaroo

[…] a

mannequin

[…] a

standing

Konan

[…] a

standing

ultraman

[…] a toy

robot

Ours

Magic3D

[…] a robot

made out of

vegetables

[…] a goose

made out of

gold

[…] small

saguaro

cactus

planted in a

clay pot

[…] a baby

dragon

hatching out

of a stone

egg

[…] a bear

playing

electric bass

1018

Ours

Magic3D

A Marble

bust of a

A palm tree,

low poly 3d

model

A dachshund

dressed up in

a hotdog

costume

style statue of

an astronaut

[…] front

view of an

easter egg

[…] a

mushroom

A.4.2 Qualitative Comparison with MVDream Baseline

A detective

Conan

Ours

MVDream

A standing

kid

Mouse, […]

A.4.3 Mesh Topology Change

We provide a zoomed-in view of the local mesh topology change in Fig. 17 to demonstrate how our method optimizes the object's geometry to make it standable.

w/ Atlas3D

w/o Atlas3D

A.4.4 Quantitative Comparison with Magic3D Baseline

The results of the quantitative experiments are shown in 2. For CLIP score calculation, we specifically employ openai/clip-vit-large-patch14-336 as the check point of CLIP model. We rendered images from various angles (0-360 degree, 3 degree as the interval). For Elo (GPT-4o) benchmark, we made use of the newly released GPT-4o model instead of the one in [90] due to capacity limitation. Additionally, we reduce the number of views to 6 because of token length limitation.

A.5 Real-world Experiments

A.5.1 Robot Manipulation Results 6

A

A

A

A

A

T

究

T

We record our robot manipulation experiment in Fig. 18 and summarize quantitative results in Table. 3.

A.5.2 User Study Results

We report the detailed results of our user studies in Table. 4.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims are meticulously demonstrated in the "Experiments" section. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the work are addressed in the "Discussion" section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper includes sufficient details of the experimental settings in the "Experiments" section to ensure reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The authors provide code to reproduce results in the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details in the "Experiments" section. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports appropriate information about the statistical aspect in the "Experiments" section.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides sufficient information about computer resources in the "Experiments" section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper conforms with the Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no new societal impact compared to previous generative models on similar topics, as the improvement in quality does not introduce additional uncertainty.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All existing assets used are clearly cited or credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to combine physical standability with existing diffusion-based text-to-3D generators in order to synthesize objects that not only follow the text description but also can stand on their own without falling. To this end, the paper proposes a physical loss function and uses it with score distillation sampling methods that generate 3D objects from texts. The paper shows that their generated model show better standability both in simulation and after physical realization.\n\nSTRENGTHS:\n- Good exposition and writing, paper is rather easy to follow. \n- The validation of results especially the 3D printing of generated models. \n- The standability and stable equilibrium losses are elegant.\n\nWEAKNESSES:\nFirst, there are good methods, such as 'make it stand' that could be used as post-processing tools for making text-based generated models stand. Why should we go through the paper's approach in the light of these methods? \nSome important ablations are missing. For example,\n- Separating of how much each loss is contributing to standability. For example, what if we use Magic 3D outputs and smoothen the geometry? \n- What are the pros and cons of using the physical loss as a post-processing or jointly with SDS? \n\n\nThe claims are more broad than the actual contributions. \n- Take \"fabrication-ready\" in line 62. The paper is concerned with stabdability and it should adapt the text (and even the title). In 3D printing, \"self supporting\" means that there is no need to support materials. The paper doesn't cover this. \n- The relationship to robotics is not well-founded except an example that uses robot grippers to do the standability tests, which is not really \n related to benefits to robotics. \n\nThe paper is heavily relied on score distillation sampling methods but does not explain them very well. I understand that the paper refers to the original papers in this area but it is worth including a paragraph about SDS. \n\nI look forward to see a justification in the rebuttal.\n\nQUESTIONS:\n- The paper mentions too many steps of integration and back propagation in Section 4.1.1. Does it still do this? If not, where exactly the differentiable physics simulation is used? \n- What is the default intimal physical state? \n- How does the paper figures out the upright position? \n- The paper mentions mesh topology (Line 246) but doesn't show it. A zoom in mesh is necessary for this. \n- What are the mesh sizes and dimensionalities of printed objects? \n- Are Figure 7 results printed? \n- x_{com} is not defined in Eq. 7.\n\nLIMITATIONS:\nA main limitation is the mesh size. The paper has mentioned it. This is important in the light of 3D printing that require sometimes huge resolutions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to make 3D assets produced by SDS-based generative models stand on their own. On top of the SDS loss, it proposes a “standability\" loss that encourages the rigid-body simulation result to be rotation-free, and a “stable” loss to encourage the generated shape to be a “local minimum” of height for the center of mass. The evaluation is conducted both in simulation and 3D-printed shapes in the real world.\n\nSTRENGTHS:\n- This paper addresses an interesting problem of making 3D-generated shapes stand. Although this problem has been studied from the perspective of 3D printing, this paper introduces new methods tailored to the new setting of SDS-based 3D generation.\n- The proposed method seems to be effective from the various presented results, with a significant improvement in the success rate of standing.\n\nWEAKNESSES:\n- The proposed method involves solving an optimization that involves multiple loss terms. No ablation study is provided to prove the necessity of each term. In particular, are both standability loss and stable loss required? How are their influence on the optimization results respectively? Are both normal smoothing Eq (8) and Laplacian smoothing Eq (9) required and why is Laplacian smoothing only applied to the bottom of the shape?\n- The traditional balancing method in computational fabrication [59] is only discussed but not actually compared in experiments.\n- What are the loss weights for each term in Eq (10)? These numbers may be tricky to set, and thus make the paper hard to reproduce.\n- How is \\(T\\) set in Eq (5)? How do different choices of \\(T\\) influence the results?\n- The success rate of the baseline method is not reported in Fig. 6 for the audience to understand the improvement.\n\nQUESTIONS:\nSee weaknesses. These are important questions to answer for a well-rounded paper.\n\nLIMITATIONS:\nSeems adequate.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a differentiable simulation-based loss to refine the existing SDS(Score Distillation Sampling)-based text-to-3D frameworks. Concretely, it relies on a differentiable simulator Warp to provide gradients for keeping the rotation of the generated mesh unchanged after a period of time. Besides, the authors also introduce a stable equilibrium loss, which favors a flat contact surface. Experiments show that the proposed method can generate self-supporting 3D models given text prompts, verified by simulation and real-world results with 3D-printed objects.\n\nSTRENGTHS:\nThe paper introduces an effective and flexible method to refine the existing SDS(Score Distillation Sampling)-based text-to-3D frameworks, like Magic3D and MVDream. The authors also provide experiments on real-world validation.\n\nWEAKNESSES:\n1. The paper lacks enough baselines to show that the task the paper tackles is non-trivial. For example, the claim that \"Directly integrating these methods with 3D generative AI as a postprocessing module is suboptimal\" (L55-L56) is not verified by the experiments. If a self-supporting mesh is needed, the simplest baseline can be cutting the mesh by a flat plane moderately higher than the lowest vertex, as a postprocessing step.\n2. The physical constraints used in this work are relatively limited. The standability loss only keeps the rotation unchanged. It can not recover some common cases like \"a standing horse\" (e.g., Napoleon Crossing the Alps), although those cases may be beyond \"self-supporting\" defined in this paper. It seems that the simulator has not been fully leveraged, since a simple post-processing baseline (e.g., flatting surfaces that contact with the ground) may also achieve good standability and stability.\n3. The current method is only applied to SDS-based methods only (already pointed out by the authors in the limitations)\n\nQUESTIONS:\n1. Can the authors provide more baselines, especially any existing post-processing method with the help of differentiable simulators, to enhance their technical contributions? One example is given in the *Weaknesses* section.\n2. Can the authors give a clear definition of \"self-supporting\"? It is a little ambiguous, since one can replace the initial pose with the pose after simulation so that Eq. (3) about standability can be satisfied, as long as the limit exists.\n\nLIMITATIONS:\nThe paper has addressed its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of generating 3D models from text that are visually appealing but often physically unstable in simulations or when 3D printed. The authors incorporates a differentiable simulation-based loss function and physically inspired regularization to ensure generated models are stable under perturbation. The proposed method involves a two-stage training process where a coarse model is first generated from text prompts and then refined with physical constraints to ensure stability. Experiments show that models created with Atlas3D maintain stability better than those from existing methods. The optimized models are also 3D fabricated to verify the stability in real world.\n\nSTRENGTHS:\nThe paper proposes an original and important solution to the issue of physical stability in text-to-3D generation, combining differentiable simulation-based loss functions with physically inspired regularization. This approach significantly enhances existing methods, reducing the need for manual post-processing and making the models immediately practical for various applications.\n\nThe method is sound, with clear and concise writing that guides the reader effectively and should be straightforward to implement for someone with a background in physical simulation. The experiment quality is high, validated through extensive experiments in both simulations and real-world scenarios, demonstrating the method's robustness and versatility. The results confirm the method's effectiveness in ensuring the stability of 3D models.\n\nWEAKNESSES:\nThe paper is strong overall, and I did not identify any significant weaknesses. However, to further support future research in this area, I would suggest releasing the code. Providing the implementation would greatly benefit the community, enabling others to replicate the results and build upon this work.\n\nQUESTIONS:\nI don't have any specific questions at this time. The paper is clear and well-executed, addressing an important issue effectively. Implementation is also described in detail.\n\nLIMITATIONS:\nI am curious about the failure modes of Atlas3D. Could you provide more details on the scenarios or conditions under which the method might not perform as expected? Understanding these limitations would be helpful for future research and applications. Additionally, addressing these potential failure modes could further strengthen the robustness of your approach.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}