Physically Compatible 3D Object Modeling from a Single Image
Authors: Minghao Guo
Bohan Wang
Pingchuan Ma
Tianyuan Zhang
Crystal Elaine Owens
Chuang Gan
Joshua B. Tenenbaum
Kaiming He
Wojciech Matusik
Venue: NeurIPS 2024 spotlight (2024)
Keywords: single-image modeling
physical compatibility
Primary area: machine_vision
TL;DR: We present a physical compatibility optimization framework that transforms single images into 3D physical objects, significantly enhancing stability and structural integrity under real-world physics.

Abstract: We present a computational framework that transforms single images into 3D physical objects. The visual geometry of a physical object in an image is determined by three orthogonal attributes: mechanical properties, external forces, and rest-shape geometry. Existing single-view 3D reconstruction methods often overlook this underlying composition, presuming rigidity or neglecting external forces. Consequently, the reconstructed objects fail to withstand real-world physical forces, resulting in instability or undesirable deformation -- diverging from their intended designs as depicted in the image. Our optimization framework addresses this by embedding physical compatibility into the reconstruction process. We explicitly decompose the three physical attributes and link them through static equilibrium, which serves as a hard constraint, ensuring that the optimized physical shapes exhibit desired physical behaviors. Evaluations on a dataset collected from Objaverse demonstrate that our framework consistently enhances the physical realism of 3D models over existing methods. The utility of our framework extends to practical applications in dynamic simulations and 3D printing, where adherence to physical compatibility is paramount.

Physically Compatible 3D Object Modeling from a Single Image
Minghao Guo', Bohan Wang"* Pingchuan Ma¹ Tianyuan Zhang', Crystal Elaine Owens', Chuang Gan Joshua B. Tenenbaum 1,4,5 Kaiming He', Wojciech Matusik
MIT CSAIL, UMass Amherst, 'MIT-IM Waston Laba, 'MIT BCS,
Center for Brains, Minds and Machines httpa:/ /tgpa:///mmha 14. gi thub. io/phys--ae comp/
3DD Objects w/o Physical Compatibility
Ieput Images
Stiff Objects
Soft Objects
3D Objects w/ Physical Compatibility
Figure 1: Existing methods for single-view reconstruction often result in objects that, when subjected to real-world physical forces (such as gravity) and user-required mechanical materials, exhibit problematic behaviors such as toppling over (top left) and undesirable deformation (top right), diverging from their intended depiction in the input images. In contrast, our approach produces physical objects that maintain stability (bottom left) and mirror the objects' static equilibrium state captured in the input images (bottom right).
Abstract
We present computational framework that transforms single images into 3D phys- ical objects. The visual geometry of a physical object in an image is determined by three orthogonal attributes: mechanical properties, external forces, and rest-shape geometry. Existing single-view 3D reconstruction methods often overlook this underlying composition, presuming rigidity or neglecting external forces. Con- sequently, the reconstructed objects fail to withstand real-world physical forces, resulting in instability or undesirable deformation diverging from their intended designs as depicted in the image. Our optimization framework addresses this by embedding physical compatibility into the reconstruction process. We explicitly decompose the three physical attributes and link them through static equilibrium, which serves as a hard constraint, cnsuring that the optimized physical shapes ex- hibit desired physical behaviors. Evaluations demonstrate that our framework consistently enhances the physical realism of 3D models over existing methods. The utility of our framework extends practical applications in dynamic simulations and 3D printing, where adherence to physical compatibility is paramount.
dataset collected from Objaverse
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
The field of single-image 3D shape modeling has experienced significant advancements over the past years, largely propelled by advances in single-view reconstruction techniques. These methods, ranging from generating view consistent images for per-scene 3D reconstruction [22, 23, 21, 24, 32, 20], to employing large reconstruction models (LRMs) for feedforward inference [12, 41, 44, 50, 47, 40], have enhanced the geometric quality and visual fidelity of the 3D shapes to unprecedented levels.
However, reconstructing a 3D shape from an image often aims to be beyond a mere visualization. These generated objects find applications in virtual environments such as filming and gaming, as well as in tangible fields like industrial design and engineering. Despite their diverse applications, common oversight in many current single-view reconstruction methods is the negligence of physical principles. As shown in the top row of Fig. 1, when subjected to real-world physics such as gravity, these 3D objects produced from these techniques exhibit issues such as instability and undesired deformation, diverging from their depiction in the input images. Such inconsistency can significantly undermine the practical utility of the models, as they fail to meet the functional and aesthetic expectations set by the original image.
Poece
Fundamentally, an image is more than a visual representation of an object: It captures a physical snapshot of the object in a state of static equilibrium, under the influence of real-world forces. In this context, the geometry seen in an image is deter- mined by three orthogonal attributes: mechanical properties, external forces, and rest-shape geometry. As shown in the inset figure, these attributes collectively model the spectrum of potential static configurations that physical object might adopt. Reconstructing such an object from an image is es- sentially an ill-posed problem, since multiple combinations of these attributes can result in identical static geometry. Current methods, however, often overlook this underlying composition; they typically assume objects are rigid or neglect the impact of external forces. The reconstructed objects thus merely replicate the visual geometry without considering the three physical attributes.
wwo.garity
Material
Goometly
To bridge this gap, we explicitly decompose these attributes for reconstructing a physical object from single image. Our framework holistically takes mechanical properties and external forces as predefined inputs, reflecting typical user specifications in real-world applications like 3D printing and simulations. The output is the rest-shape geometry tailored to these inputs. These attributes are integrated through the principles of static equilibrium physics. This explicit decomposition imposes two stringent physical constraints in object modeling: static equilibrium is enforced as a hard constraint, and the physical object must conform to user-specified material properties and external forces. These resulting physical objects are stable, robust under real-world physics, and are high-fidelity replicas inferred from the input images, as shown in the bottom row of Fig. 1.
More specifically, we propose physical compatibility optimization, which is physically constrained optimization with rest-shape geometry as the variable. In this setup, the objective is for the modeled physical object to exhibit desired behaviors, such as matching the geometry depicted in the input image under external forces and maintaining stability under gravity. The constraint the equation of static equilibrium simulation, ensuring that during optimization, the physical object remains in the equilibrium state, with internal forces generated by deformation from the rest shape balancing the external forces. We parameterize the rest-shape geometry using a plastic deformation field and solve this constrinnte optimization problem by using implicit differentiation with gradient descent.
For evaluation, we introduce five metrics designed to comprehensively assess the physical compati- bility of the modeled 3D objects under simulation. These metrics include image loss between the rendered image of the modeled physical object and the input image, stability under gravity, as well as measures from finite element analysis, such as integrity and structural robustness. Our framework's 'ss versatility is demonstrated by its integration with five distinct single-view reconstruction methods, each employing unique geometry representations. Results on a dataset collected from Objaverse [9], consisting of 100 shapes, show that our framework consistently produces 3D objects with enhanced physical compatibility. Furthermore, we demonstrate the practical utility of our framework through applications in dynamic simulations and 3D printing fabrication.
2

---

2 Related Work
Single-view 3D reconstruction. Recent strides in single-view 3D reconstruction have mainly been fueled by data-driven methods, paralleled by advancements in 3D geometry representation, including NeRF [27], NeuS [43], triplanes [33], Gaussian splatting [16], surface meshes [29], and tet- spheres [1]]. These developments have significantly enhanced the geometric quality and visual fidelity of the reconstructed 3D shapes. There are primarily two types of single-view reconstruction methods: 1) Test-time optimization-bsset methods [31, 23, 39, 45], use multiview diffusion models [21] and iteratively reconstruct 3D scenes using these diffusion priors. 2) Feedforward methods [13, 48, 38, 7, 44, 50] leverage large datasets and learn general 3D priors for shape reconstruction enable efficient one-step 3D reconstruction from single or sparse views. Unlike the aforementioned methods, work emphasizes the integration of physical modeling into the reconstruction process. This integration distinguishes our work by ensuring that the resulting 3D shapes are not only visually accurate but also physically plausible under real-world conditions.
Physics- 3D modeling. There has been an increasing interest in incorporating physics into 3D shape modeling. While many approaches utilize video input, which offers richer temporal context for inferring physical properties such as material parameters [51] and geometry [19], others approach the problem by first reconstructing
object's geometry from multi-view images and subsequently applying physical simulations [10, 46, 26, 25]. Additionally, several studies have explored extracting physical information from static images [49, 3, 37], using data-driven techniques to estimate properties like shading, mass, and material. In contrast, our work incorporates physical principles, specifically static equilibrium, hard constraints within the reconstruction process. This integration allows for the optimization of 3D models that adhere to desired physical behaviors depicted by the image.
Fabrication-aware shape design. Originating from the computer graphics community, fabrication- aware shape design systems enable designers to specify higher-level objectives such as structural integrity, deformation, and appearance with the final shape as the output of the computational system [4]. Related methodologies in this domain, particularly those addressing static equilibrium, include inverse elastic shape design [8] and sag-free initialization [14]. However, these approaches typically require manually created initial geometry, whereas our work aims to construct the physical object directly from single input image.
3 Approach
Our objective is to create 3D objects from a single image that are physically compatible, ensuring that they align with the input image in the static equilibrium state while also meeting the stability requirements. Governed by universal physical principles, the physical behavior of an object is determined by its mechanical properties, external forces, and rest-shape geometry. Our framework treats the rest-shape geometry as the optimization variable, assuming that the mechanical properties and external forces are predefined as inputs. Fig. 2 illustrates the overall pipeline.
3.1 Formulation of Physical Compatibility
In our approach, we treat the entity depicted in the input image as a solid object. We employ Finite Element Method (FEM) for robust solid simulation. The object is represented by volumetric mesh denoted as M = (x, T). Here, x E R represents the 3D positions of the vertices, with N denoting the total number of vertices. T E NZ*K describes the mesh connectivity, where Z represents the total number of elements and indicates the number vertices per element. The mesh in its rest-shape geometry, which is the state without any internal or external forces applied, is represented as Mrest (Xrest, T). The input image depicts the static geometry, which is the deformed geometry of the object under static equilibrium', denoted MMsattic (Xstatic, T). In accordance with Newton's 's laws, Xstatic adheres to the following equation:
firt (Xstatic; Xyest θ = fext (Xstatio),
(I)
Although our implementation employs quasi-static equilibrium, we use the term static equilibrium across the paper for consistency.
3

---

Physical Compatibility Optimization
Statie Geometry
Matching Loss
Initial Geumetry
Plastic
(0)
(b)
Input
Deformation
Young's Moduise E -
Xinis
F
Poisson's Ratio 11.46
Denity: 102 logg fmx
Mechanical Material Properties
(b)
@
Stability Loss
Fex (Xstasics Xynsi 0) for (Nanis))
Statie Equilibrium
Grevay: 9.8 un/a?
Rest-shape Geometry
External Forces
Stress (Pa)
Support Polygon
10
10" 30" 10'
Figure 2: Overall pipeline. Given predefined mechanical properties and external forces, our pipeline optimizes the rest-shape geometry to ensure that the shape, when in a state of static equilibrium, aligns with the target image and meets stability criteria. We visualize the stress distribution of the static geometry using colored heat map, illustrating the spatially varying deformation of the physical object under static equilibrium.
where fim (: : 13N x R3N + RAN denotes the internal forces exerted by deformed objects transitioning from Xrest to Xstatics f-xi()) RON - R3v embodies the external interaction forces such as gravity, and è represents the mechanical material properties, such as the stiffness of the object. Eq. 1 reveals that è (mechanical properties), fext (external forces), and X (the rest-shape geometry) collectively determine the static geometry Xstatic-
Given O and (-xa(-), the goal of physically compatible modeling is to ensure that the rest-shape geometry Mrest conforms to given objectives under static equilibrium. This formulated as the following optimization problem:
min yecat Matatic
J(Xrest Xstatic) L(xstaatie) + Freg (Xrest)
8.b. Fint Gint(xtttte: Xrest; θ) fext. (Xstatic).
(2)
Here, J [[rrett Xstatic) is the objective function, consisting of C(Xstatie), which measures the align- ment of the geometry Xstatic with the specified target. Lreg( (Xrest)) regularizes the rest-shape geometry Xrest, with more details discussed in Section 3.2.
Within the scope of this work, two tasks for L(xstatie) considered: 1) Xstatic replicates the geome- try depicted in the input image; and 2) Xstatic maintains stability and inherently remains upright with- toppling. In the first scenario, the loss function is [(xstatic) ||xtaatic Xtarget 11 which mea- sures the point-wise Euclidean distance between the static shape and the target geometry Mtarget (Xtarget: T). In the second scenario, the loss function is C(xstatic) where C(.) computes the center of mass of Mstatic. (roj_()) denotes the projection of the center onto the z-plane in world coordinates, and Ć represents the target position for the center of mass to guarantee stability. Minimization of this function ensures the structural stability of Metatic-
||rroj (rro)_(C(xxee (C(xatatie) ClI.
is crucial to highlight that the variables Xrest and Xstatic are tightly coupled through a hard constraint in problem formulation. This constraint, which ensures that the object remains static equilibrium, is essential to achieving physical compatibility. Enforcing this configuration guarantees that the 3D physical object conforms strietly external forces such gravity, thereby ensuring the system adheres to the inherent physical constraints.
3.2 Parameterization of Rest-shape Geometry
To solve the optimization problem defined Eq. 2, one might consider straightforward approach by directly treating X,ett. as the optimization variable. However, this brings challenges in maintaining the physical validity of the rest-shape geometry, i.c., there shall be no inversions or inside-out elements.

---

This non-inversion requirement is typically enforced through nonlinear inequality constraints [6], leading to intractable optimization. Drawing inspiration from natural modeling processes [42], we propose parameterization of rest by treating it as the result of plastic deformation applied to an initial configuration. A plastic deformation can transform objects without the volume preservation constraint [1]. Specifically, we denote the initial configuration of the rest-shape geometry as Mint (Xinit, T). Xrest is implicitly parameterized by the plastic deformation field Fp as
Festt := OFFp: p: Xinit), with fine (Xrest: imit: ( - =.
(3)
Intuitively, this equation suggests that Xrest results from applying plastic strain field Fp X without any external forces. The plastic strain field Fp is the collection of transformations, with each transformation is an R3x3 matrix applied to each material point. Throughout this paper, we also represent plastic deformation in its vector form as Fp R which corresponds to the flattened vector form of the 33x3 transformation collection. For detailed explanation of the computation of Xrest from Fp and its integration into the static equilibrium, we refer the reader to Appendix B.
There are several benefits using Fp for parameterizing rest-shape geometry: It exhibits invariance to translation, which ensures that the spatial positioning of Xint does not affect the deformation outcomes. Moreover, the non-inversion requirement can be efficiently satisfied by constraining the singular values of F.. Fp. thereby avoiding the need for complicated inequality constraints. Appendix B provides a comprehensive analysis of these advantages.
By substituing Eq. 3, we reformulate the optimization problem Eq. 2 as follows:
min F F..Sticic
[(F Xstatic) = [(xstatic) + Lrag! (Fp)
8.1. fin (Xetatic (Fpp: X min); 0) fext (Xstatic)
Here, the optimization variables are Fp. where the initial geometry configuration Xint is treated as constant. The regularization term (Fp) defined as the smoothness of plastic deformation using bi-harmonic energy [5], represented as Lreg (F = |LFpl where L 9 denotes the graph Laplacian matrix, encapsulating the connectivity of the volumetric mesh elements.
3.3 Implicit Diffcreentiaton--aae Optimization
Solving the optimization problem in Eq. 4 is non-trivial due its nonlinear objective and the nonlinear hard constraint. A straightforward approach is incorporating the constraint directly into the objective as additional loss term; however, this method may lead imperfect satisfaction of the constraint, which undermines the fundamental goal of ensuring physical compatibility.
We resort implicit differentiation, technique used in sensitivity analysis [6], to compute the gradient of the objective function J with respect to the variable Fp. This approach effectively reduces the dimensionality of the optimization variables since we only need to calculate the gradient with respect to Fp and also ensures that the gradient direction takes into account the hard constraint. Specifically, the gradient is computed as follows:
0J OF,
( OC
Enot
Or OF, p OF,
0Lreg
ixxtaticc
@xxtaticc
(5)
where fnet fint fext represents the net forces. A comprehensive derivation of this gradient formula is provided in Appendix C. By utilizing this gradient, the optimization can be solved using standard optimization tools, such the Adam optimizer [17]. This facilitates the integration of our method into existing single-view reconstruction pipelines.
3.4 Implementation Details
Given an input image, we initially utilize off-the-shhe single-view reconstruction models to obtain the 3D object's target geometry, ensuring alignment with the input image. The output of these reconstruction models varies depending on the geometric representation used. For instance, methods employing tetrahedral representations, such as TetSphere [11], yields volumetric meshes that can be directly used as Mearget. Conversely, methods that output surface meshes [44] or point clouds [40], which are often non-volumetric and typically mon-manifold, require additional processing steps to be suitable for our computational pipeline. We use TetWild [15], robust tetrahedral meshing algorithm,
5

---

to convert these unstructured outputs into high-quality tetrahedral meshes, resulting in volumetric mesh Margett. For initiating the optimization process, we set Minit Margget assuming that Mtarget is a reasonably good initial approximation for the optimization. Note that Minit is not strictly confined to Mearget; any volumetric mesh could potentially serve as the initial approximation, given the flexibility of F P to accommodate spatially varying deformations.
For the material constitutive model, we use isotropic Heo-Hookean material as detailed in [35]. The mechanical properties O including Young's modulus E, Poisson's 's ratio v, and mass density are set by users. These values can be specified directly through numerical input or chosen from a collection of pre-established material options, such as plastic or rubber. We consider gravity and fixed attachment forces as options for external forces. Gravity is always included reflect its omnipresence in the real world. The use of fixed attachment forces depends on the specific needs of the application, for instance, anchoring an object a designated site. Detailed formulations for both force types are provided in Appendix F.
4 Evaluation
In this section, we present evidence that our approach enhances the physical compatibility of 3D objects produced using state-of-the-art single-view reconstruction techniques. We conduct series of quantitative evaluations using five metrics (Sec. 4.1) to compare the physical compatibility of shapes optimized by our framework against those produced by existing methods without our method (Sec. 4.2). We also provide qualitative comparisons to demonstrate to the effectiveness of our approach (Sec. 4.3). Furthermore, we explore the practical applications of method by illustrating how it enables the reconstruction of diverse 3D shapes with different material properties from the same single image, and by demonstrating that our optimized shapes are readily adaptable for dynamic simulations and fabrication (Sec. 4.4).
4.1 Baselines and Evaluation Protocol
Existing metrics for evaluating single-view reconstruction methods primarily focus on the visual appearance of the objects. Measures such as PSNR and SSIM are used to assess image fidelity, while chamfer distance and volume IoU evaluate geometric quality. However, these metrics do not consider the underlying physics principles that govern the behavior of 3D objects. Consequently, they are insufficient for evaluating the physical compatibility of reconstructed shapes, a crucial aspect for applications requiring accurate physical interactions and structural stability.
Metries. address this oversight, we draw inspiration from the field finite element analysis [2] and introduce five novel metrics specifically designed to assess the physical compatibility of 3D models comprehensively. These metrics are tailored to ensure more thorough evaluation of method performance in real-world scenarios with rich physics:
Number of Connected Components (# CC.) evaluates the structural integrity of the object. Physical objects should not have floating or disconnected structures, idcally consisting of one single connected component.
Mean Stress calculates the average von Mises stress [28] across all tetrahedra of all objects. It measures the extent of physical deformation. Under the same external interactions, higher mean stress indicates a greater likelihood of fracture and the existence of unrealistic thin structures. Percentage of Standability (Standable.) assesses whether the object can maintain stability under gravity, remaining upright without toppling. A standable object is one that effectively supports itself against gravitational forces.
Matching loss (Img, Loss) calculates the I difference between the rendered image of the object after applying gravity and the input target image, quantifying the deviation of the physical object from the desired shape due to physical influences.
Fracture Rate measures the number of tetrahedral elements that exceed a predefined stress threshold, potentially leading to fractures. The resilience of method against physical stresses is quantified using a degradation curve, with more physically reliable methods exhibiting a smaller area under the curve for the fracture rate.
Baselines. We consider five single-view reconstruction baselines in our evaluation, each associated with a distinct geometry representation: Wonder3D [23] with NeuS, LGM [40] with Gaussian

---

Table I: Quantitative results four metrics evaluating physical compatibility. We apply our pipeline to five single-image reconstruction techniques and assess our metrics on both the initial shapes from these methods (Baseline) and the optimized shapes from the integration of our framework with each baseline (Ours). Our method demonstrates quantitative improvements in mean stress, stability rate, and image fidelity across all benchmarks. Among all methods, TetSphere integrated with our framework achieves superior performance across all evaluation metrics. This can be attributed to the explicit volumetric representation used in TetSphere. The mean and standard deviation are calculated across all examples for each method. A higher deviation in Mcan Stress suggests a larger variance in structural thickness and curvature, while a higher deviation in Img. Loss indicates a larger variance in static shape deformation.
Mean Stress (kPu)
Standable. f (%)
Method
Init. Goo. #CC. +
Img. Loss 1
Wonder3D
Baseline
NeuS
2.54 2.64
10.68 + 17.47
6.9
0.073 + 0.063
Ours
0.45 + 0.96
72.4
0.069 + 0.048
Baseline Ours
Gaussian
1.14 2.03
20.3
0.121 + 0.091
LGM
2.67 + 2.13
splatting
1.01 1.34
85.5
0.116 + 0.065
Baseline
surface mesh
0.54 1.41
29.6
0.065 + 0.042
MeshI.MM
1,55:t 2.13
Ours
0.38 1.05
74.5
0.064 + 0.042
TripoSR
Baseline
NeRF
1,43 +: 1.12
0.29 + 1.28
24.2
0.066 + 0.047
Ours
0.22 0.94
80.6
0.089 + 0.039
Baseline
0.22 + 0.51
32.8
0.061 + 0.045
TetSphere
tet. set-sp
1.00 0.00
Ours
0.19 + 0.78
92.2
0.057 + 0.040
Wonder3D
LGM
Meshl .RM
TripoSR
TetSphere
Fracture Rate (%)
Threshold (Pa)
Threshold
Threshold (Pa)
Stress Threshald
Strees Threshold
wis opeieazzation
WT oppttstzaaionn
Figure 3: Quantitative results on fracture rate. We plot the relationship between the fracture rate and the maximum stress threshold across five single-image reconstruction methods. The shapes optimized with our framework exhibit consistently lower fracture rate compared to those shapes obtained without our pipeline. McshLRM and TripoSR feature prevalent thin structures in their reconstructed shapes, whereas our approach significantly reduces the fracture rate in both cases.
splatting, MeshLRM [44] with surface mesh, TripoSR [41] with NeRF triplane, and TetSphere [11] with tetrahedral spheres. For the baseline results, we used the publicly available inference code to reconstruct the 3D objects. 2 To demonstrate the versatility of our method, we integrated our physical compatibility optimization framework with all five baseline models and reported the results to ensure a fair comparison. The implementation details of our framework are provided Appendix D.
Evaluation Datasets. The evaluation dataset was sourced from Objaverse [9]. We initially randomly selected approximately 200 shapes from the categories of plants, animals, and characters categories that demand greater physical compatibility. Single-view images were rendered using the publicly released code by the authors of Objaverse". Subsequently, these images were used to reconstruct 3D objects using the baseline methods mentioned earlier. We filtered out shapes of extremely poor quality, specifically those with more than connected components. This process resulted in a final set of 100 shapes for detailed evaluation.
Despite these shapes being a part of the training data for most baseline methods, our evaluation focuses on assessing the physical compatibility a factor overlooked by these methods. The results obtained from this dataset provide valuable insights and observations on the physical compatibility of each method, demonstrating the practical effectiveness of our approach.
FFor MeshLRM, since the pre-trained model not publicly available yet, we obtained the reconstructed shapes directly from the authors for use in our study.
'https:// // /g thub. com/a hid//lleel allaaaannn//// averse renderiing
7

---

optimization
optimization
wio optimization
w/e optimization
static
optimization
optimization
Wonder
McbLRM
TripoSR
LGM
TecSphere
Figure 4: Qualitative results on physical compatibility optimization. Left: Rest shapes optimized using our approach result in static shapes that closely match the input images when subjected to gravity. In contrast, shapes without the optimization fail to replicate the geometry in the input image. Right: our optimization process ensures that the optimized shapes are capable of supporting themselves, whereas the baseline methods fail to achieve this stability.
4.2 Quantitative Results
Table shows the quantitative results for four out of five metrics evaluated for both baselines and those integrated with our physical compatibility optimization. Fig. shows the curve of fracture rate.
Our quantitative analysis yields several observations: 1) The underlying geometry representation significantly impacts the structural integrity of reconstructed shapes, evidenced by the number of connected components (#CC.). LGM, using a point cloud representation, exhibits the poorest structural integrity, often resulting floating structures due its inability differentiate the interior from the exterior of a 3D object. In contrast, TetSphere, with its volumetric representation, maintains the most integral structure. 3D objects, as indicated by Mean Stress and Standability (Standable.) metrics. However, they tend to diverge under gravity, as shown by the Matching Loss metric (Img. Loss), compared to TetSphere. 3) Notably, our method consistently enhances the physical compatibility performance across all baselines. The improvement is particularly significant for Wonder31) and MeshI.M.. Wonder3I) typically generates multi-view images before reconstructing the 31) shape, which can lead to thin structures due inconsistencies across the views. Similarly, MeshL.MM's reliance on surface mesh could often result in thin structures. Our method strengthens the physical robustness for both cases. 4) Our method also enhances the structure robustness fracture, as demonstrated in Fig. 3. notably improves the performance of both MeshLRM and TripoSR in reducing fracture rates.
Both MeshLRM and TripoSR generally produce more physically stable
4.3 Qualitative Results
Fig. and more qualitative results in Appendix illustrate the effectiveness of our physical com- patibility optimization. Without optimization, the static shapes behave undesirably under general physical principles: they either sag excessively under gravity, diverging from the geometry depicted in the input image, or fail to remain upright, toppling over. Our optimization method incorporates physical principles ensure that the optimized rest shapes are self-supporting and stable, and match the input images under static equilibrium.

---

Young's
200k
Modulus
line
Optimized Shapes
Inpur Imagt
Shapes
240g
Shapes under EXternal Forces
Figure 5: Ablation study on Young's modulus. By changing the material properties, our method can produce various rest-shape geometries (top), which all result in the same static shapes that match the input image (middle). Although these static shapes appear identical under static equilibrium, they exhibit different deformation when subjected the same compression forces exerted by the yellow block, attributable to the differences in their material properties (bottom).
4.4 Analysis
Ablation study on Young's Modulus. We investigate the influence of predefined mechanical material properties, partieularly Young's modulus, the optimized rest shapes and their physical behaviors. Using the same input image, we obtained six optimized rest shapes with varying Young's modulus values within our framework with TetSphere. As shown in Fig. 5, although the optimized rest-shape geometries vary, they all deform to the same static geometry under the influence of gravity, matching the input image. Moreover, the physical responses to identical external forces, such as compression by a box, differ due to the variations in material properties. These results highlight how the explicit decomposition of physical attributes our framework expands the controllability of object modeling, allowing for diverse physical behaviors under uniform external forces.
Application to dynamic simulation. The immediate output of our method is a simulation-reat rest-shape geometry, which be seamlessly integrated into simulation pipeline to produce complex dynamics and motions. Fig. (left) and the accompanying video in the Supplementary Material illustrate three plants modeled using framework, demonstrating their behavior under gravity and complex interactions. Implementation details of this simulation are provided in Appendix F. These examples underscore the practical utility of our method for generating physically realistic dynamics and simulations.
Application to fabrication. We further evaluate our method real world by fabricating three shapes using 3D printing, both with and without optimization. The results, shown in Fig. 6 (right), with detailed implementation procedures available in Appendix E,, demonstrate that the 31) printed shapes align with computational results. These real-world experiments demonstrate the practical effectiveness and validate the physical realism of the objects produced by our method.
5 Conclusion
In this work, we introduced physical compatibility optimization for reconstructing physical object from a single image. Our method decomposes three orthogonal attributes governing physical behavior: mechanical properties, external forces, and rest-shape geometry. Unlike existing methods that often ignore one or more dimensions, our framework holistically considers three factors, allowing for diverse rest-shape geometries from the same input image by varying object stiffness and external

---

Inpet
Imuges
optimization uptimization
(d)
(c)
Static shapes under gravity
Dynamic simulation
Wele opfinization fion optimization
optimication optimication
Figure 6: Applications of physically compatible objects. Left: Our optimized physical objects simulation-ready and be seamlessly integrated into dynamic simulation pipeline produce complex dynamics and motions. Right: Real-world validation using 3D printing shows that shapes op- timized using method closely replicate the input images, demonstrating the practical effectiveness of our method in manufacturing.
forces. We formulate physical compatibility optimization constrained optimization problem by integrating static equilibrium as hard constraint. Our approach produces physical objects that match the geometry depicted in the input image under external forces and remain stable under gravity. Both quantitative and qualitative evaluations demonstrated improvements in physical compatibility over existing baselines. Our method's 's versatility is evident through its integration with various single-vien view reconstruction methods and its practical applications in dynamic simulations and 3D printing.
Limitations and Future Work One limitation of framework is its reliance predefined material properties and external forces as inputs. Although this provides controllability of the final optimized rest-shape geometry, automating the extraction of these parameters from single image presents potential avenue for future work. Moreover, our method relies on the use of a tetrahedral mesh, which is derived by tetrahedralizing the output geometry produced by baseline methods. natural extension of our work is the development of a differentiable converter that can transform any geometric representation into tetrahedral mesh. This would enable future research where our physical compatibility optimization could be integrated into pre-trained large reconstruction model, which could then fine-tuned directly produce physically compatible 3D objects. Lastly, our current methodology focuses solely on physical objects in state of static equílibrium. Exploring the reconstruction of 3D objects undergoing dynamics captured from video is an intriguing prospect for future research.
References
[1] E. C. Aifantis. The physics of plastic deformation. International journal of plasticity, 3(3): 211-247, 1987.
[2] G. Allaire, Numerical analysis and optimization: an introduction to mathematical modelling and numerical simulation. OUP Oxford, 2007.
[3] S. Bell, K. Bala, and N. Snavely. Intrinsic images in the wild. ACM Transactions on Graphics (TOG), 33(4):1-12, 2014.
[4] A. H. Bermano, T. Funkhouser, and S. Rusinkiewicz. State of the art in methods and representa- tions for fabrication-aware design. In Computer Graphics Forum, volume 36, pages 509-535. Wiley Online Library, 2017.
[5] M. Botsch and O. Sorkine. On linear variational surface deformation methods. IEEE transac- tions on visualization and computer graphics, 14(1):213-230 2007.
[6] Burezyáski, Kane, and C. Balakrishna. Comparison of shape design sensitivity analysis formulations via material derivative-adjoint variable and implicit differentiation techniques for
10

---

3-d and 2-d curved boundary element. Computer methods in applied mechanics and engineering. 142(1-2):89-199 1997.
[7] D. Charatan, Li, A. Tagliasacchi, and V. Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXivv2312. 2023.
[8] X. Chen, C. Zheng, W. Xu, and K. Zhou. An asymptotic numerical method for inverse elastic shape design. ACM Transactions on Graphics (TOG), 33(4):1-11, 2014.
[9] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142- 13153, 2023.
[10] Y. Feng, Y. Shang, X. Li, T. Shao, C. Jiang, and Y. Yang. Pie-nerf: Physics- based interactive elastodynamics with nerf. arXiv preprint arXiv2311.13309, 2023.
[11] M. Guo, B. Wang. K. He, and W. Matusik. Tetsphere splatting: Representing high-quality geometry with lagrangian volumetric meshes.
[12] Z. He and Wang. OpenIrm: Open-source large reconstruction models. https://githul: com/3DTopia//n / 2023.
[13] Y. Hong, K. Zhang, J. Gu, S. Bi, Y. Zhou, D. Liu, F. Liu, K. Sunkavalli, T. Bui, and H. Tan. LRM: Large reconstruction model for single image to 3D. Nov. 2023.
[14] J. Hsu, N. Truong, C. Yuksel, and K. Wu. A general two-stage initialization for sag-free deformable simulations. ACM Transactions on Graphics (TOG), (14:::1-1, 2022.
[15] Y. Hu, Q. Zhou, X. Gao, A. Jacobson, D. Zorin, and D. Panozzo. Tetrahedral meshing in the wild. ACM Trans. Graph., 37(4);60: 1-60::1::::::4 14, July 2018. ISSN 0730-0301. doi: 10.1145/ 3197517.3201353, URL http::/dooo acm. org/10..//. 10. 10111114/1119 31955/719717.../ 32013553.
[16] B. Kerbl, G. Kopanas, T. Leimküüller and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
[17] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015.
[18] M. Li, Z. Ferguson, T. Schneider, T. R. Langlois, D. Zorin, D. Panozzo, C. Jiang, and D. M. Kaufman. Incremental potential contact: intersection-ans inversion-free. large-deformation dynamics, ACM Trans. Graph., 39(4):4, 2020.
[19] X. Li, Y.-L. Qiao, P. Y. Chen, K. M. Jatavallabhula, M. Lin, C. Jiang, and C. Gan. Pac-nerf: Physics augmented continuum neural radiance fields for geometry-agnostee system identification. 2022.
[20] M. Liu, C. Xu, H. Jin, L. Chen, V. T. Mukund, Z. Xu, and H. Su. One-2-3-45: Any single image to 3D mesh in 45 seconds without Per-Shape optimization. June 2023.
[21] R. Liu, R. Wu, B. V. Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick. Zero-1-to-3: sero-shot one image to 3d object, 2023.
[22] Y. Liu, C. Lin, Z. Zeng, X. Long, L. Liu, T. Komura, and W. Wang. SyncDreamer: Generating multiview-consistere images from single-view image. Sept. 2023.
[23] X. Long, Y.C. Guo, C. Lin, Y. Liu, Dou, L. Liu, Y. Ma, S.-H. Zhang, M. Habermann,
C. Theobalt, and W. Wang. Wonder3D: Single image to 3D using Cross-Domain diffusion. Oct. 2023.
[24] I. Kyriazi, C. Rupprecht, I. Laina, and A. Vedaldi. RealFusion: 360 reconstruction of any object from single image. Feb. 2023.
[25] M. Mezghanni, T. Bodrito, Boulkenafed, and M. Ovsjanikov. Physical simulation layer for accurate 3d modeling. In CVPR, 2021.

---

[26] M. Mezghanni, M. Boulkenafed, A. Lieutier, and M. Ovsjanikov. Physically-awe generative network for 3d shape modeling. In CVPR, 2021.
[27] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. ECCV, 2020.
[28) R. v. Mises. Mechanik der festen körper im plastisch- zustand. Nachrichten von der Gesellschaft der Wissenschaften zu Göttingen, Mathhematisch--Pyysikalii Klasse, 1913: 582-592, 1913.
[29] B. Nicolet, A. Jacobson, and W. Jakob. Large steps in inverse rendering of geometry. ACM Transactions on Graphics (TOG), 40(6): 1-13, 2021.
[30] K. Perlin. Improving noise. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques, pages 681 1-121,622222 2002.
[31] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[32] G. Qian, J. Mai, A. Hamdi, J. Ren, A. Siarohin, B. Li, H.-Y. Lee, I. Skorokhodov, P. Wonka, S. Tulyakov, and B. Ghanem. Magic123: One image high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306. 17843, 2023.
[33] J. R. Shuc, E. R. Chan, R. Po, Z. Ankner, J. Wu, and G. Wetzstein. 3d neural field generation using triplane diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20875-20886. 2023.
[34] E. Sifakis and J. Barbic. Fem simulation of 3d deformable solids: practitioner's 's guide to theory. discretization and model reduction. In Acm siggraph 2012 courses, pages 1-50. 2012.
[35] B. Smith, D. Goes, and T. Kim. Stable neo-hookean flesh simulation. ACM Transactions on Graphics (TOG), 37(2):1-15, 2018.
[36] J. Smith and S. Schaefer. Bijective parameterization with free boundaries. ACM Transactions on Graphics (TOG), 34(4):1-9, 2015.
[37] Standley, O. Sener, D. Chen, and Savarese, image2mass: Estimating the mass of an object from its image. In Conference Robot Learning, pages 324-333, PMLR, 2017.
[38] S. Szymanowicz, C. Rupprecht, and A. Vedaldi. Splatter image: Ultra-fast single-view 3d reconstruction. 2024.
[39] J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng. DreamGaussian: Generative gaussian splatting for efficient 3D content creation. Sept. 2023.
[40] J. Tang, 7. Chen, X. Chen, T. Wang, G. Zeng, and 7.. Liu. LGM: Large Multi- View gaussian model for High-Resolutin 3D content creation. Feb. 2024.
[41] D. Tochilkin, D. Pankratz, Z. Liu, Z. Huang, A. Letts, Y. Li, D. Liang, C. Laforte, V. Jampani, and Y.-P. Cao. Triposr: Fast 3d object reconstruction from single image. arXiv preprint arXiv:2403.02 15,, 2024.
[42] B. Wang, G. Matcuk, and J. Barbič. Modeling of personalized anatomy using plastic strains. ACM Transactions on Graphics (TOG), 40(2):1-21, 2021.
[43] Wang, Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. June 2021.
[44] X. Wei, K. Zhang, S. Bi, H. Tan. F. Luan, V. Deschaintre, K. Sunkavalli, H. Su, and Z. Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404. /2385, 2024.
[45] R. Wu, B. Mildenhall, P. Henzler, K. Park, R. Gao, D. Watson, P.P. Srinivasan, D. Verbin, J. T. Barron, B. Poole, et al. Reconfusion: 3d reconstruction with diffusion priors. arXiv preprint 23X102331,.02981, 2023.
12

---

[46] T. Xie, Z. Zong, Y. Qiu, X. Li, Y. Feng, Y. Yang, and C. Jiang. Physgaussian: 3d gaussians for generative dynamics. arXiv preprint arXiv:2311.12196 2023.
[47] Y. Xu, Z. Shi, W. Yifan, H. Chen, C. Yang, S. Peng, Y. Shen, and G. Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint 0rXiv:2403.14621, 2024.
[48] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixeINeRF: Neural radiance fields from one or few images. 2021.
[49] A. J. Zhai, Y. Shen, E. Y. Chen, G. X. Wang. X. Wang. Wang, K. Guan, and S. Wang. Physical property understanding from language-embedded feature fields. arXiv preprint 044x:2440.04242., 2024.
[50] K. Zhang, Bi, H. Tan, Y. Xiangli, N. Zhao, K. Sunkavalli, and 7. Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:2404. 19702, 2024.
[51] L. Zhong, H.-X. Yu, J. Wu, and Y. Li. Reconstruction and simulation of clastic objects with ppring-mass 3d gaussians. arXiv preprint arXiv:4403009434 2024.
13

---

wio optimization
wi optimization
wio optimization
wio optimization
static
optimization
optimization
Wondery
McshLRM
TripoSR
LGM
TetSphere
Figure 7: Additional qualitative results of physical compatibility optimization (part 1/2).
wio optimization
w/ optimization
wio optimization
w/ optimization
rest
static
rot
static
rest
static
rest
static
Wonder3D
Mesh
TripoSR
TeiSphere
Figure 8: Additional qualitative results of physical compatibility optimization (part 2/2).
A Additional Qualitative Results
Figure and 8 show additional results of our physical compatibility optimization.
B Plastic Strain Field Fp
To enhance the understanding our framework without compromising generalizability, us consider Mintt to be a tetrahedral mesh composed of single element and four vertices. When
14

---

subject to static equilibrium influenced by gravity, the object adheres to the equation:
fi O iiit); +) Mg.
(6)
where fint (: denotes the elastic force (internal force), M the mass matrix, and g denotes the gravity acceleration. To compute this force, we first consider the elastic energy &. The definition of elastic energy unfolds as follows:
E(F Ppi θ) = (F:;))
V(Fp) p) det
F. = FFF F = 0x/0xiint;
where ((F,) represents the volume of the element under plastic strain, Vinit the initial volume of the element, F. denotes the elastic deformation gradient, represent the total deformation gradient, and ((=: θ) the elastic energy density function. This deformation gradient computed through standard methodology [34].
Consequently, the derivation of the elastic force encapsulates the computation of the first-order partial derivative of the elastic energy with respect to the vertex positions:
fint (x, o(F Xinit;; ); ()) dE (F. Fp:
0x
0Φ
OF
((F
F
OF
Ox
Notably, given the linear dependence of on x, 8x remains constant.
Given Fp and Xinit as inputs, the solution to Eq. 6 is the static shape, = Xstatie- Likewise, to calculate Xrest from Fp and Xinit in Eq. 3, we solve similar equation with zero external force. fine(,, ((Fp ) θ) = 0,
where the solution to this equation is r
Considering the elastic energy, the translation of Xinit does alter the deformation gradient F. Consequently, Fp remain unaffected and exhibit translation invariance. In terms of the elastic force, it maintains translation invariance as well, since F is not affected by any shift in X Xinit-
Finally, by using isotropic materials, our approach enables a further reduction in the DOFs of F Let us denote Fp as Fp R.S. The elastic deformation gradient then derived F. = F(RS)) FS FS- Given the invariance property (FF.; 0) o R: 0), which constantly holds for isotropic materials, the rotation component R becomes redundant and can be excluded from the formulation. This simplification implies that the only requirement for Fp is to be symmetric matrix. During the optimization process, this property facilitates the prevention of the inversion: In order to ensure that det(F O, we can simply adjust the eigenvalues of F, p to make they remain positive. This adjustment is crucial for the rest mesh X to maintain in the non-ine non-inverted state.
C Computation of Gradient
By differentiating the constraint in Eq. 4 with respect to F Fp. we obtain
Of Öfnt OF, Oxstatic OF
Ox
0.
Then, we have
Oxstatic
Ofpet
Ofae
-1 Ix
(8)
OF,
OF p
Substituting the result into the objective in Eq. 4, we get
0J
0L dCrez
OF
OF p OF,
p
Oxstatic Oxstatic OF, OF dC
0Lreg
0L Ofaet | Ofct +
GLrg
(9)
-xxttatic Ix
OFp oF
15

---

Hass and 0fast are stored as sparse matrices which is the gradient with respect to Fp. In practice, 8xstatic OF and computed based on [42]. Considering about the performance, we first compute dx -xxttaii using sparse linear solver. This results in dense vector with size 3N. We then multiply with ofant DFP
ac dfac ]-1
D Implementation Details of Evaluation
To evaluate the physical compatibility of baseline methods, which often produce shapes comprising multiple connected components, we first extract the largest connected component from each mesh. All meshes are then normalized to the unit cube. Notably, the reconstructed shapes from TripoSR and Wonder3D are not axis-aligned; thus, we manually rotate these shapes to ensure the head points towards the z-axis in the world coordinate space. For integrating our physical compatibility framework, We use two sets of Young's modulus, = x 10 and E = 5 x 10 which are selected based on whether the shape would become overly soft, potentially leading to static cquilibrium failure due to excessive stress causing numerical bounds to be exceeded. Poisson's 's ratio = 0.45 and mass density P = 1000kg/ni //n are consistent across all meshes. Evaluation metrics require solving for static equilibrium Eq. We employ the Newton-Raphe setting the maximum number of iterations to be 200. For optimizing Eq. 4, and allow up to 1000 iterations. Our experiments run on a desktop PC with an AMD Ryzen 9 5950X 16-core CPU and 64GB RAM. The average runtime for this optimization process is approximately 80 seconds.
solver with line search, gradient descent
E
Implementation Details of 3D Printing
The selected model shapes were 3D printed using stereolithography (Form3; Formlabs, 100 umm layer thickness) to create the flexible designs (using Flexible 80A, tensile modulus <3 MPa, 100% strain to failure) and rigid designs (using White Resin V4; tensile modulus 1.6 GPa), both without post-curing The flexible flowers are 55 and 65 mm in height and the rigid goose is 50 mm in length. Shapes with and without optimization were printed with similar support structures designed preserve delicate features.
F Dynamic Simulation of Deformable Objects
We model each solid deformable object using FEM with hyperelastic materials for dynamic simulation. Then, we solve the standard partial differential equation (PDE) for dynamic FEM simulation:
Mä + D(r) + felastic
+ (acr)+Jatttenn Sattachment (I) + Jooutace (x) Mg.
(10)
where r represents the node positions within the finite element meshes - we use tetrahedral meshes of the objects, M denotes the mass matrix, D the Rayleigh damping matrix, felastie is the hyperelastic forces, fattachrueut (x) is the attachment forces that constrain the objects to a specific location, and feontact() denotes the contact forces between surfaces. We employ the implicit backward Euler method for time discretization, transforming the PDE into:
n + 6"h (x**s) + fattachmentt (x*+1) + (xontac((**+1) 0,
(11)
where -1+1 is the position vector at timestep (n + 1), AA" and 6" is constant matrix and vector, respectively, derived from values at timestep n, Finally, we solve this nonlinear equation using Newton's 's method at each timestep.
The hyperclastic material selected for the deformable objects is the same as the one used for the rest shape optimization [35] in Sec. 3. Attachment forces are modeled as spring forces fattachment (z) = (Sx - H(t)), where k2 is the stiffness of the spring, the selection matrix S selects the attached vertices, and att) denotes the target attachment locations at time t. Contact forces are generated from penalizing any vertex penetration into the contact surface, expressed as f = kgd, where k represents the contact stiffness and d denotes the penetration depth, with d 0 in the absence of contact. This gives the normal contact forces. Friction forces are computed following the methods outlined in [18]. Then, the total contact force Scontact is the sum of normal contact forces and friction forces.
For the dynamic simulation in Figure 7, the attachment of each plant defined as the bottom part of each pot. We keyframe-animate animate the trajectory of attachment vertices Z(t). Gravity is enabled
16

---

throughout the entire simulation. At the end of the sequence, we apply wind forces to the plants, computed using 4D Perlin Noise [30].
G Broader Impacts
Our research presents computational framework for reconstructing physical objects from single images. This advancement holds significant potential for various applications, including dynamic simulations, 3D printing, virtual reality, and industrial design. By ensuring that the reconstructed objects adhere to real-world physical laws, our method can enhance the realism and functionality of virtual environments, improve the precision of 3D printed objects, and contribute the development of more reliable industrial designs.
There are mainly two potential negative societal impacts: Improved 3D reconstruction capabilities could potentially be misused to create highly realistic fake objects or environments for disinformation purposes. This could include generating deceptive media content that appears authentic. As the framework automates the reconstruction process, there is potential risk of it being used in automated systems without sufficient oversight, potentially leading to unintended and harmful outcomes due to errors or misuse. Developing systems to monitor the use of the technology and ensure accountability for its applications, well as providing comprchensive guidelines and training for users to promote ethical use and awareness of potential misuse, will address these potential negative impacts.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See Section 3 and 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
See Section 3, B, and C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section and D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: Project page is available at htttp:///mhh1: https::///g114..tthbbbb.. wttchub.io/hhy-coome comp/.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
assssnsoliccyyoss for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececccccccccct / ssisnnoillllo onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 4 and D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Table and Figure 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section D.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /ttp///eurrp..e//ee//e cc cctppblcc//aeccccccccccccccc ctw.eettiiiiiieeeeiiieeeeeeeeee
Answer: [Yes]
Justification: The authors fully acknowledged the code of ethics and commit to adhering to its guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Section G.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Section 4.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "ysYs3XQAyb", "reviewer": "Reviewer_cqis", "summary": "The paper presents a 3D mesh optimization framework to ensure physical plausible 3D object reconstructions from a single image. These 3D reconstructions should conform with global (e.g. gravity) and user-defined constraints (material stiffness) as well as match with a target image of the reconstructed, simulated object. The reconstructed mesh is considered as a solid object.\n\nThe method is combined with 5 existing single-image 3D object reconstruction methods, which are evaluated on a collection of 100 Objaverse samples. Several additional metrics are introduced to test for the physical 'compatibility' of the reconstructions: mean stress, connected components, standability and the difference to the reference view.", "questions": "- Figure 3: The plot for TetSphere show minimal improvements using the proposed optimization. Could the authors elaborate on this?", "limitations": "The checklist is complete and the paper discusses its limitations.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The motivation for physically plausible reconstruction is clearly stated. Ensuring physical constraints of reconstructed 3D meshes allows for further downstream applications, such as 3D printing, hence it is an important aspect to consider.\n\nThe writing and quality of the figures (including the video) is of very high quality. The set of metrics is well chosen and the paper includes very detailed descriptions of the method. Its effectiveness is demonstrated using real-world 3D printed results with and without the proposed optimization.", "weaknesses": "- While I can understand the space constraints of the submission format, the main paper should still include a complete and comprehensive overview of related work instead of deferring it to the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive questions.\n\n**Rebuttal Outline:**\n\n1. **[Adjustment] (W1)** Regarding the placement of the Related Work section.\n2. **[Question] (Q1)** Elaboration on results of TetSphere.\n\nPlease find below our detailed responses to your comments and concerns.\n\n---\n\n**1. (W1) The main paper should still include a complete and comprehensive overview of related work instead of deferring it to the appendix.**\n\nWe will adjust the manuscript to include the Related Work section within the main text. We plan to move Figure 5 to the Appendix and compress Figure 4 to free up sufficient space to accommodate the Related Work section within the page limits.\n\n---\n\n**2. (Q1) Figure 3: The plot for TetSphere shows minimal improvements using the proposed optimization. Could the authors elaborate on this?**\n\nThe modest improvement observed for TetSphere is attributed to its use of an explicit volumetric representation. As discussed in TetSphere's original paper, this representation inherently enhances robustness to slender shapes and thin structures. Unlike the other four methods we evaluated, which do not utilize explicit volumetric representations and thus are more vulnerable to fractures in thin structures, TetSphere maintains structural integrity. Nevertheless, our optimization method still enhances the robustness of TetSphere to fractures, as demonstrated in Figure 3.\n\n---\n\nPlease let us know if you have any further questions. We appreciate your time and effort. Thank you!\n\nBest regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RasXwPlfVF", "reviewer": "Reviewer_H9KU", "summary": "This paper introduces a physical compatibility optimization framework for reconstructed objects from a single image. The approach considers mechanical properties, external forces, and rest-shape geometry, integrating static equilibrium as a hard constraint. This framework improves upon existing methods by ensuring the stability and accuracy of reconstructed objects under external influences. Quantitative and qualitative evaluations show enhancements in physical compatibility.", "questions": "1. All $\\mathbf{x}\\_{static}$ may be $\\mathbf{X}_{static}$ as the defination in line 89.\n2. In Some cases discussed in the paper, like the flamingo standing on one leg, should it be standable even after optimization? I think it should not be standable under gravity.\n3. Why do you evaluate the different off-the-shelf methods using connected components when this cannot demonstrate the superiority of your proposed method?", "limitations": "The authors have discussed the limitations and potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Performance: The proposed method achieves state-of-the-art results. The experiments well validate the effectiveness of the proposed methods.\n\n2. Clarity: The paper is well-written and easy to follow.\n\n3. Technical Novelty: The main contributions of this paper are twofold: 1) They propose a physical compatibility optimization framework for 3D objects and decompose the mechanical properties, external forces, and rest-shape geometry. 2) They optimize the rest-shape geometry using predefined mechanical properties and external forces and ensure the object’s shape aligns with the target image when in static equilibrium.", "weaknesses": "In this paper, the authors primarily apply the physical compatibility optimization framework to enhance the physical attributes of 3D models obtained from existing methods rather than reconstructing them from a single image. Therefore, I think the title \"Physically Compatible 3D Object Modeling from a Single Image\" may not be suitable, as the focus lies on enhancing physically compatible modeling of 3D objects derived from off-the-shelf methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's effort in reading and evaluating our work carefully!\n\n**Rebuttal Outline:**\n\n1. **[Discussion] (W1)** Title.\n2. **[Adjustment] (Q1)** $\\textbf{x}_{static}$.\n3. **[Clarification] (Q2)** 3D Printing Results of Flamingo.\n4. **[Clarification] (Q3)** Evaluation on the number of Connected Components.\n\nPlease find below our detailed responses to your comments and concerns. We hope these responses assist you in finalizing your assessment of our manuscript's rating.\n\n---\n\n**1. (W1) The title may not be suitable, as the focus lies on enhancing physically compatible modeling of 3D objects derived from off-the-shelf methods.**\n\nWe appreciate your observation regarding the scope of our work and the paper’s title.\n\nThe title \"Physically Compatible 3D Object Modeling from a Single Image\" was chosen to highlight our method's unique process: starting with a single image and constructing 3D object under physical compatibility constraints. Although our approach utilizes existing 3D reconstruction techniques, the core of our contribution lies in the introduction and application of a novel optimization framework that ensures physical compatibility. We believe that retaining the current title accurately reflects the novelty of our contribution.\n\nHowever, we are open to considering alternative titles such as \"Enhancing Physical Compatibility of 3D Object Modeling from Single Images\" if it would provide clearer communication of our work's scope and impact.\n\n---\n\n**2. (Q1) All $x_{static}$ may be $X_{static}$ as the definition in line 89.**\n\nThank you for your attentive reading! In our manuscript, we adhere to the conventional notation system used in physical simulation and finite element analysis [1], where the rest shape is denoted by capital letters $\\textbf{X}$ and the static shape by lowercase letters $\\textbf{x}$. To resolve the confusion, we will adjust the notation in line 89 to use $\\textbf{x}$ consistently. Additionally, we will include a clarifying statement in the text to explicitly define that $\\textbf{X}$ represents the rest shape.\n\n---\n\n**3. (Q2) For the flamingo standing on one leg, should it be standable even after optimization?**\n\nOur optimization indeed successfully enables the flamingo model to stand on one leg. To address your concern, we've included a supplementary PDF in this rebuttal, showing the 3D-printed results of the standing flamingo after optimization from various camera angles. We hope this additional evidence clarifies the effectiveness of our method.\n\n---\n\n**4. (Q3) Why do you evaluate the different off-the-shelf methods using connected components?**\n\nThank you for the question. While the metric of connected components may not directly demonstrate the superiority of our proposed method, it remains a critical dimension for evaluating physical compatibility. This metric effectively captures the integrity of a 3D object, which is an essential attribute for assessing the physical realism of 3D models, since disconnected parts fall apart and thus naturally invalidate the definition of \"physical compatibility\". Including it provides valuable insights into the baseline performance of existing off-the-shelf methods with respect to physical compatibility.\n\n---\n**References**\n\n[1] Eftychios Sifakis, and Jernej Barbic. \"FEM simulation of 3D deformable solids: A practitioner's guide to theory, discretization, and model reduction.\" ACM SIGGRAPH 2012 courses. 2012. 1-50.\n\n---\nWe appreciate your time! Thank you so much!\n\nBest regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SvubZ18cef", "reviewer": "Reviewer_91LB", "summary": "This paper introduces the concept of physical compatibility into single-image to 3D generation. It proposes a post-optimization method for existing 3D generation pipelines, which takes the generated mesh as the target mesh and optimizes the deformation gradient to obtain a rest-shape geometry without external forces. This rest-shape geometry can align with the target mesh under physical conditions provided by the user. The proposed method significantly improves the results for different single-image to 3D pipelines and is demonstrated to be effective through a comparison of 3D printed results under real-world scenarios.", "questions": "The proposed method optimizes the positions of mesh vertices while maintaining the connectivity of the vertices. However, existing 3D generation models may introduce incorrect connectivity for complex or partially occluded input image. I wonder if physical compatibility approach can correct such errors in connectivity during the optimization process.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The idea of introducing physical compatibility into 3D generation is very innovative, and it is crucial for the application of current 3D generation results to real-world scenarios. The authors provide detailed derivations and explanations, and propose five reasonable evaluation metrics to assess the degree of physical compatibility. They demonstrate the effectiveness of their method in real-world scenarios through 3D printed objects.", "weaknesses": "This is not strickly a weakness of the proposed method. The paper mainly focuses on the optimization of a single object. However, current 3D generation pipelines can handle the generation of multiple coupled objects, such as a castle. I wonder whether this method can be applied to such senarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s insightful comments.\n\n**Rebuttal Outline:**\n\n1. **[Discussion] (W1)** Handling the generation of multiple coupled objects.\n2. **[Discussion] (Q1)** Correcting errors in connectivity during the optimization process.\n\nPlease find below our detailed responses to your comments and concerns.\n\n---\n\n**1. (W1) This is not strictly a weakness of the proposed method. The paper mainly focuses on the optimization of a single object. However, current 3D generation pipelines can handle the generation of multiple coupled objects, such as a castle. I wonder whether this method can be applied to such scenarios.**\n\nWe appreciate your observation highlighting a promising direction for future research. Extending our method to handle multiple objects introduces the challenge of managing interactions and collisions between objects. As a potential pathway forward, we could integrate differentiable simulation techniques and collision constraints [1] into existing 3D generative models. This adaptation would allow our framework to address the dynamic interactions and physical compatibility of multiple coupled objects.\n\n\n---\n\n**2. (Q1) Existing 3D generation models may introduce incorrect connectivity for complex or partially occluded input images. I wonder if the physical compatibility approach can correct such errors in connectivity during the optimization process.**\n\nCurrently, our method maintains fixed connectivity during the physical compatibility optimization. However, a promising extension to address incorrect initial connectivity could involve incorporating adaptive remeshing or subdivision techniques during the optimization process. Specifically, while adhering to our established physical compatibility framework, we could intersperse the optimization with periodic remeshing steps. This approach would adaptively correct connectivity errors throughout the optimization process, enhancing the accuracy and robustness of the reconstructed shapes.\n\n---\n**References**\n\n[1] Huang, Zizhou, et al. \"Differentiable solver for time-dependent deformation problems with contact.\" ACM Transactions on Graphics 43.3 (2024): 1-30.\n\n---\nWe sincerely appreciate the comments. Please let us know if you have further questions.\n\nBest regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jy7l8Mrta7", "reviewer": "Reviewer_XLVo", "summary": "This paper presents an optimization method that produces 3D shapes from a single images that considers about mechanical properties and external forces. The shapes of the generated results in a state of static equilibrium are able to match the input image. The proposed methods can be used with other single-view image shape generation methods.", "questions": "- I am curious about why the authors decided to optimize the deformation instead of optimizing the hollow of the solid geometry, similar to the method used in Make-It-Stand [Prevost et al. 2013] and the follow-up papers.\n- It is unclear to me what is the limitation of the proposed method. \n- It is unclear to me what is the performance overhead for the proposed method. I would encourage the authors to report the shape complexity and the corresponding computational time.\n- It is unclear to me whether image loss is used during the optimization? In fig. 2, the pipeline invovles image loss but I did not see the discussion in Sec. 2 but only in evaluation part.", "limitations": "As mentioned above, I think the limitations should be discussed more in the main paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The strengths of the proposed method are:\n- The proposed optimization framework is novel and versatile.\n- The method supports to produce fabricatable shapes from a single image.", "weaknesses": "The weaknesses of the proposed method are:\n- The limitation of the proposed method is not discussed in the main paper.\n- The discussion with alternative approaches is not clear in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable comments from the reviewer.\n\n**Rebuttal Outline:**\n\n1. **[Adjustment] (W1&W2&Q2)** Move Limitations and Discussions with Alternative Approaches to Main Text.\n2. **[Discussion] (Q1)** Alternatively Optimizing the Hollow of the Solid Geometry.\n3. **[Clarification] (Q3)** Performance.\n4. **[Clarification] (Q4)** Image Loss in Optimization.\n\nPlease find below our detailed responses to your comments and concerns.\n\n---\n\n**1. (W1&W2&Q2) The limitation of the proposed method and the discussion with alternative approaches is not clear in the main paper.**\n\nThank you for pointing out the placement of our discussions on the method's limitations and comparisons with alternative approaches. These discussions are indeed present in Appendices G and H. We acknowledge the importance of making this information more accessible and are considering moving these sections into the main body of the text. We plan to compress Figure 4 and move Figure 5 to the appendix to free up enough space to accommodate these critical discussions within the page limit.\n\n---\n\n**2. (Q1) I am curious about why the authors decided to optimize the deformation instead of optimizing the hollow of the solid geometry, similar to the method used in Make-It-Stand [Prevost et al. 2013] and the follow-up papers.**\n\nWhile optimizing the hollow of solid geometry is suitable for enhancing standability, it is not well-suited to address other losses such as ensuring the static shape accurately replicates the geometry depicted in the input image. Furthermore, Make-It-Stand and its follow-ups involve an extra carving stage that hollows out the interior of the 3D shape, along with a shape deformation stage. This carving process often depends on manually defined heuristics to prevent the creation of isolated parts and is computationally intensive due to its combinatorial nature. In contrast, our method using the plastic deformation on the whole shape eliminates the need for this additional and complex stage.\n\n---\n\n**3. (Q3) It is unclear to me what is the performance overhead for the proposed method. I would encourage the authors to report the shape complexity and the corresponding computational time.**\n\nThe average shape complexity in our experiments, across 100 evaluated shapes, is approximately 5,200 vertices and 14,000 tetrahedra per shape. Regarding computational performance, the average computational time required for our method on a standard PC configuration (AMD Ryzen 9 5950X 16-core CPU and 64GB RAM) is about 2.5 minutes per shape. We will include these results in the final manuscript.\n\n---\n\n**4. (Q4) It is unclear to me whether image loss is used during the optimization? In Fig. 2, the pipeline involves image loss, but I did not see the discussion in Sec. 2, only in the evaluation part.**\n\nThank you for raising this point. We did not utilize image loss directly during the optimization process. Instead, we assume that the initially reconstructed shape from the input image accurately represents the target geometry, as outlined in lines 161-162. The evaluation of image loss, however, is used to assess whether the final optimized shape, when subjected to external physical forces such as gravity, still corresponds closely with the original input image.\n\n--- \nPlease let us know if you have any further questions. We really appreciate your time. Thank you!\n\nBest regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We sincerely appreciate the detailed reviews and the thoughtful feedback provided by all reviewers and the Area Chair. In addition to addressing specific comments from each reviewer, we would like to outline our primary contributions.

- **[Motivation]** We tackle a critical issue in 3D generative modeling -- enhancing the **physical compatibility** of generated objects, a vital aspect often overlooked in contemporary research [cqis, 91LB].
- **[Methodology]** Our approach is **novel** and significantly improves the **versatility and applicability** of 3D reconstruction methods [H9KU, 91LB, XLVo], evident through the integration of physical constraints within the generative process.
- **[Experiments]** We conducted comprehensive experiments that not only demonstrate the **robustness and versatility** of our method but also extend its applicability to **real-world scenarios** like 3D printing [cqis, 91LB].
- **[Presentation]** The manuscript is recognized for **its clarity and well-written quality** [cqis, H9KU].

We hope our responses address all reviewers' concerns and help improve the review scores. We thank all reviewers and the AC again for their time and efforts!

Meta-review: All the reviewers are positive and recommending acceptance of the submission. The AC agrees with the decision. Please read the weakness comments carefully and prepare the camera ready.
Decision: Accept (spotlight)