Title: Generating Physically Sound Designs from Text and a Set of Physical Constraints
Venue: NeurIPS 2025
Accepted: yes
Keywords: generative design

structural optimization

text-image generation

differentiable physics simulation
Abstract: We present TIDES, a text informed design approach for generating physically sound designs based on a textual description and a set of physical constraints. TIDES jointly optimizes structural (topology) and visual properties. A pre-trained text-image model is used to measure the design's visual alignment with a text prompt and a differentiable physics simulator is used to measure its physical performance. We evaluate TIDES on a series of structural optimization problems operating under different load and support conditions, at different resolutions, and experimentally in the lab by performing the 3-point bending test on 2D beam designs that are extruded and 3D printed. We find that it can jointly optimize the two objectives and return designs that satisfy engineering design requirements (compliance and density) while utilizing features specified by text.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a pipeline for creating 2D designs that follows both the text description and physical constraints. It jointly optimizes a 2-D density grid with differentiable FEM simulation and a frozen CLIP similarity loss, so each design is both mechanically stiff and visually aligned with a text prompt. Experiments show it outperforms physics-only or vision-only baselines on the stiffness–CLIP-score trade-off and remains robust when 3D-printed and mechanically tested.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. First to insert a frozen CLIP semantic loss into differentiable topology optimization, enabling joint vision-physics gradients.\n2. Well-designed experiments with strong physics-only / vision-only baselines and 3D-printed real-world mechanical validation.\n3. Offers a text-driven yet mechanically sound design pipeline, potentially impactful for architecture and mechanical engineering.\n4. Pipeline diagram and loss equations are concise and easy to follow.\n\n# Weaknesses\n1. Limited to 2D structures; no convincing argument or demo for 3D extension.\n2. Many tunable hyperparameters, but no systematic sensitivity study.\n3. Alternative Design: Diffusion-based losses (e.g., SDS) could replace CLIP loss\n4. Runtime: The paper states only the hardware used but omits per-iteration or total wall-clock times, making efficiency hard to judge.\n\nQUESTIONS:\n1. Show a minimal 3D example—e.g., plugging your differentiable FEM module into a DreamFusion-style pipeline and optimizing a 3D structure?  This would directly demonstrate that the method generalises beyond 2D.\n2. SDS-Loss Comparison: Replace the CLIP loss with the SDS loss and see if it provides better results.\n3. Provide analysis on hyperparameter sensitivity. How hard is it for a user to find suitable parameters? This will reveal how robust the framework is and offer guidance to practitioners.\n4. Timing and Scaling: Report the wall-clock time per iteration and total training time for each resolution (64 → 512), and compare with other solutions on the same hardware.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a generative framework that can synthesize physically sound structural designs according to text instructions. Given a design instance, it leverages a physics simulator to compute deformation compliance and material cost, and uses CLIP to obtain the similarity between the visual appearance and the text guidance. Then, it jointly minimizes a compliance loss, a material cost, and an image-text alignment loss to achieve a functional design that can withstand applied forces. Experiments on several design problems, as well as a 3D-printing evaluation, demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThis paper presents an end-to-end framework for generating images of structures from text while ensuring physical accuracy. It leverages the strengths of a multi-modal language model, CLIP, to enhance the alignment between text and images. Additionally, it incorporates a differentiable physics simulator to optimize the structural design for physical soundness. The overall concept is straightforward and effective, as demonstrated by real-world 3D printing experiments.\n\n**Weaknesses**\n\nThere is ample room for improvement in the paper's structure and organization. Specifically, it lacks clear sections and many parts are mixed together, making it difficult for readers to understand. For example, in Section 2, background and related work are intermixed, making it difficult to find an introduction to similar existing works on the same task. Similar issues can also be found for Section 3, and it is suggested to use subsections to organize the paper better.\n\nAdditionally, there are concerns regarding the technical novelty and completeness of the evaluation of the proposed method; see the questions below for details.\n\nQUESTIONS:\n1.\tThe use of differentiable physics to enhance the physical soundness of generated assets has been explored in several existing methods [a-c]. What distinguishes the proposed method from these works?\n\n2.\tThe paper starts with a design encoding value of 1. Is it possible to initialize the proposed method using existing designs (e.g., from artists) that may not initially withstand the applied force, and then optimize these designs with minimal modifications to ensure they can support the force? This can improve the practical usage of the method.\n\n3.\tThe paper employs varying values for $\\beta_1$ and $\\beta_2$ across different design problems (see Section B.5), suggesting that the proposed method may be sensitive to the selection of hyperparameters. How are these hyperparameters chosen? Did the authors conduct experiments to determine a set of hyperparameters that perform well across all the considered problems?\n\n4.\tThe experiments only compare the proposed method with its own ablations (w/o visual loss or w/o physics loss). How does the method compare with state-of-the-art text-to-image models like Stable Diffusion[d]?\n\n5.\tWhat is the running time for generating a single design?\n\n\nReferences\n\n[a] Physically Compatible 3D Object Modeling from a Single Image. NeurIPS 2024.\n\n[b] Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabrication. NeurIPS 2024.\n\n[c] DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundness. arXiv 2025.\n\n[d] High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper presents TIDES, a differentiable pipeline that jointly optimizes a CLIP-based visual loss with structural compliance and material-cost losses so that 2D density fields evolved from a prompt end up both visually faithful and mechanically stiff. Overall, the study demonstrates that adding differentiable physics to a pre-trained generative model can produce manufacturable, prompt-conditioned structures.

STRENGTHS AND WEAKNESSES:
**Strengths**:
* The paper explicitly co-optimize a differentiable finite-element compliance loss (modified SIMP) and a CLIP text-image similarity loss in a single end-to-end loop, which unifies topology optimisation and text-to-image generation.
* This article provides a detailed explanation of the hyperparameters and visualization results of the experiment, and it has strong reproducibility. Meanwhile, conduct a thorough analysis of the potential limitations and impacts.

**Weaknesses**:
* All 3D results are 2D densities simply extruded,  so claims of “physically sound designs” are somewhat overstated for real engineering practice.
* Comparisons are limited (limited comparison methods and datasets/tasks). The paper omits stronger co-design baselines such as recent diffusion-based inverse design methods, making it hard to judge relative performance gains.
* The manuscript’s novelty derives mainly from combining these known techniques in one pipeline (most core components—SIMP compliance optimization, CLIP-based text supervision and Hill-type density projection are adopted from earlier papers.), not from introducing new algorithms or theory, which is more of a incremental. It would be best to explain what the methodological contributions are.

QUESTIONS:
Please refer to the "Weaknesses" section.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a structural optimization (topology optimization) technique for using text to add control of visual appearance. This is done through the addition of an augmented clip similarity loss and a compliance mask to remove disconnected components from a somewhat standard differentiable simulator based approach to topology optimization over a 2D pixel grid. Mapping of structural grids to the CLIP domain is achieved on the image side by interpreting density grids as greyscale RGB, and on the text side by biasing the text inputs towards black silhouettes by appending phrases such as \"dark, black solid outline\"\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- A simple, straightforward, and clean architecture that (except for a few details) should be easy to reproduce\n- Qualitative results look very good -- well aligned to input text prompts.\n\nWeaknesses:\n- Does not give details on the tuning necessary to achieve these results. Specifically, which weight parameters in the optimization loss were used, if these were consistent across trials or tuned for each one, and how much effort was used to do so. Similarly, was tuning (and possibly per-example tuning) required for the appended text strings to bias CLIP towards silhouette?\n- Does not give details on how the compliance masking works\n- Does not give any evaluation on how close to the target density design get (in absolute terms and compared to physics baseline). Since this is the only physics constraint (a soft constraint on density) that is explored, it is important to know how well this technique can balance constraints and optimization goals relative to a setup without the additional vision goal.\n\nQUESTIONS:\n- What, if any, tuning of parameters and CLIP text prompts (specifically the appended text like \"black\" and \"silhouette\") was needed to get these results, at what granularity was this done, and what were the final parameters used for the experiments? E.g., did the loss weights vary by image size, or by text target, or was there a universal set of values that worked well across experiments, and how big of an impact do these parameters have? Are there cases where one would want to use these weights to meaningfully control the designs?\n- Please give technical details on how the compliance masking works sufficient to fully reproduce the experiments.\n- How well is the density constraint achieved across the test examples, especially compared to the physics only baseline? Is there a useful tradeoff between aesthetic considerations and constraint satisfaction?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces TIDES, a method that jointly optimizes structural soundness and text-guided visual features in design. It combines a differentiable FEM simulator + a frozen CLIP model to generate 2.5D designs that are both physically viable and visually aligned but via CLIP guidance. Key contributions include a co-optimization framework, practical tricks like compliance masking and hill-based binarization, and some physical validation via 3D printed tests.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Quality: The method is technically sounds and includes good design choices like compliance masking and hill-based binarization. Experimental coverage is good and includes 3D-printed validation.\n- Clarity: The paper is well-written with clear figures and thorough ablations that support the claims.\n- Significance: Combines text-conditioned generative models with physics-aware structural design, and relevant in the AI-based computational design space.\n- Originality: While the general idea is not entirely new, the implementation is clean, practical, and backed by real fabrication.\n\nWeaknesses\n- Originality: The core idea (CLIP + FEM optimization) overlaps with prior work (e.g. Topology Optimization with Text‑Guided Stylization, Zhong et al. 2023) and is not acknowledged or compared against.\n- Scope: Designs are 2D with extrusion (~2.5D); while the paper references 3D printing and 3D design domains, the method itself does not support native 3D topology optimization or volumetric simulation\n- Visual control: Visual constraints rely solely on text-to-CLIP embeddings, with no direct geometric control, which limits precision and style control.\n- Limited generalization: The method is currently restricted to single-material, planar settings without manufacturability-aware constraints.\n\nQUESTIONS:\n- The core idea closely resembles Topology Optimization with Text-Guided Stylization Zhong et al. (2023), which also combines CLIP-based semantics with FEM optimization. Why is this not cited, and how does your method differ meaningfully?\n- At higher resolutions (e.g. 512×512), designs become delicate. Are such fine structures actually manufacturable with  3D printing methods?\n- The paper mentions 3D design, but the method operates on 2D grids with extrusion. Can it support native 3D FEM? If not, the terminology should be clarified.\n- How sensitive is the method to prompt phrasing, e.g. “robot” vs. “bot”? Some discussion on this would clarify the controllability and robustness of your approach.\n\nLIMITATIONS:\nYes, the authors include a limitations section (Appendix A) where they acknowledge the 2.5D constraint of their simulator, risks of misuse in generating real-world load-bearing structures, and the potential for bias amplification via CLIP. These points cover both technical and societal concerns sufficiently.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}