TopoGaussian: Inferring Internal Topology Structures from Visual Clues
Authors: Xiaoyu Xiong
Changyu Hu
Chunru Lin
Pingchuan Ma
Chuang Gan
Tao Du
Venue: ICLR 2025 Poster (2025)
Keywords: Gaussian Splatting
Differential Simulation
Topology Optimization
Neural Implicit Surface
Primary area: learning on graphs and other geometries & topologies

Abstract: We present TopoGaussian, a holistic, particle-based pipeline for inferring the interior structure of an opaque object from easily accessible photos and videos as input. Traditional mesh-based approaches require tedious and error-prone mesh filling and fixing process, while typically output rough boundary surface. Our pipeline combines Gaussian Splatting with a novel, versatile particle-based differentiable simulator that simultaneously accommodates constitutive model, actuator, and collision, without interference with mesh. Based on the gradients from this simulator, we provide flexible choice of topology representation for optimization, including particle, neural implicit surface, and quadratic surface. The resultant pipeline takes easily accessible photos and videos as input and outputs the topology that matches the physical characteristics of the input. We demonstrate the efficacy of our pipeline on a synthetic dataset and four real-world tasks with 3D-printed prototypes. Compared with existing mesh-based method, our pipeline is 5.26x faster on average with improved shape quality. These results highlight the potential of our pipeline in 3D vision, soft robotics, and manufacturing applications.

Published as a conference paper at ICLR 2025
TOPOGAUSSIAN: INFERRING INTERNAL TOPOLOGY STRUCTURES FROM VISUAL CLUES
Xiaoyu Xiong', Changyu Hu', Chunru Lin?, Pingchuan Ma', Chuang Gan', Tao Du
1 Tsinghua University University of Massachusetts Amherst
3 Massachusetts Institute of Technology 4 "Shnnhaii Qi Zhi Institute
ABSTRACT
We present TopoGaussian Da holistic, particle-based pipeline for inferring the in- terior structure of opaque object from easily accessible photos and videos input. Traditional mesh-based approaches require tedious and error- prone mesh filling and fixing process, while typically output rough boundary surface. Our pipeline combines Gaussian Splatting with a novel, versatile particle-based differ- entiable simulator that simultaneously accommodates constitutive model, actua- tor, and collision, without interference with mesh. Based on the gradients from this simulator, we provide flexible choice of topology representation for optimiza- tion, including particle, neural implicit surface, and quadratic surface. The resul- tant pipeline takes easily accessible photos and videos as input and outputs the topology that matches the physical characteristics of the input. We demonstrate the efficacy of our pipeline on a synthetic dataset and four real-world tasks with 3D-printed prototypes. Compared with existing mesh- based method, our pipeline is 5.26x faster on average with improved shape quality. These results highlight the potential of our pipeline in 3D vision, soft robotics, and manufacturing appli- cations.
INTRODUCTION
The idea of seeing through the surfaces of an opaque object is an intriguing concept contemplated by scientists, engineers, and science- fiction writers. The capability of inferring the internal topolog- ical structure of an object from its surface information unlocks many creative applications, e.g., 3D model design and industrial manufacturing. Traditional approaches from scientific or engineering disciplines either require intrusive probing sensors operate on expensive computed tomography (CT) or magnetic resonance imaging (MRI) equipment, underscoring the challenge of this prob- lem. Recent developments in 3D vision provide powerful tools for reconstructing 3D scenes with easily accessible, non-intrusive visual input only, e.g., NeRF (Wang et al. 2021) or Gaussian Splat- ting (Kerbl al., (023) with photos or videos input. However, these methods only provide visually plausible surface information of objects with no or hallucinated interior topology structure.
This paper focuses tackling the following problem:
Given the motion (images or videos) of the object, find one interior topological design that can explain this motion physically. We develop a holistic pipeline for inferring the interior topology of an opaque object in order recover its major physical characteristics with easily accessible visual input (photos and videos) only. Our pipeline, TopoGaussian, begins with collecting multi-view images and uses Gaussian splatting obtain point cloud encoding the object's 's exterior surface shape and appearance (Kerbl et al., 2023). Next, we construct a volumetric point cloud by filling interior points (Xie al. 2023). and we use topology representation to attach the physical parameters to each point. Here we provide three options to encode the topology structure. One option depicts the boundary directly based on a topology parameter of each particle, and map this parameter to its physical attribute, inspired by standard topology optimization methods on voxels (Sigmund) (001a)) and particles (Li et al. 2021]; parametric boundary surface based neural network (Park et al.)
the other two options leverages 2019) and quadrics (Hafner et al. 2024) respectively to represent the topology structure, and assign physical attributes based this boundary. To accommodate the particle representation, develop
htaps://topo-gasssine wtwp://wwwwwaiaaaaiiuiii/iiaaa/e hinp

---

Published as a conference paper at ICLR 2025
novel, versatile particle-based based differentiable simulator with simultaneous support for rigid and soft objects with different topology structure. Finally, we compare the simulation results and the object's dynamic motion in video and optimize the loss defined on the difference between them. With access to the simulation gradient, we run gradient-based methods to optimize the parameters in topology representation and obtaín interior topology structure that leads similar motions established in the input video. Our pipeline presents a mesh-free representation for flexible and expressive modeling, simulation, and optimization of opaque objects gracefully unified in holistic framework of particles only. This avoids tedious and a more smooth and AD--prnt-friendd
mesh fixing and filling, and outputs result which can be used in manufacturing. While traditional, intrusive methods can output an accurate structure that we cannot match, they require expensive and complex equipment. Our non-intrusive pipeline only uses easily accessible images and videos as input with low cost, exhibiting potential in the applications that focus on general outside behaviours (like generative models).
We design several synthetic tasks and four real-world tasks to demonstrate the efficacy of our pipeline. We present a synthetic dataset and perform benchmark experiments comparing our method with two strong mesh-based based baselines PGSR (Chen et al. 2024) and Gaussian Surfels (Dai et al.) 2024). based on the processing time and reconstruction quality (3D printability). Our method is 5.26x faster and 4.81x faster than PGSR in two different setrings, respectively, and 1.28x faster than Gaussian Surfels, where Gaussian Surfels fails to output visually correct result in two examples. Meanwhile, our method exhibits significantly higher quality of reconstructed topology structure than the baselines, with 2.33x, 2.5x and 2.55x respectively. Then we 3D print one example to verify the feasibility in real manufacturing. Moreover, we extend our method to more synthetic validations, four real world validations, and several ablation studies, which show the ability of our pipeline to handle input under various conditions.
In summary, this work makes the following contributions:
We develop a novel, non-intrusive pipeline for reconstructing motion with a physically correct internal topology structure of solid objects from visual inputs only, i..., multi-view images and videos of motion.
We present a particle-based differentiable simulation compatible with three flexible topology rep- resentations including particle, neural implicit surface, and quadratic surface. Together with Gaus- sian splatting, our simulation and optimization methods lead to a completely meshless pipeline.
We demonstrate the quality and efficiency of our pipeline quantitatively against two strong base- lines, and we further validate the generalizability our method on a wide range of synthetic and real-world scenarios with diverse yet complex physics.
2 RELATED WORK
3D Scene Reconstruction
Reconstructing 3D scenes from visual inputs is fundamental problem extensively studied computer vision for decades. Recent representative methods for this prob- lem include NeRF (Wang et al., 2021) and Gaussian Splatting (Kerbl et al.) 2023). NeRF bases its scene geometry representation on implicit surfaces Iearnable from multi-view images. Several improvements conducted in its speed, quality, and applicability in dynamic and deformable ob-
jects (Barron et al. 202T Deng et al. 2022 Pummarlla et al. 2021a Martin-Brualle et al.) 2021) Gaussian Splatting Kerbl et al. 2023) is optimization saanee based and built upon point-cloud represen- tation, and follow-up studies have proposed various improvements in its efficiency and reconstruc- tion quality (Chen al. 2023 Matsuki et al. 2023 Tang et al. 2023 Wu al. 2023 Yi et al., 2023 In particular, Xie et al. (223) showcases novel synthesis of motions from editing a dynamic model recovered from Gaussian Splatting, and Guo et al.] (2024) reconstructs physically compati- ble objects using a quasi-static simulation. However, these works primarily focus on reconstructing visually plausible surface information of an object without inferring a dynamically plausible volu- metric structure.
Differentiable Simulation Differentiable simulation extends traditional simulation with gradients and revives gradient-bare based optimization in graphics, learning, and robotics applications YYu et al.) 2023 Werling et al. 2021 Xu et al., 2021 Lin et al.) [021]] Qiao et al. [202]] Li et al., (022). Pre- vious studies have developed differentiable simulators for various physical systems including rigid
2

---

Published as a conference paper at ICLR 2025
(a) Input
(b) Pre-processing
(c) Optimization
Vileo Matios
Malti-sie Images
Cruusiia Splatting
Refineenee Filliag
Valunctri
Toyelags Reproccntatios
Pozzaa
(d) Output
Ieffrrntiable Navalattes
Physical Parameters
Nanma
Tapeing Sirveture
Opuimioed to
Datt
Muu Diathaake
Figure I: Pipeline overview (See. 3p. Our pipeline takes as input multi-view photos of an opaque object and its video of motion. We run Gaussian splatting on the multi- -iew images to obtain point cloud characterizing its surface geometry and appearance. Next, we refine and fill in internal points to obtain volumetric point cloud and use our topology representation (with three flexible choices) to attach physical parameters on We then simulate the volumetric point cloud with our particle- based differentiable simulator, which compares its simulated motion with that in reference image or video and backpropagates the gradient of the motion difference to the topology representation. Finally, we perform the optimization algorithm based the gradient from the simulator and obtain the resulting structure that matches the input motion.
objects (Geilinger al., 2020 Wang al. 2019) deformable solids (Liu al. 2023 Gjoka al. 2022). cloths YYu et al. 2023), and fluids (Li et al. 2024). Many of these differentiable simulators mesh- fimie-element-based due their beneñts handling irregular shapes with explicit sharp boundary representation. However, these methods require high-quality mesh discretization, which can be computationally expensive to obtain. In contrast these works, we focus on purely particle-based differentiable simulator supporting shapes with different topology structures in ac- commodation with Gaussian splatting representation. This avoids tedious mesh fixing and filling by native support for particle-only pipeline.
Topology Optimization Topology optimization an optimization based structure design tech- nique originating from structural and mechanical engineering (Sigmund [000aa] Eschenauer & Ol- hoff 2001 Wu et al. 2021 Deaton & Grandhi 2014). Traditional topology optimization techniques consider density- field representation of structural geometry a regular grid and optimizes in- dependent decision variables at each voxel for optimal structural or mechanical performance, e.g., minimum compliance (Sigmund, 2001aa] Yang Chen 1996 Liu al. 2018). Previous studies have also explored the idea with different geometric representations, e.g., level sets (Wang et al., 2003) and particles (Li et al. 202T). Recent years, neural implicit surface method has been pro- posed Zehnder et al., 2021j, which encodes the shape with deep neural network (Park et al., 2019). Moreover, for rigid bodies, |Hafner al. (2024) has proved that quadratic surfaces are suf- ficient for topology optimization problems that only involve rigid physical characteristics. Existing topology optimization works primarily focus on applications in engincering disciplines, c.g., struc- tural and mechanical analysis, acronauties, and architecture (Jensen & Sigmund 2011 Zhu et al. 2016 Dühring et al. 2008), while our pipeline extends topology optimization technique to the ap- plications in 3D computer vision and robotics.
METHOD OVERVIEW
The overarching goal of our pipeline is to find the internal topology structure of opaque object to match its physically simulated motion with its dynamic motion from a reference input. Our pipeline contains four steps: volumetric shape generation (Sec. 4 which takes input multi-view photos a possibly hollowed opaque object and generates densely sampled volumetric point cloud; topology representation (Sec. 5) which encodes the topology structure with flexible topology
3

---

Published as a conference paper at ICLR 2025
representations and attaches physical parameter (e.g. density Young's modulus) to each point in the point cloud; simulation (Sec. 6). which evaluates the dynamic motion of the point cloud for a given interior topology design configured by each point's physical parameter. optimization (Sec. 7). which runs gradient-based optimization to adjust the parameters in the topology representation based on the loss of the simulated motion and the reference motion, and output shape that replicates the dynamic motion of the object in a reference video. Fig. [ gives an overview of the pipeline.
4 VOLUMETRIC SHAPE GENERATION
We begin by collecting multi-view photos of the object and running Gaussian Splatting due to its efficient and high-quality reconstruction of 3D objects from images. The output point cloud from Gaussian Splatting gives visually and geometrically plausible surface information about the object based on particles. This point-cloud representation of surface information serves as basis towards the optimization interior topology structure. Following PhysGaussian (Xie et al. (2023). and fill in points to the output point cloud from Gaussian Splatting, leading to a clean and densely sampled volumetric point cloud {X;; } where N denotes the number of points, X, their 31) locations, and Vú the volume associated with X; The point cloud can be viewed as particle-based discretization of the solid object.
refine
5 TOPOLOGY REPRESENTATION
Now we have volumetric point-cloud, and we need to represent different interior topology within including pneumatic hollows and bone-musele structures. Based on the point-cloud and previous work (Li et al. 2021 Park et al.) 2019 Hafner et al.) 2024). we provide two topology representations,
point-based and surfcc-bbssd. explained Sec. 5. and Sec. 5.2 respectively.
5.1 POINT REPRESENTATION
For an internal topology structure, we use an indicator function r(x) to define. For example, in a pneumatic actuator, use r(x)= 1 to indicate the hollow part and r(x) 0 to indicate the solid part. Then we discretize this indicator function to the point-cloud by assigning a value to each point that controls the nearby area of this point. While strict binary variable is discrete and not compatible with gradient-based optimizer (Sigmund) 2001b), we consider assigning a decision variable to each point and using sigmoid function to map it to the binary variable. Then the topology structure is represented by this decision variable.
5.2 SURFACE REPRESENTATION
Point representation exhibits highly flexible topology representation, but it is hard to encode specific geometric prior information into it. Based on this, our pipeline further provides two topology repre- sentations based on parametric surface. We use a signed distance function (SDF) s(x) to encode the boundary surface of the object, where signed distance from this position to the boundary surface. Then the surface is the zero level set (s(x) = 0) of the SDF (Osher et al. 2004). Similarly, we discretize the SDF the point-cloud, and use a sigmoid function to map the signed distance to a binary indicator function. We provide two concrete SDF representations to parameterize the surface: neural networks (Park et al. 2019) and quadrics (Hainer et al. 2024). In particular, quadrics are proved to be sufficient for topology optimization on specific rigid-body dynamic tasks (Hafner et al. 2024) with only 10 parameters.
any position inside the object, and s(x) outputs the
6 SIMULATION
With the volumetric point-cloud containing physical parameters, the next step in our pipeline in- volves simulating the dynamic motion which we can draw comparison with the input motion and perform optimization. Our work focuses rigid and soft solid objects, both of which have ma- ture simulation solutions based on (finite-clement) mesh representations (Sfakis & Barbic (2012a); Sin et al. (2013): Lee al. (2018). However, as pointed out, obtaining high-quality mesh is
4

---

Published as a conference paper at ICLR 2025
Six rigid experiments (Sec. 8.2). For each experiment, left: initial balancing position of the object with fully solid topology structure; middle: final balancing position the object with the result from pipeline; right: optimization target.
Figure 2:
non-trivial, requiring error-prone and tedious efforts, while the output from Gaussian Splatting is point-cloud. Therefore, our pipeline proposes to consider a fully particle-based simulator instead and aims build completely mesh-free pipeline. We note that previous works like PhysGaus- sian (Xie et al., (022) have discussed similar motivation behind fully particle-based pipeline, but we are faced with extra technical challenges from representing the internal geometric structure the particle system. We discuss the detail of our simulator in the following parts: constitutive models, actuation model, collision model, time integration and differentiability.
6.1 CONSTITUTIVE MODELS
Rigid Bodics The dynamic motion of rigid bodies is governed by the Newton-Euler equa- tion (Lanczos 2012 Liu & Jain 2012) on six DoFs. Specifically, the governing equation requires access to the mass mi, the center mass c, and the moment inertia I. From the indicator function r(z) defined before,
calculate the mass of the particle near i from m, pVr(w,,). Then the dynamic properties can computed from {(x,, m,) in a straightforward manner. For example, the Σ; Mi, and the center of mass be computed as
Σ, mma0
m
Similarly, the moment of inertia can be computed from
Σ
-m,(2; c|[ c).
where maps a vector its cross-product (skew-symmetric) matrix.
Soft Bodies We consider simulating soft body made of hyperelastic materials. The critical step is to estimate the deformation gradient F,, a 3 x 3 tensor characterizing the local deformation around each X;. Here, we follow the standard practice in particle-based simulation for deformable solids (Mäller et al.) (2055] 2022) to construct F, More concretely, given deformed point cloud {æx} during simulation, we estimate for any point æ, its F, with the method in Becker et al. 2009);
F. =
(x, x.) (X, X.) -x,)
where j loops over particles in a neighborhood of X, and R. a rotation transformation based on the relative poses of these neighboring particles (Becker et al. 2009). With the definition of F; hand, we can calculate the energy density and stress tensor P based on the Saint Cenant-Kirchher model (Sifakis & Barbic 2012b).
5

---

Published as a conference paper at ICLR 2025
Table Reconstruction quality.
The lower the better. The optimal results are bolded.
Method
frog
swim-ring
cow
kangaroo
pear
tumbler
Ours
0.306
0.242
0.300
0.388
0.245
0.305
PGSR (voxel size=0.05)
0.745
0,464
0.410
0.841
1.078
0.953
PGSR (voxel size=0.2)
0.781
0.894
0.801
0.480
1.027
0.755
Gaussian Surfels
0.851
0.882
Fail
0.976
1.148
0.963
rooel
1750
PiGk Caloiar So shppow)
Cousiia Semtir Sendriceppie
10-4
Time
10'4
30"4
Figure 3: Comparison on optimization loss (left) and preprocessing time (right). "Fail" the top of the bar means that the method fails to output visually correct result in the example.
6.2 ACTUATION MODEL.
In order to extend the applications of the previous soft-body model and design soft bodies which can perform active motion, we adopt the actuation model in soft robotics. In particular, the pneumatic chamber embedded in the soft body is typically characterized by customized chamber shape design invisible from the soft body's exterior surface. We assume that the air pressure force induced by the pneumatic actuator is the conservative force from the volume energy (Luo al. 2022):
= pdv,
Ea Ea({x;)) =
(4)
where denotes the air pressure. Note that the integral domain is the volumetric region of the deformed shape. With F; at hand, we can approximate Ep as
E. Σ pv, ΣPlEIV.
where F,J represents the determinant of F, Therefore, to support pneumatically actuated soft-body simulation, we use the conservative force derived from E., as the actuation force.
6.3 COLLISION MODEL
We follow the existing study penalty-based collision simulation (Andrews et al. 2022 build our collision model, leveraging the advantage of the penalty- based method that it properly balances accuracy, speed, and differentiability. More concretely, we add for each particle of the soft body penalty energy density
1 k (min(8 d: 03) Fpp
6
2
to discourage it from penetrating the obstacles. In Eqn. (6) k is the penalty stiffness, 8 is small translation parameter to avoid penetration, and d; is the signed distance between particle and the obstacle (e.g. ground). Then, the contact force is also the conservative force derived from the penalty energy.
6.4 TIME INTEGRATION
With the total energy and force of the whole system, perform time integration solve the motion. For rigid bodies, based on the Newton-Euler equations (Lanczos) 2012)) Liu & Jain 2012). perform RK2 integration. For soft bodies, Leapfrog integration for non-actuation cases

---

Published as a conference paper at ICLR 2025
(vosolairee-.o stre-O
POSA
FCSR www--h0-0.o
Suftll
Ceussian
Ground Trath
kangaroo
frog
tumbttr
ring
pear
Figure 4:
Comparison between method and mesh baselines (Sec. 8.1 Red part represents solid
part and blue part represents hollow part. Top row: our method; bottom three rows: mesh baselines.
and implicit Euler integration with incremental potential (Martin et al. 2011 in actuation cases. The details of incremental potential can be found in Appendix Sec. A.2
6.5 GRADIENT COMPUTATION
We apply the chain rule and the standard adjoint equation (Du et al. 2021 Geilinger et al. 2020
Hahn al. 2019 McNamara et al. (2004) to equip the proposed particle based simulator with gradi- ent information. Detailed derivations of the gradients in rigid-body and deformable-body simulators
can found in previous works, e.g., ChainQueen (Hu et al. 2019) and DiffPd (Du al. 2021
for deformable bodies, Huang al. (2024) for pneumatic actuators, and Andrews et al. (2022 for collision models.
7 OPTIMIZATION
The final step of our pipeline is the optimization based on the L2 loss from the difference between the simulation motion from the previous section and the input reference motion. (See Fig, [ We extract certain motion characteristics (c.g. vibration frequency, balancing pose) and define the loss based the characteristics. The optimization variable is the parameters in the topology represen- tation (Sec.5). We use gradient-based optimizer (L-BFGS-B B and Adam), reading the loss from the differentiable simulator (Sec. 6.5).
RESULTS
8.1 EXPERIMENT SETUP
Task Description We comprehensively evaluate our method on diverse tasks, including nineteen rigid bodies and two soft bodies with static and dynamic settings and their extension four real world experiments. Each task takes multi-view images and the motion video of the object as input. Our goal to find an interior topology structure whose simulated motion matches the reference motion, characterized by static/dynamic features like balancing position or frequency. Fig. 2 and Fig. 5 exhibit several examples these tasks.
Baselines
Our main task is build particle- based pipeline find the physically plausible in- ternal topology structure, which is a new task with few baselines. Recent PhysGaussian (Xie et al. 2023 does not consider the optimization of internal topology structure. Therefore, build the

---

Published as a conference paper at ICLR 2025
Figure 5: One soft body expereiment (Sec. 8.2). Left part: Top row: frames from simulated motion of a fully soft initial guess; middle row: simulated motion of our optimized result; bottom row: target motion. The red rectangles mark the major different parts. Right part: slice the optimized topology structure with red representing soft part and blue representing hard part.
baseline based on two current nesh-based Gaussian Splatting SOTA PGSR (Chen al. 2024) and Gaussian Surfels (Dai et al. 2024). We replace the topology representation with the output triangle surface mesh from the model, and chain into the rest of our pipeline. By comparing the efficacy of mesh- and particle-based methods, we show the advantages of insisting fully particle-bast method.
Metries We don't compare the output shape with the shape set in generating the target motion due to two reasons. One reason is that only consider image/video input, and especially in rigid tasks, this comparison is ill-defined since multiple solutions can share the same motion, which means that this metric fails measure the performance of the method. The other reason is that in practical applications, this ground truth topology structure can not be easily acquired, requiring expensive equipment like CT/MRI. Instead, we use the following three metrics test the performances:
The 1.2 loss between simulation motion and reference motion (Sec. 7 Time: Volumetric shape generation time from reading multi-view images input simulation start (Sec. 4 Hnnd Sec. 55)
Optimization Loss:
Smoothness: The Laplacian of surface normal. This motivated by manufacturing applications (e.g., 3D printing) which favor smooth surface.
8.2 MAIN RESULTS
Rigid Body Rigid bodies with different shapes are hung by string at given position, and our goal is to optimize the topology structure of their internal cavities to remain horizontal. As this static equilibrium example, parameter configuration s. We stress that varying s changes the interior shape, which adjusts the weight distribution of the hawk and, therefore, affects the center of mass. We define the loss the distance between c and the hanging position. Fig. 2 shows six examples and visualizes the initial, optimized, and target pose of the rigid bodies. The full results can be found in Appendix Sec. C.2
first extract the center of mass of the hawk from shape
Fig. 3 exhibits the optimization loss and volumetric shape generation time of our method and each baseline the six experiments, and the full figure can be found in Appendix Sec. C It can be observed that all these methods exhibit similar and sufficiently small final loss, meaning that they all simulate a motion close reference. On the other hand, for volumetrie shape generation time, our method is 5.26 times faster than PGSR (voxel size=0.05), 4.81 times faster than PGSR (voxel size=0.2), and 1.28 times faster than Gaussian Surfels on average. This result shows that our pipeline is more efficient owing to leveraging the flexibility of particles, thus avoiding expensive mesh processing.
8

---

Published as a conference paper at ICLR 2025
Figure 6: Wobbly doll (Sec. 8.3). Left: photo of the wobbly doll. Middle: frames from video recording of the harmonic oscillation of the doll (top), simulated motion with an interior design optimized from our pipeline (middle), simulated motion of solid doll (bottom). Right: a cross- section visualization of the optimized interior geometry from our pipeline, where black represents solid part and grey represents hollow part.
For the quality of reconstruction, we try neural implicit surface and quadrics topology representations in our methods, and find that there no obvious quality difference between two representations, which corresponding to the conclusions in Hafner al. (2024). Tab. [ lists the quality index of our method and the baselines on the six ex- amples, and Fig. 4 visualizes these results (complete data and visu- alization results in Appendix Sec. C.2). More concretely, the quality of our method is 2.33 times better than PGSR (voxel size=0.05), 2.50 times better than PGSR (voxel size=0.2), and 2.55 times better than Gaussian Surfels on average. Furthermore we observe that in the cow task. Gaussian Surfels fails to output a visually correct result. This showeases the advantages of the our particle-based pipeline, which does not require an expensive and error-prone high quality mesh division. Based on the smooth geometry from our pipeline, we 3D print the pear example to exhibit our application in manufacturing, Moreover, we perform several ablation studies shown in Appendix Sec. C.4
Left: optimized result; right: fully solid initial guess. Red line: target pose; green line: actual pose.
We also extend our synthetic dataset to soft bodies with two experiments with point topology representation. Here we introduce one of them and the other can be found in Appendix Sec. The experiment is soft pear bouncing on the ground, with a hard part inserted into it. Different topology structures of this hard part lead different bouncing behaviors, while our goal is to optimize this structure to reproduce the target video. The loss is defined the distance of the motion of several pivot points on the pear to the target.
Soft Body
Fig. 5 visualizes the result of the experiment. The fully soft initial guess output a motion deviating large from the target, while our optimized result can closely reproducing the target motion.
8.3 REAL-WORLDD VALIDATIONS
Finally, we validate our pipeline on four real-world experiments. These experiments include rigid and soft bodies with static and dynamic settings. One rigid and one soft body examples are shown here, and the rest can be found in Appendix Sec. C.3
Wobbly Doll The task involves inferring the interior structure of a wobbly doll, a classic rigid- body toy with oscillating motions (Fig. 6 left and top middle). The interior of wobbly doll is typically non-empty. The goal is to infer one possible interior structure of a wobbly doll from video recording of its harmonic oscillations (Fig. top middle).

---

Published as a conference paper at ICLR 2025
Figure 7:
Pneumatic finger (Sec. 8.3). Left: a photo of the pneumatically actuated soft finger. Middle left: the bending finger with maximal air pressure. Middle: bending finger simulated with an optimized design. Middle right: the finger simulated with an initial guess of the chamber location; Right: the optimized chamber location where the black is solid part and the grey is chamber.
We define an objective function that penalizes the discrepancy of the oscillation period between the reference video and the simulated motion. We extract the period of oscillation as well as the maximal tilting angle 0 by OpenCV. Then, we compare these reference values with the simulated value from the physical parameters and compute the loss.
Fig. (right) shows our optimized interior structure of the doll, where most of the mass is clustered to one side of the bottom. The simulated motion of this optimized design (Fig, middle) closely matches the maximal tilting angle and the period the recorded video (4.0Hz. See Fig. toop middle). In contrast, an initial guess of a fully solid doll leads to a substantially different oscillation period (2.4Hz. See Fig. 6).
Pneumatic Finger This experiment is inspired by a classic application in soft robotics, which showcases the ability of our pipeline in processing soft bodies with pneumatic actuators. The task is to infer the chamber design an opaque, pneumatically actuated soft finger, a widely used model in soft robotics (Fig. left). This task is substantially more challenging than previous ones as it involves deformable bodies and pressure bending motion of the soft finger with increasing air pressure, our goal is to deduce the inflatable chamber design to reproduce the final bending angle (Fig. middle left). We extract reference angle from input by OpenCV, and compare it with our simulated value. Fig. midddle right shows that there no obvious bending with initial guess, and a visually correct bending with our final result. The result indicate that we need to add small chamber in the middle bottom part, which is reasonable to achieve the target bending.
pneumatic chamber. Given a video recording the
9 CONCLUSIONS AND LIMITATIONS
We present a holistic pipeline for optimizing the interior topology structure of opaque objects from visual inputs. Our pipeline consists of Gaussian Splatting, topology optimizer with flexible repre- sentation choices, and a novel differentiable point-cloud simulator. Our whole pipeline built upon a gracefully unified particle-based representation free from error-prone mesh extraction and process- ing in modeling, simulation, and optimization. We validate our pipeline on a diverse set of tasks, covering rigid bodies, soft bodies, collision and actuation with various static and dynamic scenes. Through the comparison with mesh-based methods, we show that our particle- based method has the advantages of higher speed and smoother output. Our pipeline enables promising applications in reverse engineering tasks in 3D vision and soft robotics.
Our work has several limitations. First, our pipeline currently focuses on a single object with two types of material. Second, Dynamic NeRF (Pumarola et al.) 2021b) and 4D Gaussian (Wu et al.) 2024) also performs 3D scene reconstruction and we do not compare with them. The main reason is that these works only consider the surface information without guarantee of a physically plausible output, meaning that they may not be able to handle the manufacturing applications like 3D printing. Despite of this, it would still be interesting compare optimization, especially for neural implicit surface and soft bodies, still costs certain time and is only offline. It would be promising accelerate them to real time and supports interactive demos.
approach with them. Third, current
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to thank Tsinghua University, Shanghai Qi Zhi Institute, University of Massachusetts at Amherst, and the Massachusetts Institute of Technology for their financial support throughout this research. We also extend our gratitude to Zijian Lyu from Tsinghua University for his technical assistance and insightful discussions. Special thanks to Shuguang Li from Tsinghua University for providing the necessary hardware support that greatly facilitated our work.
REFERENCES
Sheldon Andrews, Kenny Erleben, and Zachary Ferguson. Contact and friction simulation for com- puter graphics. In ACM SIGGRAPH 2022 Courses, pp. 1-172. 2022.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: multiscale representation for anti- neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5855-5864, 2021.
Markus Becker, Markus Ihmsen, and Matthias Teschner. Corotated sph for deformable solids. NPH, 9:27-34, 2009.
Danpeng Chen, Hai Li, Weicai Ye, Yifan Wang, Weijian Xie, Shangjin Zhai, Nan Wang, Haomin Liu, Hujun Bao, and Guofeng Zhang. Pgsr: Planar-based gaussian splatting for efficient and high-fidelity surface reconstruction. arXiv preprint arXiv:2406.06521 2024.
Zilong Chen, Feng Wang. and Huaping Liu. Text-to-3d using gaussian splatting. arXiv preprint arXiv:2309.16585 2023.
Pinxuan Dai, Jiamin Xu, Wenxiang Xie, Xinguo Liu, Huamin Wang, and Weiwei Xu. High-quality surface reconstruction using gaussian surfels. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
Joshua D Deaton and Ramana V Grandhi. A survey of structural and multidisciplinary continuum topology optimization: post 2000. Structural and Multidisciplinary Optimization, 49:1-38, 2014.
Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12882-1288 12891, 2022.
Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg. Daniela Rus, and Wojeiech Matusik. Diffpd: Differentiable projective dynamics. ACM Transactions on Graphics (TOG), 41 (2):1-21, 2021.
Maria B Dühhing, Jakob S Jensen, and Ole Sigmund. Acoustic design by topology optimization. Journal of sound and vibration, 317(3-5);557-57,, 2008
Hans A Eschenauer and Niels Olhoff. Topology optimization of continuum structures: review* Applied Mechanics Reviews, 54(4):331-390, 07 2001. ISSN 0003-6900. doi: 10.1115/1.1388077
Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski. and Stelian Coros. Add: Analytically differentiable dynamics for multi-body systems with frictional contact. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020.
Arvi Gjoka, Zizhou Huang, Davi Colli Tozoni, Zachary Ferguson, Teseo Schneider, Daniele Panozzo, and Denis Zorin. Differentiable solver for time-dependent deformation problems with contact. arXiv preprint 1XXx:2225.11644 2022.
Minghao Guo. Bohan Wang, Pingchuan Ma. Tianyuan Zhang, Crystal Elaine Owens, Chuang Gan, Joshua B. Tenenbaum, Kaiming He, and Wojeiech Matusik. Physically compatible 3d object modeling from single image. Advances in Neural Information Processing Systems, 37, 2024.

---

Published as a conference paper at ICLR 2025
Christian Hafner, Mickaël Ly, and Chris Wojtan. Spin-it faster: Quadrics solve all topology opti- mization problems that depend only on mass moments. ACM Transactions on Graphics (TOG), 43(4):1-13, 2024.
David Hahn, Pol Banzet, James M Bern, and Stelian Coros. Real2sim: Visco-elastic parameter estimation from dynamic motion. ACM Transactions Gruphics (TOG), 38(6): 2019.
Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua B Tenenbaum, William T Freeman, Jiajun Wu, Daniela Rus, and Wojciech Matusik. Chainqueen: A real-time differentiable physical simu- lator for soft robotics. In 2019 International conference on robotics and automation (ICRA), pp-
IEEE, 2019.
Zizhou Huang, Davi Colli Tozoni, Arvi Gjoka, Zachary Ferguson, Teseo Schneider, Daniele Panozzo, and Denis Zorin. Differentiable solver for time-dependent deformation problems with contact. ACM Transactions Graphics, 43(3):1-30, 2024.
Jakob Søndergaard Jensen and Ole Sigmund. Topology optimization for nano-photonics. Laser & Photonics Reviews, 5(2):308-32) 2011.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
Comelius Lanczos. The variational principles of mechanics. Courier Corporation, 2012.
Jeongseok Lee, Michael X. Grey, Sehoon Ha, Tobias Kunz, Sumit Jain, Yuting Ye, Siddhartha S. Srinivasa, Mike Stilman, and C Karen Liu. Dart: Dynamic animation and robotics toolkit. The Journal of Open Source Software, 3(22):500, 2018.
Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang. and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for soometry-agnostie system identification. In The Eleventh International Conference on Learn- ing Representations, 2022.
Yifei Li, Yuchen Sun, Pingchuan Ma. Eftychios Sifakis, Tao Du Bo Zhu, and Wojeiech Matusik. Neural fluidic system design and control with differentiable simulation. Advances in Neural Information Processing Systems, 37, 2024.
Yue Li, Xuan Li, Minchen Li, Yixin Zhu, Bo Zhu, and Chenfanfu Jiang. Lagrangi multi-density topology optimization with the material point method. International Journal for Numerical Methods in Engineering, 122, 03 2021. doi: 10.1002//me.6668.
Xingyu Lin, Zhiao Huang, Yunzhu Li, Joshua B Tenenbaum, David Held, and Chuang Gan. Diff- skill: Skill abstraction from differentiable physics for deformable object manipulations with tools. In International Conference on Learning Representations, 2021.
Karen Liu and Sumit Jain. quick tutorial on multibody dynamics. Online tutorial, June, pp- 7, 2012.
Haixiang Liu, Yuamming Hu, Bo Zhu, Wojeiech Matusik, and Eftychios Sifakis. Narrow-band topology optimization on a sparsely populated grid. ACM Transactions on Graphics (TOG), 37 (6):1-14, 2018.
Min Liu, Gang Yang, Siyuan Luo, Chen Yu, and Lin Shao. Softmac: Differentiable soft body simulation with forecast-basse based contact model and two-way coupling with articulated rigid bodies and clothes. arXiv preprint arXiv:2312.03297.200 2023.
Yiyue Luo, Kui Wu, Andrew Spielberg. Michael Foshey, Daniela Rus, Tomás Palacios, and Wo- jeiech Matusik. Digital fabrication of pneumatic actuators with integrated sensing by machine knitting, In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-13, 2022.
Sebastian Martin, Bernhard Thomaszewski, Eitan Grinspun, and Markus Gross. Example-bar elastic materials. In ACM SIGGRAPH 201 papers, pp. 1-8. 2011.
12

---

Published as a conference paper at ICLR 2025
Ricardo Martin Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovit- skiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 7210-7219, June 2021.
Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew Davison. Gaussian splatting slam. arXiv preprint arXiv:2312.0674 2023.
Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. Fluid control using the adjoint method. ACM Transactions On Graphics 240)444500 2004.
Joseph J Monaghan and John C Lattanzio. A refined particle method for astrophysical problems. Astronomy and Astrophysics (ISSN 0004-6361), vol. 149, no. 1, Aug. 1985, 135-143., 149: 135-143, 1985.
Matthias Müller, Bruno Heidelberger, Matthias Teschner, and Markus Gross. Meshless deformations based on shape matching. ACM transactions on graphics (TOG), 24(3):471-4)) 1478, 2005.
Matthias Müller, Miles Macklin, Nuttapong Chentanez, and Stefan Jeschke. Physically based shape matching. In Computer Graphics Forum, volume 41, pp. 1-7. Wiley Online Library, 2022.
Stanley Osher, Ronald Fedkiw, and K Piechor. Level set methods and dynamic implicit surfaces. Appl. Mech. Rev., 57(3):B15-115, 2004.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 165-174, 2019.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10318-0022,,, June 2021a
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318-10327, 2021b.
Yiling Qiao, Junbang Liang, Vladlen Koltun, and Ming Lin. Differentiable simulation of soft multi- body systems. Advances Neural Information Processing Systems, 34:17123-17135, 2021.
Eftychios Sifakis and Jernej Barbic. Fem simulation of 3d deformable solids: a practitioner's guide to theory, discretization and model reduction. In ACM SIGGRAPHI 2012 Courses, SIGGRAPH '12, New York, NY, USA, 2012a. Association for Computing Machinery. ISBN 978 9781450316781 doi: 10.114522343483.343550
Eftychios Sifakis and Jernej Barbic. Fem simulation of 3d deformable solids: a practitioner's s guide to theory, discretization and model reduction. Acm siggraph 2012 courses, pp. 1-50. 2012b.
Ole Sigmund. Sigmund, o.: A 99 line topology optimization code written in matlab, structural and multidisciplinary optimization 21, 120-127. Structural and Mulidissiplinary Optimization, 21: 120-127, 04 2001a. doi: 10. 10000//001500000170
Ole Sigmund. 99 line topology optimization code written matlab. Structural and multidisci- plinary optimization, 21:120-127, 2001b.
Fun Shing Sin, Daniel Schroeder, and Jernej Barbič. Vega: non-linear fem deformable object simu- lator. In Computer Graphics Forum, volume 32, pp. 36-48. Wiley Online Library. 2013.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint vrXiv:2309.16655 2023.
Michael Yu Wang. Xiaoming Wang, and Dongming Guo. A level set method for structural topology optimization. Computer methods in applied mechanics and engineering, 192(1-2):2272246 2003.
13

---

Published as a conference paper at ICLR 2025
Ying Wang, Nicholas J Weidner, Margaret A Baxter, Yura Hwang, Danny M Kaufman, and Shinjiro Sueda. Redmax: Efficient & flexible approach for articulated dynamics. ACM Transactions on Graphics (TOG), 38(4):1-10, 2019.
Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07066 2021.
Keenon Werling, Dalton Omens, Jeongseok Lee, loannis Exarchos, and Karen Liu. Fast and feature-complete differentiable physics engine for articulated rigid bodies with contact con- straints. In Robotics: Science and Systems, 2021.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic rendering, arXiv preprint arXiv:2310.0852. 2023.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20310-20320, 2024.
Jun Wu, Ole Sigmund, and Jeroen P Groen. Topology optimization of multi-scale structures: review. Structural and Multidisciplinary Optimization, 63: 63:1455-1480, 2021.
Tianyi Xie, Zeshun Zong, Yuxin Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. arXiv preprint arXiv:2311.12198, 2023.
Jie Xu, Tao Chen, Lara Zlokapa, Michael Foshey, Wojeiech Matusik, Shinjiro Sueda, and Pulkit Agrawal. An end-to-end differentiable framework for contact-aware robot design. In Robotics: Science Systems, 2021.
RJ Yang and CJ Chen. based topology optimization. Structural optimization, 12:98-105, 1996.
Taoran Yi, Jiemin Fang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text 3d gaussian splatting with point cloud priors. arXiv preprint S.ii::230008855 2023.
Xinyuan Yu, Siheng Zhao, Siyuan Luo, Gang Yang, and Lin Shao. Diffclothai: Differentiable cloth simulation with intersection- free frictional contact and differentiable two-way coupling with articulated rigid bodies. In 2023 JEEE/RS, International Conference on Intelligent Robots and Systems (IROS), pp. 400-407. IEEE, 2023.
Jonas Zehnder, Yue Li, Stelian Coros, and Bernhard Thomaszewski. Ntopo: Mesh-free topology optimization using implicit neural representations. Advances in Neural Information Processing Systems, 34:10:::400 14:10368-1033, 2021.
Hi-Hong Zhu, Wei-Hong Zhang, and Liang Xia. Topology optimization in aireraft and aerospace structures design. Archives of computational methods in engineering, 23:595-622, 2016.
A DETAILS OF ALGORITHMS
A.1 SOFT BODY DYNAMICS
Discretization We start from the discretization and interpolation of functions on the object space. For any function f(x) the object, records its value each particle ft. Then for any position x, the function value can be interpolated as
(f())) E(o:) -, h)
Pt
where W(r, h) is the kernel function used to smooth the interpolation. h is the interpolation range parameter and usually set to 1 to 2 times the average distance between particles. There many
14

---

Monaghan & Published as a conference paper at ICLR 2025
forms of the kernel function, while we use the version based spline functions Lattanzio (1985). with the following form
22 q)3
f
W(r,h) = Th
if c
(8)
otherwise
where q |r|/h. m, is the mass of particle i, and ps is the density near particle i, which can be computed from
= Σ m
where æ/e is the undeformed displacement from particle j i.
Geometry
When the soft body deforms, we can write the deformation map between the deformed position and undeformed position x. x + x: x u, while the Jacobian of the deformation map can be writen in J I Vu". Note that Vu can be decomposed into a scaling part and rotation part, while only the scaling part effective for mechanical computations. Thus, we need to extract the rotation matrix R, for each particle based on its initial neighborhood. Define
Apgi Σ m, (alg) h) ((x; - (.)( -²!)²)
(10)
Then R& can be computed from the polar decomposition of Apgis and
Vu;
Σ
m Uii
VW (x) 1t)
(11)
Pj
where
Ujz = R (a. xx. (3r)) x!))
(12)
Mechanics
Then we can use either non-linear Sainn---nnnt-nn strain tensor
(J"J
(13)
or the linear Cauchy-Green strain tensor
<= - (J+
(14)
calculate the stress energy of the object. Based on Saint Venant-Kirchhoff model, the stress energy can be written as
Σ mi Pi
X + ((r())))) μu +
(15)
where f and the Lame parameters of the material. Then the stress force on each particle can be computed from the gradients of stress energy with respect to the position of each particle.
A.2 TIME INTEGRATION
We will briefly explain the math details of the implicit time integration based on incremental poten- tial in this section. Implicit time integration requires us to solve the following two equations
Uk+1 Uk + M f(xx+1)At
(16)
Ft-1 24 + Ut+1
(17)
where æy and Øx+1 is the position time step i and k 1, vk and uk+1 is the velocity at time step k and &s + 1, f is the total force, At is the size of time step, and M is the mass matrix. Notice that complex form, meaning that there usually no analytical solution to this equation. To numerically solve it, we write f(x) EV((x) feitt splitting it into internal and external part. Then we have
usually
VE(x4+1) + At2
M (Ix+1 - (Ix + Atee + At M - ((x)))) =
(18)
15

---

Published as a conference paper at ICLR 2025
If we rewrite this into
(E(OL+i) + 2A
1
tr ((0k-1 yu) - yi))
(19)
where Vv: Atex + -- Notice that this can be converted into optimization problem, and then we can use Newton optimizer to solve 2k+11-
A.3 VOLUMETRIC REPRESENTATION
We will more information this section to explain the neural implicit network and quadratic surface representations in our pipeline.
Neural Implicit Network and output the SDF value this position by encoding the internal topology into the network. More concretely, we can write it
The neural network reads the position in the Euclidean space as input,
SDF(x) = fo
(20)
where is the parameter representing the internal topology.
Quadratic Surface Based on the conclusion from (Hafner et al. 2024), we can use quadratic function parametrize the internal topology of a rigid body if the problem only involves the rigid dynamic characteristics of the object. Therefore, the SDF can be written
SDF(#) xx Ax b²x +
(21)
Considering the symmetry, there will be only 10 degrees of freedom in 3D cases, which signifi- cantly simply the problem.
B DETAILS OF EXPERIMENTS
B. IMPLEMENTATION DETAILS
We implemented the core steps in our pipeline in C++ and Python. We parallelized the differentiable simulator with both CUDA on GPUs and OpenMP on CPUs, combining with Nvidia Warp. We used Open3D for point cloud processing and OpenCV for reference video processing, and PyTcch for the implementation of neural implicit surface. We evaluated our pipeline on the tasks below on a server with an AMD EPYC 9754 1 128-Core CPU, 12x DDR5 4800 16GB (384GB in total) RAM, and x NVIDIA RTX 4090 24G GPU. We used Open3D to perform point cloud processing, OpenCV for video processing, and L-BFGS-B and Adam for optimization.
B.2 HYPER- PARAMETERS
Neural Implicit Surface We implement this neural network with eight fully connected layers, each which applied with dropouts. All internal layers have size 512 512, and are connected with ReLU non-linearities. The network is trained with weight-normalizatie from the differentiable simulator. We train for 10000 epochs with learning rate 3e-6.
upon the gradients
Simulation
The basic simulation parameters are listed in Tab. while the actuation pressure of the soft hand is set to be 5e5 Pa.
C ADDITIONAL RESULTS OF EXPERIMENTS
C.1 FULL GALLERY OF SYNTHETIC DATASET
We visualize full gallery of our synthetic dataset in Fig. 8 In addition to the complex geometries, we also attach heterogeneous textures on the shapes to (I) match the case in real world and (2) alleviate the ambiguity for multi-view reconstruction.
16

---

Published as a conference paper at ICLR 2025
Table Simulation parameters. N/A means that the experiment does involve this parameter.
Experiment
Time step (s)
Density (kg/m²)
Young's modulus (Pa)
Poisson's ratio
Rigid
5e-3
1e3
N/A
N/A
Soft
5e-4
1e3
1.5e5 (softer): 3e7 (stiffer)
0,4
Rigid (real)
5e-3
5e3
N/A
N/A
Soft (real)
5e-4
5e3
1e7
0.49
Figure 8: Full gallery of our dataset. We create a diverse set of different geometries with complex textures to evaluate our method.
17

---

Published as a conference paper at ICLR 2025
Optimization 1)
POSR rvesel Fioc-d, 05)
pect tocael eine-.0
10"2
Gaussion Surfels tdepth-d;
10
Loss
10
10
10-6
frog
fish-1 swim-ring
toy-2 fandisk Optimization Loss (Part 2)
toy-1
hippo
toy-3
bird
PSSR tvewel Fibe-0.00
PGSi tvenel sczemb.2
10
Gauesion idepth-a)
10-3
Lo55
10
10
10-6
kangaroo
slone
pear
pig
spot
leapol
tumbler
fish-2
rabbit
Figure 9: Full data of the optimization loss of our method and the baselines, where "fail" on the top means that the method fails to output visually correct result.
C.2 FULL RESULT OF THE COMPARISON WITH BASELINES
We compare method against three strong baselines
proposed extensive dataset (Sec. 8.2
Fig. Jsshos the complete optimization loss of our method and the baselines, where we observe that all these methods give similar and sufficiently small loss except for the failure cases. Fig. 11 shows the volumetric density difference between the optimized result and ground truth, Fig. 12 shows the difference of the result some unseen test sets, and Fig. T3 shows the difference of center of mass. In these three figures, we all observe that our method achieve the most accurate result. Fig. [0 the complete volumetric generation time, where we can observe that our methods is faster than the baselines in all examples. Tab. 3 lists the complete reconstruction quality, where our method beats all baselines except the toy-1 example.
Note that in Fig. [4 and Fig. [5 we visualize the difference of optimized physical parameters, and our results more uniform and natural than all baselines. In Fig. [6 we show respectively the initial state, final state, and physical distribution of the rigid bodies, and we plot an additional vertical line to help with interpretation. With our optimization of the internal structure, originally unstable object stands steadily with reasonable alignment to the vertical line.
We visualize the simulation and optimization result in Fig. 14 Fig. 15 and Fig. 16
C.3 ADDITIONAL REAL-WORLD EXAMPLES
Hawk A balanced hawk (Fig. 17p is with meticulously designed hollowed structure inside so as to balance itself with support at beak only (Fig. 17 left). Similar the rigid experiments in synthetic examples, the loss is defined the distance between and the support center (the bird beak location).
Fig. 17 summarizes the results of our pipeline this task. We begin with initial guess of solid hawk stuffed with homogeneous materials, which easily falls off from its support location (Fig. 17 top middle). After optimization, our pipeline deduces hollowed design (Fig. riight) that successfully reproduces a convincing balanced hawk, verified by simulating its dynamic motion and observing negligible derivation tendency of falling off (Fig. 17 bottom middle).
18

---

Published as a conference paper at ICLR 2025
Volumetric Shape Generation Time
PGSR (vowel 2eem0.05)
PGSR (voxel 570-0.2)
1500
Gaussian (depth=0)
Ours
Time (6)
500
Fail
frog
fish-1 swim-ring
toy-2 fandisk Processing Time (Part 2)
hippo
toy-3
bird
PESR (voxe! s ze-0.22
1500
Gaussien Surfels (depth=8)
Time (s)
1000
500
kangaroo
pear
pig
spot
teapot
tumbler
fish-2
rabbit
Figure 10: Full data of the volumetric shape generation time of our method and the baselines, where "fail" on the top means that the method fails output visually correct result.
Average Densily Difference (Part 1)
PoSit sicemo NSSR twcnel stre- 21
0.6
Caussion Surfels idasth-d)
Ours
Difference
0.4
0.2
frog
fish-1
swim-ring
fandisk Average Density Difference (Part 2)
cow
toy-1
toy-2
hippo
toy-3
bird
POSR (voxel sizo-0 05)
0.6
pest tonael sine....)
Caussion Surfels tdoath-d)
Difference
0.2
0.0
kangaroo
pear
pig
spot
teapot
tumbler
fish-2
rabbit
Figure 11: Full data of the density difference our method and the baselines, where "fail" on the top means that the method fails output visually correct result.
19

---

Published as a conference paper at ICLR 2025
Test Loss (Part 1)
Pasl
NNsst (scsel kinem
PSSR (vorel stoo 21
Gauassan Surten idepthod;
Ours
Loss
0.4
0.2
frog
fish-1
swim-ring
toy-1 Test Loss (Part 2)
toy-2
fandisk
hippo
bird
PoSit (veaal steemo (b)
IoSR twnaet sine-d 2)
Caussion Surfes (dapth-d)
0.6
Loss
IILLINN tumbler
0.2
0.0
kangaroo
pear
pig
spot
teapot
fish-2 rabbit
Figure 12: Full data of the test set loss of our method and the baselines, where "fail" on the top means that the method fails to output visually correct result.
CoM Loss (Part
PoSit sicemo
NCSR twcnel stre- 21
2.5
Pail
Caussion Surfels idasth-d)
Ours
1.5
0.5
frog
fish-1
swim-ring
cow
toy-1 CoM Loss
toy-2 (Part 2)
fandisk
hippo
toy-3
bird
POSR (voxel stzo-o, 05)
pest tonael sine...2)
2.5
Caussion Surfels tdoath-d)
2.0
L053
1.5
1.0
0.5
0.0
kangaroo
pear
pig
spot
teapot
tumbler
fish-2
rabbit
Figure 13: Full data of the center of mass (CoM) loss of our method and the baselines, where "fail" on the top means that the method fails output visually correct result.
20

---

Published as a conference paper at ICLR 2025
frog
fish-1
swim ring
Cow
toy-1
toy-2
fandisk
hippo
toy-3
bird
Ours
PGSR (voxel_eize-0.e0 06)
PGSR (voxel_sizee siie__izze0
Gaussian Surfels
Ground Truth
Figure 14: Full result (part 1) of the comparison of slices between our methods and baselines
(Sec. 8.1
21

---

Published as a conference paper at ICLR 2025
kangaroo
stone
pear
pig
spot
fish-2 tumbler teapot
rabbit
Ours
PGSR sizee__szzee000
PGSR (voxel_sizen0 sizem0.2)
Gaussian Surfels
Ground Truth
Figure 15: Full result (part 2) of the comparison of slices between our methods and baselines
(Sec. 8.1)
22

---

Published as a conference paper at ICLR 2025
swim-ring
frog
rabbit
toy-4
fish-1
Cow
toy-1
toy-2
fandisk
hippo
toy-3
bird
kangaroo
stone
pig
pear
teapot
spot
tumbler
fish-2
Figure 16: Full result (part 2) of the simulation of initial and final poses of the objects (Sec. 8.1
23

---

Published as a conference paper at ICLR 2025
Table 3: Reconstruction quality. Lower numbers indicate better performance. The optimal results bolded.
Dataset id
frog
fish-1
swim-ring
Cow
toy-
toy-2
fandisk
Ours
0.306
0.358
0.242
0.300
0.294
0.274
0.335
PGSR (voxel size=0.05)
0.745
0.998
0,464
0.410
0.263
0.761
0.904
PGSR (voxel size=0.2)
0.781
0.907
0,894
0.801
0.595
0.845
0.878
Gaussian Surfels
0.851
Fail
0.882
Fail
0.552
0.939
0.865
Datset id
hippo
toy-3
bird
kangaroo
stone
pear
pig
Ours
0.663
0.285
0.268
0.388
0.290
0.245
0.343
PGSR (voxel size=0.05)
0.910
0.478
0,846
0.841
0,907
1.078
0.891
PGSR (voxel size=0.2)
0.869
0.720
0.850
0,480
0,904
1.027
0.867
Gaussian Surfels
0.939
0.459
0.906
0.976
0.890
1.148
0.939
Dataset
spot
teapot
tumbler
fish-2
rabbit
Ours
0.347
0.264
0.305
0.352
0.298
PGSR (voxel size=0.05)
0.822
0.822
0.953
0.605
0.930
PGSR (voxel size=0.2)
0.901
0.841
0.755
0.889
0.903
Gaussian Surfels
0.724
0.850
0.963
0.472
0.940
Left: real-world photo of the balanced hawk resting on bottle at its beak; Top middle: the falling-off motion from simulating the hawk point cloud with homogeneously stuffed materials; Bottom middle: the balanced motion from simulating the same hawk point cloud with optimized hollowed structure; Right: the optimized structure with black and gray particles indicating the solid and empty region, respectively. Note that the boundary particles are fixed as solid during optimization and removed from this figure for better visualization of the interior structure.
Figure 17: Hawk (Sec. 8.3
24

---

Published as a conference paper at ICLR 2025
Figure 18: Qigi (Sec. 8.3). Left: a photo of qiqi showing a tilting angle. The blue arrow indicates the up direction of qiqi. Middle: the pose changes of qiqi when increasing its capacity from empty to maximal; Middle: simulated pose changes of qiqi with the same exterior from Gaussian splatting and interior structure optimized from our pipeline; Bottom: simulated motion of a qiqi with an initial guess of uniformed hollowed interior. Right: the optimized interior structure by our pipeline.
Qiqi (Tilting Vessel) Qiqi is tilting vessel originating from ancient China (Fig. 18 left). It has a carefully crafted interior shape that exhibits represeniative poses at three different states: it tilts by a specific angle when empty; the tilting angle gradually decreases to zero when the vessel becomes half full; the vessel overturns (upside down) when reaching its maximal capacity. Fig. t8p middle demonstrates this behavior of qiqi with representative frames from a video recording of filling water into qiqi. The blue arrow in the figure indicates its up direction. Note how the up direction is eventually upside down when qiqi reaches its capacity. Our goal is to infer an interior geometry of qiqi that reproduces all three representative poses.
We assign one objective for each representative pose of qiqi. The objective consists of two terms characterizing the preferred height and tilting angle from the center of mass. Note that for given shape parameter 8, the three poses compute three centers of mass based on considering the union of the shell characterized by s and its stuffed interior at three specified height. This initial guess (a shell with uniform thickness) leads to design resilient to changing tilting angles (Fig. [8 bottom middle), which shows that an interior geometry that can exhibit desired behaviors not designed trivially.
Fig. 18 right reports our optimized interior shape of qiqi. The result clearly shows that an asymmetric design is necessary and expected due to the initial nonzero tilting angle when the vessel is empty. We simulate our inferred design with an increasing capacity and observe similar pose changes the video recording (Fig. 18 middle), confirming the efficacy of our optimization pipeline.
C.4 ABLATION STUDIES
The density of points In order to investigate the impact of the density of points to our method, thoroughly sample five different initialization: 12226, 20480, 29160, 40000, and 69120, and we show the result this ablation study in Fig. 19 spite of minor variances on the top of the wobble doll, the general quality of distribution decreases marginally, indicating the efficiency and the robustness of our method in terms of the number of initial points.
The material property and object size To study how the choice the size and the material of 3D object impacts the result of optimization, we design ablation study by doubling the initial size and the material density of the hawk respectively. We show the results in Fig. 20 where the left figure is the original result, the middle the doubled size, and the right the doubled material density. As shown in the figure, the optimization results look very similar to each other, indicating that our method is not sensitive to the choice of material and the overall scale.
25

---

Published as a conference paper at ICLR 2025
18888
Figure 19: The density of points. Black points represent solid part and grey points represent hollow part. From left to right, the density of points are 12226, 20480, 29160, 40000, 69120 respectively. We can observe that the quality of the result does not decrease much when the density of points decreases.
Figure 20: Ablation study on the material and size. Black points represent solid part and grey points represent hollow part. The first graph is the original setting. The second graph is doubling the seale of the hawk. The third graph is doubling the material density of the hawk. We can observe that with the same relative shape and relative physical parameters, the output does not change with its absolute value.
26

---

Published as a conference paper at ICLR 2025
Figure 21: Rigid-bodi simulation with collision. The top left part demonstrates the appearance the ball. The four pictures in top right part is four frames from the input video (ground truth). The four pictures in middle right part is the corresponding frames of the final solution, and the four pictures in bottom right part is that of the initial guess. The bottom left part is the internal structure, where black points represent solid part and grey points represents hollow part.
The type of simulation show the extensibility of our method to more types of physical phe- nomena, we combine the rigid-body physical simulation with the proposed collision handling and apply our method on ground-truth trajectory shown the top output motion trajectory matches the visual observation. We want to highlight that in the original video the rigid ball was rotating, which is quite hard for traditional methods to capture. Our method, however, reconstructs the behavior nicely as indicated by the textural rotation.
shown in Fig. 2T, we drop the rigid ball on the floor and record the
After the optimization of interior structure, the
The lighting condition and camera pose In addition to the real-world wobbly doll, we create simulated wobbly doll in our synthetic environment as well for better reproducibility. We vary the lighting condition and the camera pose in order test the robustness of our method as shown in Fig. [2 We did three experiments: (left) the normal light with front pose, (mid) the dark light with front pose, and (right) the normal light with side pose. We also attach the corresponding internal solution on the side. As indicated by the solutions, our method is robust in terms of varying lighting conditions and camera poses. We also argue that changing camera pose is equivalent to transforming the local frame of the dynamics, and our method is insensitive to both the changes.
27

---

Published as a conference paper at ICLR 2025
Figure 22: Synthetic wobbly doll. The experiment with the same setting of the real world wobbly doll but performed on the synthetic one in our dataset. Besides, we change its lighting and view point. Black points represent solid part and grey points represent hollow part. The first graph is bright lighting with front view, and the second graph is its solution. The third graph is dark lighting with front view, and the fourth graph is its solution. The fifth graph is bright lighting with side view, and the sixth graph is its solution. We can observe that the output of our pipeline does not change with the lighting and camera position.
Figure 23: Experiments which supports continuously varying material. The two experiments share similar setting with the previous wobble doll and horse experiment, while the main difference is we allow a continuously varying material. The first and third figures are two typical input images, while second and fourth figures are the optimized result. The color indicates the density of each point, where red means higher density (solid) and blue means lower density (hollow).
Figure 24: Experiments using multi-video for optimization. The two experiments share similar setting with the previous wobble doll and horse experiment, while we leverage multi-video opti- mization, with videos of difference view points, light conditions, vibration amplitudes, etc. From left to right, the first figure represents the rendered object, the second represents the ground truth, third represents the single-video result, the fourth represents the multi-video result.
28

---

Published as a conference paper at ICLR 2025
PSNR (Part
40
PSWR
20
fish-1
swim-ring
1
2
fandissk
hippo
toy-3
bird
PSNR
(Part 2)
40
35
PSNR
20
kangaroo
pear
pig
teapot
tumbler
fish-2
rabbit
Figure 25: Full data of the PGSR of rendered result with respect to the ground truth, where "fail" on the top means that the method fails to output visually correct result.
29

---

Published as a conference paper at ICLR 2025
kangaroo
frog
fish-1
stone
rabbit
pear
COW
pig
toy-1
spot
teapot
toy-2
fandisk
tumbler
hippo
fish-2
toy-3
rabbit
bird
Figure 26: One example of the comparison in the unseen set. The left figure represents the ground truth and the right figure represents our simulated result.
30


Reviews:
[{"review_id": "jgZNA2E7JR", "reviewer": "Reviewer_vXS1", "summary": "The authors present a holistic approach to estimating the internal topology of objects from images. The work relies on the use of a particle-based differentiable simulator to estimate probable internal topology directly from the seen motion of an object. By this approach, the author is able to generate even real-world 3D printed versions with reasonable structure directly from a small video sequence.", "questions": "- The authors claim one of the applications is in 3D printing; since I lack any knowledge of 3D printing, I am unsure about its weaknesses. But to enhance my understanding, why do we require the internal topology to be known for this? Shouldn't having the surface not be enough?\n- Please fix the typo in line 175 \"point clout\"\n- Please keep writing style consistent for example, line 234, \"point-cloud\"\n- Would it be possible again to summarize for me what actually is the main goal and main application of this work?\n- Not so much a question but rather a comment regarding Abstract Style (improvement/suggestion): Having this kind of structure usually improves understandability and readability: 1. What is the problem and why is it important? 2. What are the limitations of existing solutions? 3. What are the advantages of the proposed approach? 4. How does it work? 5. Summary of results", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The author works on a novel approach to solving a relatively novel problem for modern 3D Computer Vision. While the idea of reconstruction of internal topologies is not novel to the best of my knowledge, I have not seen much work trying to solve this for 3D Gaussian Splatting or NeRF-based approaches.\nThe work itself is decently written and shows a great evaluation to validate the quality of their methodology.\nThe authors can propose a holistic pipeline that should make their work easily usable for users, with the authors giving a significant amount of design possibilities.\nIn general, I am more than in favor of this work's results and core concept being interesting.", "weaknesses": "A large issue in this work is the convoluted writing. While still quite understandable, this work packs a significant amount of results, ideas, and concepts from many different fields.\nAs such, while many parts of the simulation (core contribution) have been well explained, much information regarding the volumetric representation is missing. As such, to improve the work (and make it complete), I would suggest the author add more information in the appendix.\nAnother larger issue is the motivation. While in Computer Graphics/Computer Vision, the challenge of estimating internal topologies is quite interesting, the estimation part might be a larger issue for practical, real-world use cases. In many real-world applications that rely on internal topologies, it is quite important that exact information is given, as this cannot assured by your model. I am still quite unsure when this work will be usable in real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nZncYJH0ZL", "reviewer": "Reviewer_q35E", "summary": "The paper presents TopoGaussian, a pipeline for inferring the internal structure of opaque objects using only photos and videos as input. The pipeline works by first using Gaussian splatting on multi-view images to obtain a point cloud, then optimizing the internal topology structure through a differentiable physics simulator to match observed motion patterns.\n\nThe key contributions include a particle-based, mesh-free pipeline that combines Gaussian splatting with a differentiable physics simulator; three flexible topology representation options: particle-based, neural implicit surface, and quadratic surface; a particle-based differentiable simulator supporting both rigid and soft objects with different topology structures.", "questions": "Please see my questions in the previous weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Novel combination of Gaussian splatting with physics-based optimization for internal structure inference.\n2. Uses a mesh-free approach that avoids common issues with mesh processing; presents particle-based differentiable simulations that are compatible with three flexible topology representations, including particle, neural implicit surface, and quadratic surface.\n3. Well-structured presentation with a clear pipeline overview.", "weaknesses": "The reviewer appreciates the authors' effort in building a particle-based pipeline to find a physically plausible internal topology structure. As the authors have also mentioned in the paper, this task is relatively new, and there are fewer baselines to compare with (at least other baselines do not use point-based representation). I have several concerns about the measuring metrics and their validation to support the claims from the authors:\n\n1. Optimization Loss: This measures the difference between simulated motion and reference motion, and it directly indicates whether the internal topology structure is physically plausible. However, in Figure 3, the current method does not achieve the lowest loss among baselines in multiple test samples.\n2. Comparison Implementation: When exporting the mesh from other baselines and chaining it into the rest of the pipeline in this paper, how can the mesh-based representation be made compatible with the rest of the system that is particle-based?\n3. Time / Smoothness: It is unclear whether this improvement comes from the GS representation itself or from the authors' method. The reviewer encourages the authors to elaborate more on this or provide ablation studies to explain that the improvement comes from the proposed method itself.\n4. Inner Structure: Can the authors provide the reference ground truth for the inner structures when making comparisons with other baselines? The reviewer understands that, in practice, the inner structures are hard to acquire, but in synthetic data, it is practical to obtain the ground truth of inner structures.\n\nOther question:\n\nHow is the decision variable applied to the point cloud representation to obtain a continuous indicator function from the point cloud? (Line. 193)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DAHXEXPlv9", "reviewer": "Reviewer_2FAv", "summary": "The paper introduces TopoGaussian, a particle-based approach that uses visual clues from photos and videos to infer internal topology structures of opaque objects. Key contributions include:\n1. Mesh-Free Topology Inference: TopoGaussian combines Gaussian splatting and a particle-based differentiable simulator to infer interior topology without requiring mesh-based representations, which traditionally entail extensive fixing and filling processes.\n2. Flexible Topology Representations: The pipeline supports different topology representations (particle, neural implicit surface, and quadratic surface), allowing for optimization and simulation within a unified framework.\n3. Experimental Validation: TopoGaussian is evaluated on synthetic and real-world tasks, showcasing its capability to generate 3D-printable reconstructions that exhibit high fidelity and reduced processing time compared to existing mesh-based methods.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Efficient and High-Quality Reconstruction: The particle-based approach of TopoGaussian achieves efficient reconstructions, with the authors reporting a significant speedup (5.26x faster) and superior boundary reconstruction quality (2.33x improvement) compared to mesh-based methods like PGSR and Gaussian Surfels.\n2. Annotation-Free and Flexible: The pipeline’s independence from intrusive sensors or annotation requirements makes it practical and applicable in fields like robotics and manufacturing. Additionally, the three topology representation options offer flexibility based on application needs, from rigid to soft-body simulation.\n3. Simplicity and Smoothness of Output: By eliminating the need for mesh processing, TopoGaussian produces smoother outputs conducive to 3D printing and manufacturing.", "weaknesses": "1. Evaluation Limitation: The method optimizes the interior structure based on a single motion, which may lead to overfitting and an inaccurate reconstruction of the true internal structure. Since ground truth data for internal topology is unavailable, it is difficult to verify if the inferred structure is correct or merely adapted to the given motion. Although the authors acknowledge this limitation and propose alternative metrics, these may not fully reflect the true structure. To strengthen the evaluation, the authors could consider obtaining ground truth through simulation (maybe with physics simulation) or testing the inferred structure on new motion videos as a test set. If the predicted structure is accurate, it should exhibit consistent behavior across these unseen motions, providing stronger validation.\n2. Potential Overfitting to Single Motion: The current approach optimizes the internal structure based on a single motion video, which may lead to overfitting and limit the model’s generalization capability. To improve the robustness and accuracy of the inferred structure, the authors could consider optimizing based on multiple motion videos. By incorporating a variety of motions, the resulting model may better capture the true internal structure and provide more reliable and generalizable results.\n3. Limitation to Simple Material Compositions: The current framework supports only single-object, dual-material compositions, which may limit applications involving complex, heterogeneous materials or multi-object interactions. Future work could focus on extending support to more intricate material compositions.\n\n---\n\nThe authors have provided additional results to address my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q6LYGVKULK", "reviewer": "Reviewer_HpwZ", "summary": "This paper proposed a particle-based pipeline for motion reconstruction with physically corrected internal topology based on visual inputs. Specifically, they used a vision-based reconstruction method to construct the point cloud and optimize the physical properties of each particle through a differentiable simulation. Through several rigid-body and soft-body motion tasks, they verified the validity of the approach.", "questions": "● Is there a unique solution for recovering the internal structure of an object based solely on visual observation? How can we determine if the recovered internal structure is reasonable?\n\n● The article mentions that the volumetric shape generation stage is faster than mesh-based methods. Is this stage entirely consistent with the PhyGaussian?\n\n● What is the relationship between the metrics used in the article—reconstruction quality and optimization loss? How is reconstruction quality defined? Does the \"Smoothness\" mentioned in Section 8.1 Metrics represent reconstruction quality? Why does the proposed method achieve similar optimization loss to the baseline, yet show significantly better reconstruction quality than the baseline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "● Extracting an object's internal topology through a visual solution sounds interesting and is a novel task. The feasibility of the solution is also demonstrated.\n\n● The article constructs multiple synthetic data to validate the effectiveness of its scheme, and more importantly, it accomplishes multiple sets of validation experiments in real-world scenarios.", "weaknesses": "● The main goal of the article is to build the internal structure of the object by visual motion, but there is no proper metric or GT comparison to measure the correctness of the internal structure recovery, it only compares the rendering results\n\n● The comparative experiments are insufficient; the paper only compares optimization loss, reconstruction quality, and time with two mesh-based approaches. This fails to highlight the main contribution of the paper, which is the recovery of internal structures. Additionally, the visual comparison of internal structures does not fully reflect the accuracy of the internal structure recovery.\n\n● The proposed method involves multiple steps, such as Gaussian Splatting, Volumetric Shape Generation, and Topology Optimization, etc., but lacks thorough validation for these steps. For example, there is no detailed analysis of the impact of filling quality on topology optimization or the robustness of the entire system to the visual input quality and movement amplitude.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)