PhysX-3D: Physical-Grounded 3D Asset Generation
Authors: Ziang Cao
Zhaoxi Chen
Liang Pan
Ziwei Liu
Venue: NeurIPS 2025 spotlight ()
Keywords: 3D generation; Physical 3D assets
Primary area: deep_learning

Abstract: 3D modeling is moving from virtual to physical. Existing 3D generation primarily emphasizes geometries and textures while neglecting physical-grounded modeling. Consequently, despite the rapid development of 3D generative models, the synthesized 3D assets often overlook rich and important physical properties, hampering their real-world application in physical domains like simulation and embodied AI. As an initial attempt to address this challenge, we propose \textbf{PhysX}, an end-to-end paradigm for physical-grounded 3D asset generation.
\textbf{1)} To bridge the critical gap in physics-annotated 3D datasets, we present \textbf{\ourname}\ - the first physics-grounded 3D dataset systematically annotated across five foundational dimensions:
\textbf{\textcolor{color2}{absolute scale}}, \textbf{\textcolor{color3}{material}}, \textbf{\textcolor{color1}{affordance}}, \textbf{\textcolor{color4}{kinematics}}, and \textbf{\textcolor{color5}{function description}}. In particular, we devise a scalable human-in-the-loop annotation pipeline based on vision-language models, which enables efficient creation of physics-first assets from raw 3D assets. \textbf{2)} Furthermore, we propose \textbf{PhysXGen}, a feed-forward framework for physics-grounded 3D asset generation, injecting physical knowledge into the pre-trained 3D structural space.
Specifically, PhysXGen employs a dual-branch architecture to explicitly model the latent correlations between 3D structures and physical properties, thereby producing 3D assets with plausible physical predictions while preserving the native geometry quality. Extensive experiments validate the superior performance and promising generalization capability of our framework. All the code, data, and models will be released to facilitate future research in generative physical AI.

PhysX-3D: Physical-Grounded 3D Asset Generation
Ziang Cao
Zhaoxi Chen'
Liang Pan2
Ziwei Liu"
Nanyang Technological University Shanghai Al Lab
https: /pyy/x 3d.github.io/
Physical properties
Absolluo
Affordance
Material Density
Kinematics
Function
Frist Second
Child Parent
Description
(100 te R3.4)
thtt Supports the
Chair, back
Fanted 4.7022 user's left arm.
Physical
dimension:
120+70x70
Officer Chair
Keyboard Touchpod
Glasss 28 apmes
Rang ELIEAG)
Find that dispays
Physical
dimension:
Leather Swod ChiCEE
Biraci3
nd part that holds and stores
Laptop
Physical
Bog. body
items
Handbag
dimension:
30<10x25
Figure 1: Visualizations of our PhysXNet for phsycial 3D generation. 3D assets in our dataset have fine-grained physical property annotations, including 1) absolute scale, 2) material, 3) affordance, 4) kinematics, and 5) function descriptions (basic, functional, and kinematical descriptions).
Abstract
3D modeling is moving from virtual to physical. Existing 3D generation pri- marily emphasizes geometries and textures while neglecting physical-grounded modeling. Consequently, despite the rapid development of 3D generative models, the synthesized 3D assets often overlook rich and important physical properties, hampering their real-world application in physical domains like simulation and embodied AI. As initial attempt address this challenge, an end-to-end paradigm for -irscall-rroudde 31) asset generation. 1) To bridge the critical gap in saaysiics-anntteed 3D datasets, we present PhysXNet the first grrsiddgg 3D dataset systematically annotated across five foundational dimensions: absolute scale, material, affordance, kinematies, and fuunction description. In particular, we devise scalable human-in- -itn-----the-ooppp annotation pipeline based on llniun-la models, which enables efficient creation of physics-first assets from raw 3D assets. 2) Furthermore, we propose PhysXGen,
propose PhysX,
"Corresponding author, ziwei. liuEntu.edi
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

a feed-forward framework for grrounde image-to-3D asset generation, injecting physical knowledge into the pre-trained 3D structural space. Specifically, PhysXGen employs dual-branch architecture to explicitly model the latent cor- relations between 3D structures and physical properties, thereby producing 3D assets with plausible physical predictions while preserving the native geometry quality. Extensive experiments validate the superior performance and promising generalization capability of our framework. All the code, data, and models will be released to facilitate future research in generative physical AI.
Introduction
The creation of diverse and high-qualityy 3D assets has gained significant prominence in recent years, driven by their expanding applications across gaming, robotics, and embodied simulators. Substantial research efforts have been focused appearance and geometry only, from high-quality 3D datasets | 2 3 4.
efficient 3D representations, to generative modeling. However, most of them predominantly emphasize structural characteristics while overlooking physical properties inherent to real-world objects. Given the rising demand for physical modeling, understanding, and reasoning in 3D space, we argue that a comprehensive suite for physics-grounded 3D objects is important, from upstream data annotations pipeline to downstream generative modeling.
Beyond purely structural attributes like geometry and appearance, real-world objects intrinsically possess rich physical and semantic characteristics comprising: 1) absolute scale, 2) material, 3) af- fordance, 4) kinematics, and 5) function descriptions. By integrating these fundamental properties with classical physical principles, we can derive critical dynamic metrics, including gravitational effects, frictional forces, contact region, motion trajectories, and interaction. However, existing datasets/annotation pipelines only offer partial solutions towards physically grounded knowledge in 3D objects that cover the entire spectrum. Recent efforts support articulated object applications have yielded datasets like PartNet-Mobility [5], which provides 2.7K human-annotated articulated 3D models. Yet, this collection still lacks essential physical descriptors including dimensional specifications, material composition, and functional affordances that are crucial for physically accurate simulations and roboties applications.
To bridge this representational gap, we propose PhysXNet the first comprehensive physical 3D dataset containing over 26K richly annotated 3D objects, illustrated in Figure 1 Except for the object-level annotation, i.e., 1), we annotate 2) and 5) for each part. Besides, for 3), we provide the affordance rank for all parts, while we annotate the 4) detailed parameters of kinematic constraints, including motion range, motion direction, child parts, and parent parts. Besides, we introduce an extended version, PhysXNet-XL, featuring over 6 million procedurally generated and annotated 3D objects.
Most importantly, PhysXNet is built with efficient, robust, and scalable labeling pipeline. We introduce a numan-in-the-loon annotation pipeline to annotate the properties for the existing object- level 3D dataset, i.e., PartNet (6). The pipeline proceeds in three stages: 1) target visual isolation, in which we render each component via alpha compositing to get the best visual prompts with minimized visual interference. 2) automatic VLM labeling, where large vision-language model (VLM) to annotate most of the properties; and 3) expert refinement, combining systematic spot-checks with focused human annotation of complex kinematic behaviors. To the best of our knowledge, PhysXNet is the first 3D dataset with abundant properties for each part.
To bridge the modeling gap of thysical-groundee 3D assets, we further introduce PhysXGen, feedforward model for physical 3D generation. Given the fact that physical properties are spatially related to geometry and appearance, repurpose pretrained 3D generative priors to generate physi- cal 3D assets, enabling efficient training with reasonable generalizability, Specifically, PhysXGen leverages a dual-branch architecture jointly model the latent correlations between 3D geometric structures and physical properties, which is naturally compatible with existing 3D native generative priors. Moreover, this formulation makes the best use of pretrained latent space, leading to plausible physical predictions while keeping the decent geometry quality from the pretrained model. Compre- hensive experiments prove the promising performance of PhysXGen. We hope our work reveals new observations, challenges, and potential directions for future research in embodied Al and robotics.
To summarize, our main contributions are:
2

---

Table 1: Comparison of related datasets which can support research in physical 3D generation. While the ABO dataset 7 contains material metadata and keywords, its object-level annotation granularity constrains part-aware applications like robotic manipulation or physical simulation. In contrast, PhysXNet provides part-level annotations.
Dotaset
# Objs
Part anno
Physical Dim
Material
Affordance
Kinematic
Description
Year
ShapeNet 0
5IK
X
2015
PartNer [
26K
X
X
2019
Mutbiit-
GAPartNet 8
1.1K
V
K
x
K
2022
X
Obj-level
Obj-level
D
6K
X
X
x
X
2023
Objaverse
818K
X
K
x
X
2023
PhysXNel (ours)
26K
V
Part level Part-level
Part level
2025
PhysXNet-XT. (ours)
We pioneer the first end-to-end paradigm for physical-grounded 3D asset generation, ad- vancing the research frontier in physical-groundee content creation and unlocking new possibilities for downstream applications in simulation.
We build the first physical-grounded 3D dataset, PhysXNet, and propose human-in- loop annotation pipeline to convert existing geometry-focuser datasets into fine-grained physics-annotated 3D datasets efficiently and robustly. In addition, we present extended version, PhysXNet-XL, which includes over 6 million annotated 3D objects generated through procedural methods.
We design a dual-branch feed-forward framework, PhysXGen. It can model the latent interdependencies between structural and physical features to achieve plausible physical predictions while maintaining the native geometry quality.
2 Related Work
2.1 3D Datasets and Benchmarks
Due the time-consuming and expensive in realistic data collection, current large-scale 3D datasets prefer to collect data online 1
According to the type of 3D) data, existing 3D datasets can be divided into synthetic and real-world datasets. To facilitate the development of 3D vision,
2 3
ShapeNet [1] collects 51,300 CAD models. Building upon it, the PartNet dataset (6) introduces an annotation framework that provides part annotations at significantly finer granularity levels. Furthermore, PartNet-Mobility (5] annotates the kinematic constraints and provides 2.7K articulated 3D objects for 3D vision, especially for embodied Al and robotics. ABO 7 is a high-quality datasets with around 7.9K CAD models with fine fine-grained geometric and textures. Compared with prior work, includes the physical dimension, material, and keywords. However, the material information and descriptions focus on object-level, limiting the part-aware applications. Recently, Objaverse [2] has alleviated the scarcity of 3D data. collects and filters over 800K 3D data. To bridge the gap between synthetic and real data, Omniobject3D provides over 6k high-quality 3D scans. A detailed comparison shown Table []
Despite significant advances in 3D data acquisition, prevailing 3D datasets primarily emphasize geometry and appearance fidelity or narrowly defined physical attributes, creating critical bottleneck for developing physics- aware 3D vision models and their real-world applications. To bridge this foundational gap, we present PhysXNet 3D dataset with comprehensive physical properties encompassing physical dimension, part-level material, affordance rank. kinematic parameters, and part-level description. Furthermore, extend our dataset with PhysXNet-XL, comprising more than 6 million annotated 3D objects created via procedural generation.
2.2 3D Generative Models
As one of the most representative optimization- method in 3D generation, DreamFusion 9 proposed the SDS loss function. By utilizing the prior knowledge of the 2D diffusion model, it achieves impressive generative performance. Despite various works, optimization methods still suffer from the multi-face Janus problem and low optimization efficiency. Recently, benefiting from

---

Definition of physical and semantic properties
Identification
Function
Operation
Kinematic types
pricrity
Highent
sonstiarss
In ihed with a lonce of
Pryaical
Affordance
Prisratt jein
space
Parent
Ravolutt
Name Catagory&
Mesh
plant
Child parts
ringg join
Uniimited revvmment
Absolltte
Physkal
Function Description
Human-in-the-lne annotation pipeline
Visual Input
VLM Output and Human check &
Kinematic parameters determination
Rome: Wull Calenct.
oyen
Citegory: sumttrr,
Oirension 200m40018C, prinity,
Rlsstion rurge: (-5,43)
Annotations
Clild grapt purt Parent grous part
reighbes: i.
4 Rosution (-,00]
test, of mesement grou movemntt tape 'C",
contact region
paronL/sbal
Locaion: 10.2...1 200) care (4.25009)) Pareet (2.4] Determine
[-_]
Baik docrtian: "Tee man actural hume of sabens: waxt
componeers"
desepption:
(2.b) (2.4 Candidate generation
Mousewrr, desseetion "ibed and mpsi rowe
and selection
panamaters
Figure Top: Definition properties PhysXNet. By defining and annotating properties across three categories, common physical quantities can be systematically calculated to enable physical simulations. Bottom: Overview of our human-in-the-loop annotation pipeline. We utilize GPT-4o to gather foundational raw data, which is subsequently verified through human oversight. The kinematic parameters are then rigorously determined and finalized through human review.
its impressive efficiency and robustness, feed-foward models [10 10 gained more and more attention. However, those methods still focus on geometry and appearance quality, neglecting the physical properties of 3D assets.
13
16 17 have
2.3 Articulated and Physical 3D Object Modeling
Articulated object modeling mainly consists of tasks like perception, reconstruction, and generation. Some works try to estimate articulation pose [18] and identify articulation parts [19]. while others (20] focus learn joint parameters from images. In the reconstruction field, existing works try to reconstruct articulated models from RGB 21], RGBD [22], and point cloud [23] Recently, some
methods have tried generate articulated 3D assets by utilizing vision-language model [4] 25 or adopting an optimization- with real applications, many works aim incorporate the physical properties into 3D modeling,
framework (26). To bridge the critical gap between existing methods
Some works try to learn material parameters from videos [27] or images (28), while other methods
aim introduce physical guidance via simulation
29
30
physical principles (31).
In contrast to fragmented paradigms in physical 3D modeling, this work introduces PhysXGen unified physics-integrated generative framework capable of learning cross-property consistency to generate 3D assets with all necessary physical properties. By exploiting the relationship be- tween physical and structural features, our method achieves promising performance in physical 3D generation.
3 PhysXNet Dataset
In this section, we will introduce physical properties and the human-in-the-loop annotation pipeline. Besides, we will report the statistics and distribution of PhysXNetand PhysXNet-XL.
3.1 Definition of Physical Properties
As shown in Figure we systematically categorize object properties into three progressive stages: a) Identification determining the basic nature of the object; b) Function understanding its potential applications; and c) Operation detailed usage methodologies. To streamline the annotation process, posit that the internal composition of component is homogencous, exhibiting uniform property

---

invariance throughout its structure. For stages a), we set absolute scaling and material (material name, Young's modulus, Poisson's ratio, and density). Besides, for b). we establish functional affordance analysis and function descriptions (basic, functional, and kinematic descriptions). Finally, we use kinematic parameter quantification to represent c). Speeifically, we grade the priority of being touched on all available parts to obtaín the affordance score for all parts from 1 to 10. We set five possible kinematic types: A. No movement constraints (like water in a bottle), B. Prismatic joints (like drawer), C. Revolute joints (like a laptop), D. Hinge joint (like hose in shower system), or E. Rigid joint and a combined kinematic type: CB. Revolute and Prismatic joints (like a lid of a bottle). Except for A and E. we will annotate the parent, child parts, and detailed kinematic parameters (such as rotation direction, rotation range, and so on). Note that, due to the challenges in precisely quantifying the absolute physical movement range of B, within the 3D coordinate system. Besides, to avoid the unnecessary and meaningless annotation of over-fine-gee graineegggddd parts in PartNet, pre-defined threshold with their neighboring parts. We manually refine the results of the merging process to ensure that the merged outputs reasonable and consistent.
the movement range
merge the tiny parts whose vertices and are smaller than
3.2 Humen-in-the-loop Annotation Pipeline
Following the establishment of target annotation specifications, we implement a systematic and streamlined semi-automated annotation framework, structured into two distinct operational phases (see Figure 2 1) Preliminary Data Acquisition and 2) Kinematic Parameter Determination. Specifi- cally, we utilize GPT-4o to obtain the basic information. Besides, to ensure the quality of raw data, a human candidate will check and refine the output of the vision-language model (VLM).
For the second phrase, we split into four subtasks: (2.a) calculate contact region, (2.b) plane fitting, (2.c) candidate generation and selection, and (2.d) kinematic parameters. Note that (2.c) and (2.d) accomplished by human candidate. For all constraint movable parts (kinematic type is not A or E), we will calculate the contact region with the neighboring parts. We first extract point cloud data from the child-parent mesh pair, formally designated Pa and P,,. respectively. The workflow subsequently calculates Fuclidean distance between points in P. and Pp. followed by spatial filtration that climinates point pairs failing to meet a predetermined distance threshold. Subsequently, we employ plane-fitting algorithm. We sample several axes uniformly on the fitted plane as candidates. Note that for kinematic type C, we additionally need determine the location of the rotation axis. Therefore, we will perform a k-means algorithm in the contact region for type C to generate several candidates. After selecting the candidate location, we can finalize the kinematic parameters.
3.3 Statistics and Distribution of PhysXNet
Comprises over 26K physical 3D objects, the part number of objects in PhysXNet exhibits long- tailed distribution illustrated in Figureß] where each object contains an average of around 5 constituent parts. Besides, we document the PhysXNet encompasses objects spanning from relatively small scale indoor entities to large-scale outdoor structures, the physical dimension exhibits significant variation among objects. For kinematic types and material in PhysXNet, we show detailed proportional composition. Note the density in our Netadheess to the metric standardization framework, i.e., /cme². Furthermore, Figure 3 (d) shows the frequency of the popular object tags, including the name and category. Finally, we also report the component category in our procedurally generated 3D objects, including intra-category combination: cabinet, bottle, faucet, chair, oven, shower, knife, table, and laptop; b) cross- cross-category combination: drawer and door. More details about PhysXNet-XL are released in the appendix.
sell--withhhhhhhe distributions of objects in (b). Given that
4 PhysXGen Framework
As mentioned above, physical 3D generation is still a challenging and promising task. Most prior works only focus on a single or specific physical property. In this section, we aim to build a unified generative framework to generate physical 3D assets directly. While our PhysXNet dataset contains 26K assets, this scale remains insufficient for training SOTA generative architectures from scratch. Therefore, we leverage a model pre-trained on massive geometry-only 3D scans and fine-tune it to adapt to physical 3D generation. Building upon the well-established 3D representation space of it, present PhysXGen, novel yet straightforward framework that combines physical properties
5

---

PhysXNet
Part Number Occurrence Frequency
Physical Dimension Frequency
Number Objects
Number of Objects
4530
30P
2250-
10
730
309
125
200
200
200
600
500
602
702
000
300
1002
Number parts Objects
Physical dimension (cm)
(a) Part number occurrence distribution
(b) Physical dimension distribution
Kinemattc type distribution
Young's modulus distribution
Poisson's 's ratio distribution
Density distribution
4.7%
3T%
43%
0.6%
21.0%
3MS
6.0-000
2,0-1
16-10
0.25-0.30
0.055
0.5-1
10:15
C. Alsclltte joint
Ch. Revoktrim joint
4.0-1..
10-11
11-100
0.30-- 85
0.35-..00
1.5.2.0
2.0...5
2..3.00
300-200
200
411
040-0.45
3.0-7.5
7.5-3.0
6.0-1.00
(c) Kinematic type and material distribution
PhysXNet-XL
Paoic CORCEMAIINN Searr Doors Kn
Containet ADd iance Dining
Office
Fooor EE Elgegr Ee wer.ciicceeee EE
0.02%
1.8%
1.3%
cabinet
table
Clock
Ad justaaa Bratii
0.4%
Pot
e
drawer door
2.1%
Deco at tye wes
Desk
Coffee
Bookshel
liceee
aab
e
3%
bottle faucet
Armchal
piskittci
or awerr
Lamp-
Lant
die rolatee
Plunbing
Round
charrac
Deyi
Side
Laptop
dater-
Storage
Cabine
Heasphone
Pendant
53%
shower knife
chair
even
Keinung Monitor
Computer
Fixture
Buspan
Lighting
wnkkttapaaeellrr kbeeeh Knterr inita
Glass
Lounge
Wall
coman
lding
laptop
(d) Word cloud of object tags
(e) Procedurally generated data distribution
Figure 3: Statistics and distribution of PhysXNet and PhysXNet-XL. (a) Distribution histogram of part number in PhysXNet. (b) Dimensional distribution analysis in PhysXNet, showing physical measurements (length/widd higghh//wwthhheighh/ frequency. (c) Proportional composition kinematic states and material, including density. Young's 's modulus, and Poisson's 's ratio distribution in PhysXNet, visualized through sectoral ratios. (d) Tag frequeecyy statistics for prevalent object labels PhysXNet-XLL (e) Component-Catege distribution of procedurally generated 3D objects in PhysXNet-XI.
with geometry and appearance shown in Figure Our approach achieves this dual objective by simultaneously integrating fundamental physical properties into the generation process while optimizing the structural branch through targeted fine-tuning. This joint optimization enables the production of physically consistent 3D assets that maintain impressive geometry and appearance fidelity.
4.1 Physical 3D VAE Encoding and Decoding
In this subsection, we take the textured mesh output example. To reduce the influence caused by the domain gap between geometric and physical latent space, we build a similar physical VAE for property encoding, following [10). Besides, considering the interdependencies among physical properties, we encode them into a unified latent space.
adopt physical properties: physical scaling (converted by physical dimension) Patm E R x1 affordance priority Pass E Nx1 density
P, E Nx1 and kinematic parameters Pmov 13 Nxi1 (including child g n.xl and parent group

---

Physical Assets VAE Encoding & Decoding
Physical Latent Generation
VAE Decoder
Diffusion
preppro.
Neri Callars Clrrane 30
Vouriiee
Phytical Sparne VAE Ensoder
Phypical Sparse VAE Decoder
Plyskcl
Froperty retirnl
Flow Tranfforme
Spare
VAE Decoder & Output
Physical
BOLK
Pre Frat:
Condition
cnvecio score
Preciain VAE Encoder
Pretran Sparse
Preciain Sparss Flow
Veedies
Phyecal 3D smat in PhyMPartnat
Tranforma
Average
Strudured Latent
Nolt
Figure 4: The architecture of PhysXGen framework. PhysXGen features two-stage architecture comprising: a physical 3D VAE framework for latent space learning, and a physics- aware generative process for structured latent. The former focuses establishing compressed yet information-ric -ich latent representation that encodes physical properties, while the latter specializes in generating physical latents.
index IR 1x1 movement direction I Nx3, movement location RNx33, movement range RNxx2 and kinematic type Rx11),, where N is the number of voxel. The physical properties phy E R can be obtained by channel-wise concatenation. For the function descriptions, we adopt the CLIP model (32 to obtain the text embedding. Similarly, the description features (Psem R 768x33 are formed by concatenating the basic, functional, and kinematic description embeddings. Besides, the structural branch adopts the DINOv2 to extract features. Therefore, the dimensions of structural 1024 For clarification, we denote the pretrain VAE encoder and decoder as Eass and D Daes while the physical VAE encoder and decoder Ephy and D Pahg- The physical latent Ppiat E Nx8 and structured latent Pa stot E RNx8 be formulated follows:
feature is Pa Paes E X
Pptot = Ephy (Ppky: Psem), Pelat = Eacs (Peee)
To study the effects of physical properties on geometry and appearance quality, introduce branch from Dpky to Does via residual connection. We will analyze the performance of the independent and dependent VAE decoder in the experiments. After decoding the structured and physical latents, we can implement loss function L as follows:
Lyee Cealor + Cgeametry oes
Lypoy + feee + Lua + freg
where Cootor and Cgcometry represent the color loss (including L2loss, lpip loss) and geometry Laes
loss (including mask, normal, and depth loss). For Lphy and Lsem, we normalize the groundtruth respectively and adopt 1.2 loss. Lki aims to constraint the distribution of Pplat while Lreg can reduce the unnecessary structures of textured mesh.
4.2 Physical Latent Generation
Following the acquisition of the compressed physical latent representation, construct transformer- architecture diffusion model to jointly generate physical and structural attributes. To effectively leverage the inherent correlations between physical properties and structural features while main- taining compatibility with pre-trained components, we implement a dual-branch architecture that integrates structural guidance through residual connections. Specifically, the additional branch from the structural module is fused with the primary physical generation module via learnable skip-conne coan-econntiion layers, enabling cross-domand feature interaction. Comprehensive ablation studies quantitatively validate the design rationale through systematic component comparisons. Follow- ing [10, we adopt the Conditional Flow Matching (CFM) as the objective of optimization. Therefore, the loss of the geometric branch is formulated:
Lues = Ee,
(I (xx, (6-ro))
where and represent the noise and timestep while 2n is sampled from Psiat Adopting similar objective for the physical branch, the final loss of the latent diffusion model can be calculated as: Laiss Loes Ephy.
7

---

Table 2: Quantitative comparison of different methods on the test sets of our PhysXNet. There are two types of evaluations: structural and physical property evaluations. PhysPre represents separate physical property predictor after TRELLIS.
Methods
Geometry
Absolute J
Material Affordance
Kinematic parameters COVt
Description
PSNR 1 CD F-Score
MMD
TRELLIS 10
24.31
13.2
TRELLIS PhysPre
24.31
13.2
76.9
13.21
8.63
7.23
0.24
0.12
6.55
PhyxXGen
Image Prompts
Geometry and appearance
Physical properties
Absolute
Affordance
Function Description
Physical dimension:
T
anyoff adjust temperature.
cm
Material
Kinematics
Child
Porent
lange: (42.1187
T T
Pos. [0S5, 0.04, 0.073)
Absolute Scale
Affordance
Function Description
Physical
Find the mush fabric backrest surface the chair
dimension:
58. 38.92x68. 2x64.3
Material AU yplin?
Kinematics
Ghild
Panett part
Fange: 159.5,148.1
(A,22,.854 auaa)
Poe 0.110001
Figure 5: Visualization of the generated results. Given single image the prompt, our PhysXGen can generate the grrsical-grounee 3D assets.
5 Experiments
5.1 Implementation details
In our experiments, we partition PhysXNet dataset into 24K training samples, 1K validation samples, and 1K test cases. By analyzing the performance on the test cases, we can evaluate the generalizability of our method. During the VAE and diffusion model training, we adopt Adam with an initial learning rate of 1 x 10-4 to optimize the models. 'he inherent correlation between geometric configuration and physical properties in our methodology creates critical dependency where the structural fidelity of the 3D representation will affect the final generative performance. In this paper, we repurpose the geometry- and appearance-riee -ich structural space of TRELLIS (10] for our task. Our PhysXGen is trained 8 NVIDIA A100 GPUS. More details about the architecture released in the supplementary.
5.2 Evaluation Metrics
Physical properties evaluation. Our framework establishes a multi-property feature space en- compassing five core attributes: absolute scale, material, affordance, kinematics, and function descríptors. Note that the kinematics attribute manifests as dual configuration parameters: 1) struc- tural grouping (parent-child part hierarchies) and 2) kinematic parameters. Specifically, we evaluate absolute scale using Euclidean distance, density and affordance images via Peak Signal-to-Ne Ratio (PSNR), kinematics with instantiation distance [33, and functional description through PSNR on cosine similarity score maps.
Geometry evaluation. For appearance evaluation, we sample 30 random views from unit sphere to calculate the mean PSNR. Besides, to evaluate the quality of geometry, we calculate the standard shape metrics of Chamfer Distance (CD) (x10-3) and F-score (FS) (x 10-2) with thresholds of 0.05.

---

Table 3: Ablation studies about the physical 3D VAE and diffusion model. Dep- PAp-VAE and Dep-Diff represent the model that utilizes the interdependencies between structural and physical information. Thus, PresPis+PhysPre and PhysXGen are corresponding to the first and last lines.
Dep-VAE Dep-Diff
Geometry
Absolate scale
Material f Affordance
Kinematic parameters
Description
PSNR
CD L F-Score
COV f
MMD
24.31
76.9
13.21
8.63
7.23
0.24
0.12
6.55
24.31
13.2
76.9
12.01
10.69
8.95
0.26
0.11
7.71
77.0
10.57
24.53
12.7
77.3
7.24
13.01
11.30
0.33
0.08
10.11
Prompts
Property
PREsee
PhysXGen
Ground Truth
Affordance
TRELLIS+PhysPre
PhysXGen
Function Description
Find the part that serves as the main surface for placing items
PhysPes+PhysPre
PhysXGen
Ground Truth
Physical Dimension:
Physical Dimension: 31.2x62.4x88..5 cm
Physical Dimension: 60x60x90 cm
30cm
72.3x71.5x97. cm
Absolute
Scale
Physre
PhysXGen
[
Material & Density
0.583 g/cm^3
3.448 g/cm33
7.48 g/cm/3
PResLi
PhysXGen
Type:
Type: C
Range: (-174.6, 1169.2]
Range: (-159.8, 160.8)
Dir: [-0.42-0..33.0.26]
Dir: [0.13,0.05,0085
Kinematics
Pos: [0.32,0.16,0052]
Pos: (0.06,0.01,0.83]
Child part Parent part
Child part
Parent part
Figure 6: Qualitative comparison of different methods. Compared with our baseline, PhysXGen
achieves significant improvements, clearly demonstrating its strong performance in physics-grounder 3D generation.
5.3 Quantitative Results
As shown in Table 2 we implement the quantitative evaluations two types of metrics: 1) geometry and appearance evaluation; and 2) physical properties evaluation. Note that TRELLIS+PhysPre is our baseline that adopts the independent structure to predict the properties. Compared with the separate physical property predictor, our PhysXNet utilizes the correlation between physical and pre-defined 3D structural space, achieving significant improvement in physical property generation while enhancing the aesthetic quality.
Ablation studies. The core design of our framework is to integrate both geometry and physics in 3D modeling. Therefore, we conduct ablation studies validate its effectiveness (reported in 3). By introducing geometry and appearance features in the diffusion model, the generative model can gain improvement physics generation compared with the independent models, PhysPre. Additionally, the correlation between geometry and physics in VAE can enhance the geometry of generated assets. Finally, relying on the dual-architecture and joint training, our PhysXGen obtains impressive performance in all physical property generation.

---

5.4 Qualitative Results
3D assets generated by our PhysXGen. By learning the interdependencies between physical and structural space, PhysXGen achieves impressive performance in generating physical properties. Besides, we perform qualitative comparisons with our baseline shown in Figure [ As we mentioned above, for absolute scaling, we use the Euclidean distance while we adopt PSNR to evaluate the material maps, affordance maps, function description similarity score maps. By utilizing the interdependencies between physical properties and structural information, especially geometry, our PhysXNet obtains higher overall scores. Furthermore, our PhysXGen can distinguish the properties of different parts and achieve more stable and robust performance in physical property generation of neighboring structures, especially in function description, material, and affordance. More experimental results are shown in the supplementary.
Figure D showcases the
6 Conclusion
In this paper, to fill the gap between existing synthesized 3D assets and real-world applications, we propose an end-to-end generative paradigm for physical-grounded 3D asset generation, including the first
3D dataset and the novel physical property generator. Specifically, we develop a human-in in-the-loop annotation pipeline that transforms current 3D repositories into physics-enabled datasets. Mcanwhile, the novel end-to-end generative framework. PhysXGen, can integrate physical priors into cruuctral-focusee architectures to achieve robust generation performance. Through comprehensive experiments PhysXNet, we reveal the fundamental challenges and direction in physical 3D generation. We believe that our dataset will attract research attention from different communities, including but not limited embedded AI, robotics, and 3D vision.
Limitations and Future works. Despite impressive performance, our method exhibits limitations in learning fine-grained properties and suffers from artifacts. In our future work, we will try to handle it. Besides, we will include more 3D data from synthetic to real to improve the diversity of our dataset and integrate additional physical properties and kinematic types to better simulate material behavior and movement.
Acknowledgment
This study is supported by the National Key R&D Program of China (2022/D0160201), and Shanghai Artificial Intelligence Laboratory. This study is also supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOET2EP20221 10012, MOÉ-T2EPEE DDEE-22P20223-0002 This research is also supported by cash and in-kind funding from NTU S-Lab and industry partner(s).
References
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 152.0:151..03012, 2015.
[2] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi. and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13142-13153, 2023.
[3] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe 10m+ 3d objects. Advances in Neural Information Processing Systems, 16:35799-35813 2023.
[4] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren. Liang Pan. Wayne Wu, Lei Yang, Jiaqi Wang. Chen Qian, al. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 803-814, 2023.
[5] Fanbo Xiang. Yuzhe Qin, Kaichun Mo. Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan. He Wang, et al. Sapien: A simulated part based interactive environment. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11097-11107, 2020.
10

---

[6] Kaichun Mo. Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 909-918. 2019.
[7] Jasmine Collins, Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente, Thomas Dideriksen, Himanshu Arora, al. Abo: Dataset and benchmarks for real-world 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21 2116-21136, 2022.
[8] Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and Hc Wang. Gapartnet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7081-7091, 2023.
[9] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[10] Jianfeng Xiang. Zelong Lv. Sicheng Xu, Yu Deng, Ruicheng Wang. Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:4412.01506, 2024.
[1I] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwci Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1-18. Springer, 2024.
[12] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan, Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191, 2024.
[13] Ziang Cao, Zhaoxi Chen, Liang Pan, and Ziwei Liu. Collaborative multi-modal coding for high-quality 3d generation. arXiv preprint arXiv:2508. 15228, 2025.
[14] Fangzhou Hong. Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Shuai Yang. Tengfci Wang. Liang Pan, Dahua Lin, et al. 3dtopia: Large text-to-3d generation model with hybrid diffusion priors. arXiv preprint 2xXhv:2433.02234 2024.
[15] Zhaoxi Chen, Jiaxiang Tang, Yuhan Dong, Ziang Cao, Fangzhou Hong, Yushi Lan, Tengfei Wang, Haozhe Xie, Tong Wu, Shunsuke Saito, et al. 3dtopia-xl: Scaling high-quality asset generation via primitive diffusion. arXiv preprint arXiv:2409. 129554 2024.
[16] Ziang Cao, Fangzhou Hong, Tong Wu, Liang Pan, and Ziwei Liu. Large-vocabulary 3d diffusion model with transformer. arXiv preprint 2rXiv:230,.07920, 2023.
[17] Ziang Cao, Fangzhou Hong. Tong Wu, Liang Pan, and Ziwei Liu. Difftf++: 3d-aware diffusion transformer for large- vocabullay 3d generation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025.
[18] Liu Liu, Han Xue, Wengiang Xu, Haoyuan Fu, and Cewu Lu. Toward world pose estimation. IEEE Transactions on Image Processing, 31:1072-1083, 2022.
category- caterooryyy
articulation
[19] Vicky Zeng, Tabitha Edith Lec, Jacky Liang, and Oliver Kroemer. Visual identification of articulated object parts. in 2021 icee. In RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2443-2450.
[20] Xiaohao Sun, Hanxiao Jiang, Manolis Savva, and Angel Chang. Opdmulti: Openable part detection for multiple objects. 2024 International Conference on 3D Vision (3DV), pages 169-178. IEFF, 2024.
[21] Zoey Chen, Aaron Walsman, Marius Memmel, Kaichun Mo, Alex Fang, Karthikeya Vemuri, Alan Wu, Dieter Fox, and Abhishek Gupta. Urdformer: A pipeline for constructing articulated simulation environments from real-world images. arXiv preprint MrXiv:2405.11665 2024.
[22] Yijia Weng. Bowen Wen, Jonathan Tremblay, Valts Blukis, Dieter Fox, Leonidas Guibas, and Stan Birchfield. Neural implicit representation for building digital twins of unknown articulated objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3141-3150, 2024.

---

[23] Cheng-Chun Hsu, Zhenyu Jiang, and Yuke Zhu. Ditto in the house: Building articulation models of indoor scenes through interactive perception. In 2023 IEEE International Conference Robotics and Automation (ICRA). pages 3933-3939. IEEE, 2023.
[24] Long Le, Jason Xie. William Liang. Hung-Ju Wang. Yue Yang. Yecheng Jason Ma. Kyle Vedder. Arjun Krishna, Dinesh Jayaraman. and Eric Eaton. Articulate-any annthhhnn Automatic modeling of articulated objects via a vision- Saiguale foundation model. arXiv preprint arXiv:2410.13888 2024.
[25] Ziang Cao, Fangzhou Hong, Zhaoxi Chen, Liang Pan, and Ziwei Liu. Physx- ayyx-anything: Simulation-reat -rady physical 3d assets from single image. arXiv preprint arXiv:2511.1306 2025.
[26] Xiaowen Qiu, Jincheng Yang. Yian Wang, Zhehuan Chen, Yufei Wang, Tsun-Hsuun Wang. Zhou Xian, and Chuang Gan. Articulate anymesh: Open-vocabulary 3d articulated objects modeling, arXiv preprint arXiv:2502.025500 2025.
[27] Licheng Zhong. Hong-Xing Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring poring-mass mass 3d gaussians. In European Conference on Computer Vision, pages 407-424 407-423. Springer, 2024.
[28] Albert Zhai, Yuan Shen, Emily Y Chen, Gloria X Wang, Xinlei Wang, Sheng Wang, Kaiyu Guan, and Shenlong Wang. Physical property understanding from language-embest embedded feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28296 28296-28305, 2024.
[29] Mariem Mezghanni, Théo Bodrito, Malika Boulkenafed, and Maks Ovsjanikov. Physical simulation layer for accurate 3d modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 135 1514-13523, 2022.
[30] Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Yutao Feng. Yin Yang. and Chenfanfu Jiang. Physgaussian: Physies-integrated 3d gaussians for generative dynamics. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 4389-4398, 2024.
[31] Minghao Guo, Bohan Wang, Pingchuan Ma. Tianyuan Zhang. Crystal Owens, Chuang Gan, Josh Tenen- baum, Kaiming He, and Wojciech Matusik. Physically compatible 3d object modeling from single image. Advances in Neural Information Processing Systems, 37:1 10111-1229-990 19222, 2024.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[33] Jiahui Lei, Congyue Deng, Bokui Shen, Lconidas Guihas, and Kostas Danilidis, Nap: Neural 3d articulation prior. arXiv preprint arXiv:2305. 16315, 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section 6
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We included the implementation details in the main paper and the supplemen-
tary.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release the code of our generative model and examples of our dataset in the supplementary. We will release the full data as soon as possible.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We release those information in the supplementary
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: did not repeat experiments multiple times since experiments for all bench- mark methods are time-consuming.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: See Section 5
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htti:gtttp/e//eerpepcc/ooooaaaaa
13

---

Answer: [Yes]
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
14


Reviews:
[{"review_id": "iOLdrxF9WN", "reviewer": "Reviewer_NyNc", "summary": "This paper introduces PhysX, an end-to-end pipeline for physical-grounded 3D asset generation. The core contributions are the creation of PhysXNet, a large-scale, systematically annotated 3D dataset featuring part-level physical properties (such as absolute scale, material, affordance, kinematics, and rich function descriptions), and PhysXGen, a feed-forward dual-branch generative model that jointly models geometry and physical attributes for 3D asset synthesis. PhysXNet uses a human-in-the-loop annotation pipeline leveraging vision-language models for scalable property extraction, and PhysXGen combines geometry- and physics-aware components to inject physical realism in 3D synthesis. The paper demonstrates advantages over baselines through quantitative metrics and qualitative visualization, and benchmarks the impact of architectural choices via ablation studies.", "questions": "1: Where is the original data (such as paired part meshes and materials) from? Is it only from PartNet?\n\n2: Do authors try to curate existing datasets into PhysGen to scale up the dataset, although some of them may miss some kinds of annotations?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1: The introduction of PhysXNet is a strong contribution, as summarized in Table 1 and discussed in Section 3. It offers over 26K 3D objects annotated with five key physical properties at the part level, which fills a significant gap in current 3D datasets. Its human-in-the-loop annotation pipeline (Fig. 2) with VLM assistance is thoughtfully developed and demonstrated to be practical and scalable.\n\n2: The taxonomy of physical properties (absolute scale, material and density, affordance, kinematics, and function description) is systematically discussed and collected, which is important for both simulation and embodied AI robotics tasks.\n\n3: The paper supports its claims with thorough empirical results. Experimental results show that the proposed dataset can facilitate 3D generation methods.\n\n4: The paper is well organized and written. Figures 4, 5, and 6 clearly show the architectural layout, generated outputs, and property-specific qualitative comparison, providing strong visual evidence of the method's efficacy. For example, Table 2 quantitatively demonstrates improvements in both physical property predictions and geometric fidelity, central to the paper's claim of joint modeling. Figure 6 visualizes similarity scores for physical attributes, supporting detailed comparative claims.\n\n5: The PhysXGen model is built on a dual-branch architecture that effectively utilizes pretrained structural priors and jointly learns correlations with physical properties. The architecture choices are clearly described and reasoned. The use of VAEs and conditional diffusion models for both geometry and physics is up to date, and training protocols appear sound.\n\nWeaknesses:\n\n1: **The experimental baselines are somewhat limited.** Although TRELLIS and TRELLIS+PhysPre are sensible, they are little compared to other recent strong generative models, especially those capable of handling physical aspects or integrating physical priors. The performance improvement is a somewhat narrow scope of empirical benchmarking (PSNR and CD).\n\n2: **Generalization and Diversity**. While the dataset is larger than some previous efforts, its generalization to other data domains (e.g., real-world scanned data, non-PartNet-derived assets) is not addressed. The claim of generalization is not fully substantiated with, for example, cross-dataset or out-of-distribution evaluations. \n\n3: **The applications are limited.** The authors only discuss that the proposed dataset can be used to generate physical 3D objects. From my understanding, the proposed dataset can also help 3D part-level perception models (like 3D parts segmentation) and part-based 3D generation models. It would be promising if the dataset could also improve the performance on these tasks (or this question could be simply left to the research community).\n\n4: **Physical Dimension Evaluation**. The evaluation of physical properties is reduced to a Euclidean comparison between rendered maps, which may not fully capture the semantics of errors in properties such as kinematics or affordances. This could be especially problematic for parts of the evaluation with a strong qualitative or functional component.\n\nAlthough I raise some minor weaknesses, it is noted that this paper makes a strong contribution to collecting data, along with many necessary physical attributes. Therefore, I vote for acceptance at this time and hope the authors could open-source the collected dataset.", "was_revised": true, "final_justification": "This is a solid paper with comprehensive experiments and analysis. I maintain my rating of accept.", "rebuttal": "We sincerely thank you for the encouraging comments and we have carefully addressed all your comments one-by-one as follows:\n\n> **Weakness 1**:  Limited experimental baselines.\n\n**Answer 1**: We appreciate you for the careful suggestion. As detailed in our supplementary material, we have included the GPT-based predictor during evaluations, where our method demonstrates superior performance across most physical properties. Additionally, we provide a new comparison with OOAL [1] on affordance prediction, which further highlights the impressive performance of our approach.\n| Methods | Affordance|\n|-------|------|\n| OOAL [1] | 0.441  |\n| PhysXGen  | 0.372 |\n\nWe would like to kindly clarify that the primary focus of our method is on capturing physically-grounded properties, rather than pursuing geometric fidelity alone. Furthermore, it is important to note that our model achieves improvements while being trained on less data (PhysXNet), compared to original Trellis, which relies on the much larger Objaverse dataset.\n\n[1] Li G, Sun D, Sevilla-Lara L, et al. One-shot open affordance learning with foundation models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 3086-3096.\n\n> **Weakness 2**:  Generalization and diversity problem & data sources\n\n**Answer 2**: Thank you for this advice. Currently, our dataset is primarily based on PartNet. However, we would like to emphasize the strong scalability of our framework. On one hand, our human-in-the-loop annotation pipeline can be readily extended to real-world scans and other 3D datasets beyond PartNet. On the other hand, we also leverage procedural generation to expand the dataset. In fact, we are actively working on incorporating more diverse data sources and extending more data using procedural generation.\n\n> **Weakness 3**:  Limited applications\n\n**Answer 3**: Thank you for the valuable suggestion. While our dataset is designed with a focus on physically-grounded 3D asset generation in this work, we believe it has broad applicability to a range of related tasks such as 3D affordance prediction, 3D segmentation, and part-level generation. The dataset offers rich physical annotations that could benefit these tasks by providing a deeper understanding of object functionality and interaction. We view this work as a foundational step, and we plan to explore and support these downstream applications in future work.\n\n> **Weakness 4**:  Unconventional metrics in physical dimension evaluations.\n\n**Answer 4**: We sincerely thank you for your insightful comments. In response, we have revised some evaluation metrics in the paper to better align with different physical aspects. Specifically, we now evaluate absolute scale using Euclidean distance, density and affordance images via PSNR, kinematics with instantiation distance, and functional description through PSNR on cosine similarity score maps. We believe these metrics provide a more comprehensive assessment of the physical properties.\n\n| Methods             | Geometry        |                 |                 | Absolute scale ↓ | Material ↑ | Affordance ↑ | Kinematic parameters |                 | Description ↑ |\n| :------------------ | :-------------  | :-------------  | :-------------  | :--------------- | :--------- | :----------- | :------------------- | :-------------- | :------------ |\n|                     | **PSNR ↑**      | **CD ↓**        | **F-Score ↑**   |                  |            |              | **COV ↑**            | **MMD ↓**       |               |\n| **TRELLIS**    | 24.31           | 13.2            | 76.9            | –                | –          | –            | –                    | –               | –             |\n| **TRELLIS + PhysPre** | 24.31           | 13.2            | 76.9            | 13.21            | 8.63       | 7.23         | 0.24                 | 0.12            | 6.55          |\n| **PhysXGen**        | **24.53**       | **12.7**        | **77.3**        | **7.24**         | **13.01**  | **11.30**    | **0.33**             | **0.08**        | **10.11**     |\n\n> **Weakness 5**:  Curate existing datasets into PhysGen\n\n**Answer 5**: Thank you for your meticulous suggestion. As physical property learning typically requires a multi-task loss to jointly optimize diverse attributes, it becomes challenging to train effectively when only partial annotations are available. To address this issue, we are actively extending our dataset using procedural generation and annotating more diverse 3D data."}, {"review_id": "PxoR08PoiC", "reviewer": "Reviewer_aaVJ", "summary": "The paper introduces PhysX, an end-to-end paradigm for generating 3D assets with physical properties, addressing a key gap in current 3D generative models. A central contribution is PhysXNet, \"the first comprehensive physics-grounded 3D dataset,  comprising over 26K richly annotated 3D objects with fine-grained physical attributes like scale, material, and kinematics. This dataset, built via a \"scalable human-in-the-loop annotation pipeline,\" enables PhysXGen, a novel framework that integrates physical knowledge into 3D asset generation, demonstrating strong performance in creating physically plausible models.", "questions": "1. What is the estimated accuracy of the VLM's initial annotations for each property type (e.g., material, affordance, kinematics), and what is the average human time/effort required for the \"expert refinement\" per object, especially for complex kinematic behaviors?\n\n2. How well would PhysXGen generalize to generating physical properties for 3D objects belonging to entirely novel categories or exhibiting physical behaviors not extensively represented in the PhysXNet dataset?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n  \n- PhysXNet is the first dataset to systematically annotate 3D objects with a wide range of physical properties (scale, material, affordance, kinematics, function descriptions) at a fine-grained, part-level granularity. This goes beyond existing datasets that often provide only object-level or limited physical information.  \n- With over 26,000 objects, PhysXNet provides a substantial foundation for training and evaluating physics-aware 3D models.  \n- The proposed human-in-the-loop pipeline, integrating vision-language models (GPT-40) for initial labeling and human refinement, is an efficient and robust approach to creating such a detailed dataset.  \n- The paper proposes an end-to-end paradigm for physical-grounded 3D asset generation, addressing a critical and underexplored area in 3D content creation. This work has high potential impact for fields like robotics, simulation, and embodied AI. \n\nWeakness  \n\n- The paper's comparisons primarily focus on geometry-centric 3D generation or separate physical predictors. It would strengthen the evaluation to include comparisons with methods that directly learn material concepts from images/videos, such as those proposed by Zhai et al. (2024) or Zhong et al. (2024), to better contextualize PhysXGen's performance in specific physical property prediction.  \n\n- While the paper demonstrates performance on PhysXNet's test set, the generalization capability of PhysXGen to generating physical properties for 3D objects belonging to entirely novel categories or exhibiting physical behaviors (e.g., highly deformable objects, fluid interactions) not extensively represented in the PhysXNet dataset remains underexplored.", "was_revised": true, "final_justification": "Thank you for the detailed rebuttal. It has addressed all of my concerns. I will keep my original score.", "rebuttal": "We thank you for your valuable suggestions and efforts to refine our work. We have carefully addressed all your comments one-by-one as follows:\n\n> **Weakness 1**: Limited comparisons with approaches that directly learn properties from images/videos.\n\n**Answer 1**: We appreciate you for the careful suggestion. As detailed in our supplementary material, we have included the GPT-based predictor that learn from image during evaluations, where our method demonstrates impressive performance across most physical properties. Additionally, we provide a new comparison with OOAL (learn from image) [1] on affordance prediction, which further highlights the impressive performance of our approach.\n\n| Methods | Affordance|\n|-------|------|\n| OOAL [1] | 0.441  |\n| PhysXGen  | 0.372 |\n\nWe would like to kindly clarify that the primary focus of our method is on capturing physically-grounded properties, rather than pursuing geometric fidelity alone. Furthermore, it is important to note that our model achieves improvements while being trained on less data (PhysXNet), compared to original Trellis, which relies on the much larger Objaverse dataset.\n\n[1] Li G, Sun D, Sevilla-Lara L, et al. One-shot open affordance learning with foundation models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 3086-3096.\n\n> **Weakness 2**: Limited evaluations on novel cases and novel categories.\n\n**Answer 2**: We thank the reviewer for the helpful suggestion. To further evaluate the generalization ability of our method, we have conducted an additional user study. In this study, we assess our method's performance on novel objects and novel categories, particularly focusing on deformable objects and selected physical attributes. Participants were asked to rate the plausibility of predicted properties on a 5-point scale: 5 – Highly Reasonable, 4 – Mostly Reasonable, 3 – Somewhat Reasonable, 2 – Barely Reasonable, 1 – Completely Unreasonable. \n\nAfter collecting the scores, we normalized them and present the results in the table below for clarity. It prove the promising generalization capability of PhysXGen in some physical properties. \n\n|          |                         |                         |                         |                         |\n| :------- | :---------------------- | :---------------------- | :---------------------- | :---------------------- |\n|          | **Novel cases**          |                         | **Novel categories**      |                         |\n| **Methods** | **Affordance**         | **Material**           | **Physical scale**      | **Material**            |\n| **Baseline** | 0.515                  | 0.547                  | 0.523                 | 0.481                  |\n| **PhysXGen** | **0.811**              | **0.780**              | **0.760**              | **0.742**              |\n\nTo fundamentally address this generalization challenge, we are actively extending our dataset using procedural generation and annotating more diverse 3D data.\n\n> **Weakness 3**: Detailed questions about our annotation pipeline\n\n**Answer 3**: We appreciate you for this detailed question. The initial annotation quality from VLM models is closely related to the complexity of the physical property. For relatively straightforward properties such as physical dimensions, material (density), and functional descriptions, the success rate exceeds 90%. For more challenging attributes like affordance, the success rate drops to around 60–70%. For the most difficult property—kinematic information—the total success rate is close to zero. However, it can provide some useful information such as kinematic group information and neighboring parts. Based on it, kinematic-related annotations are manually annoated, which takes approximately 1 minute per object on average."}, {"review_id": "TS3GLqN900", "reviewer": "Reviewer_zbQv", "summary": "This paper introduces PhysXNet, a new dataset with rich part-level physical annotations, and a corresponding new physical generative model PhysXGen. PhysXNet is semi-automatically annotated with a human-in-the-loop review, ensuring its high quality. PhysGen further demonstrates the effectiveness of this dataset.", "questions": "- Are PhysXNet and PhysXPartNet the same thing?\n- What is the metric for 'material'? I suggest authors add a brief introduction about metrics.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- A new dataset with rich physical annotations. Happy to see a new PartNet-like dataset built with the help of VLM. This will advance physical-based generation and related fields such as robotics.\n- A new method based on the dataset introduced, showing better performance than the combination of previous methods.\n\n### Minor Weaknesses\n- The design of PhysXGen is not very novel. It appears to be a new branch of adaptor based on pre-trained models. I suggest that authors show the computational cost of the adaptor and the whole model. It would be great if physical parameters could be predicted efficiently.", "was_revised": true, "final_justification": "Since my concerns are addressed, I keep my original rating.", "rebuttal": "We thank you for your meticulous comments and suggestions to improve our work. We have carefully addressed all your comments one-by-one as follows:\n\n> **Weakness 1**: Missing computational cost of model.\n\n**Answer 1:** Thank you for the insightful comment. We have compared our model with the original Trellis model in terms of the number of parameters and inference time (evaluated on an NVIDIA A800 GPU). The results show that our model achieves impressive efficiency.\n| Methods | Parameters| Inference time|\n|-------|------|----|\n| Trellis | 1.2B  | 12.3s |\n| PhysXGen  | 1.87B | 15.4s|\n\n> **Weakness 2**: Naming inconsistency\n\n**Answer 2:** Yes, we apologize for the typo and will correct it in the revised version of the paper.\n\n> **Weakness 3**: Lack of a brief introduction about metrics.\n\n**Answer 3:** Thank you for the valuable suggestion. We have revised several evaluation metrics in the paper to better align with different physical aspects of the task. Additionally, we will include a brief introduction to these metrics to clarify them."}, {"review_id": "y37QE6cXBz", "reviewer": "Reviewer_47JR", "summary": "The paper presents a dataset of physical properties of 3D objects, building up on PartNet, including material, size, affordance, kinematics and functionality. The data was generated by prompting an LLM under human supervision. The paper further develops an approach to generate the physical properties within a 3D generation pipeline.", "questions": "See weaknesses above, summarized here:\n1) How to the labels compare to the ones from prior datasets, e.g. PartnetMobility?\n2) What parts of the annotation pipeline, especially the kinematics, is done by humans?\n3) How is the MAE between rendered images used to evaluate the property generation?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe dataset is very valuable for research on 3D generation, as generating 3D objects for simulations or other applications requires such physical properties. The generative approach is compatible with existing 3D generation methods (e.g. Trellis) and the results demonstrate that the proposed joint approach improves over a separate physical property predictor.\n\nMajor weaknesses:\n1) The quality of the dataset is not validated in the paper. It depends a lot on the rigor of the human annotator (of 26k objects) whether the annotations are correct, since it is unlikely that GPT-4 has sufficient physical understanding to label these properties on a part level. How much correction was necessary? To verify the validity of the annotations, one could compare selected properties with the annotations from other datasets. For example, the labeled kinematics could be compared to the ones in the PartNetMobility dataset.\n2) The methodology is not explained well. Section 3.2 is very brief and not clear to me, e.g. how can k-means determine the location of the rotation axis? How is the type of kinematics (A-E) determined? What tasks in part 2 (kinematic parameter estimation) are done by the human annotator?\n3) The metric used (MAE of rendered images) is very unconventional. How are the properties encoded in these images? Why not use suitable per-property metrics from prior work? E.g., Euclidean distance for the absolute scale, text embedding cosine similarity for material and functionality, and suitable measures for kinematics such as instantiation distance [1]?\n\nMinor weaknesses:\n1) The dataset is presented as if it was a new 3D dataset, but the objects are entirely taken from PartNet. It should be clarified that this dataset is providing annotations for an existing dataset. \n2) Regarding “we merge the tiny parts whose vertices and area are smaller than a pre-defined threshold”: This sounds a bit dangerous with respect to kinematics. What if these merged parts have a non-rigid relationship?", "was_revised": true, "final_justification": "By doing additional validation studies and integrating further metrics, the authors have fully addressed my concerns.", "rebuttal": "We appreciate you for your encouraging comments. We have carefully addressed all your comments one-by-one as follows:\n\n> **Weakness 1**: Lack of a quality comparison for PhysXNet.\n\n**Answer 1**: We sincerely thank you for the detailed comments. Indeed, the quality of the initial GPT-generated annotations is closely correlated with the complexity of the physical property. For relatively straightforward properties such as physical dimensions, material (density), and functional descriptions, the success rate exceeds 90%. For more challenging attributes like affordance, the success rate drops to around 60–70%. For the most difficult property—kinematic information—the total success rate is close to zero. However, it can provide some useful information such as kinematic group information and neighboring parts. Based on it, kinematic-related annotations are manually annoated. To validate the accuracy of the annotations, we conducted a user study comparing PhysXNet with PartNet-Mobility. Participants were asked to evaluate each annotation pair based on three options: (1) PhysXNet is better, (2) PartNet-Mobility is better, or (3) both are equally reasonable. Based on the collected responses, we computed the normalized scores, which demonstrate the high quality of our dataset.\n\n\n| Datasets | Score (Kinematic) |\n|-------|------|\n| Partnet-mobility | 0.502  |\n| PhysXNet   | 0.498 |\n\n\n> **Weakness 2**: Detailed questions about our annotation pipeline.\n\n\n**Answer 2**: Thank you for the thoughtful question. For axis selection, we first use k-means clustering to generate multiple candidate points, from which the final rotation axis is manually selected. The annotations for kinematic A–E are produced via a human-in-the-loop pipeline that combines GPT-generated predictions with manual verification. As for the kinematic properties, all components require manual refinement or are fully annotated (kinematic parameters: location of axis, direction of axis, and kinematic range) by human experts to ensure correctness and consistency. We will include an additional explanation of our annotation pipeline in the new version.\n\n> **Weakness 3**: Unconventional metrics in physical dimension evaluations.\n\n**Answer 3**: We sincerely thank you for this valuable comments. In response, we have revised some evaluation metrics in the paper to better align with different physical aspects. Specifically, we now evaluate absolute scale using Euclidean distance, density and affordance images via PSNR, kinematics with instantiation distance, and functional description through PSNR on cosine similarity score maps. We believe these metrics provide a more comprehensive assessment of the physical properties.\n\n| Methods             | Geometry        |                 |                 | Absolute scale ↓ | Material ↑ | Affordance ↑ | Kinematic parameters |                 | Description ↑ |\n| :------------------ | :-------------  | :-------------  | :-------------  | :--------------- | :--------- | :----------- | :------------------- | :-------------- | :------------ |\n|                     | **PSNR ↑**      | **CD ↓**        | **F-Score ↑**   |                  |            |              | **COV ↑**            | **MMD ↓**       |               |\n| **TRELLIS**    | 24.31           | 13.2            | 76.9            | –                | –          | –            | –                    | –               | –             |\n| **TRELLIS + PhysPre** | 24.31           | 13.2            | 76.9            | 13.21            | 8.63       | 7.23         | 0.24                 | 0.12            | 6.55          |\n| **PhysXGen**        | **24.53**       | **12.7**        | **77.3**        | **7.24**         | **13.01**  | **11.30**    | **0.33**             | **0.08**        | **10.11**     |\n\n\n> **Weakness 4**: Concerns regarding the novelty of the dataset.\n\n**Answer 4**: Thank you for the constructive comment. While our dataset is based on PartNet, we have manually refined and integrated the original data to improve consistency and usability, and we have also added new physical attributes. Furthermore, we are actively expanding the dataset in two directions: (1) generating entirely new objects—**beyond those in PartNet**—using procedural generation methods, and (2) incorporating **more diverse 3D data** sources (Objaverse).\n\n\n> **Weakness 5**: Question about our merge processing.\n\n**Answer 5**: Thank you for pointing this out. We manually refine the results of the merging process to ensure that the merged outputs are reasonable and consistent. We will include an explanation of this step in the revised version of the paper."}]

Author rebuttal:

Meta-review: This paper introduces PhysX, a framework for generating 3D assets with physical groundings, including a new dataset (PhysXNet) and a generative model (PhysXGen). The review process resulted in a clear consensus for acceptance. The reviewers recognized the novelty and significance of the contributions, particularly the new dataset.

Summary
The paper makes two primary contributions to bridge the gap between virtual 3D asset generation and real-world physical plausibility. First, it introduces PhysXNet, a 3D dataset systematically annotated with a rich set of fine-grained physical properties, including absolute scale, material, affordance, kinematics, and function descriptions. The dataset is constructed using an efficient human-in-the-loop pipeline that leverages vision-language models. Second, it proposes PhysXGen, a novel dual-branch generative framework that is trained on PhysXNet to explicitly model the latent correlations between 3D structure and physical properties, thereby generating assets that are both geometrically and physically plausible.

Strengths
There is a consensus among reviewers regarding the paper's strengths:
- The introduction of PhysXNet is a major contribution. It addresses a clear and important need in the field for 3D data with rich physical annotations. The scale of the dataset (~26K objects) and the comprehensive taxonomy of physical properties make it a valuable resource for future research in robotics and simulation.
- Moving 3D generative models beyond purely geometric and textural fidelity to encompass physical realism is a next step for real-world applications.
- The proposed PhysXGen model is well-designed, with a dual-branch architecture that effectively learns the joint distribution of shape and physics. The paper is supported by thorough experiments and ablation studies that demonstrate clear improvements over relevant baselines.

Weaknesses
While the overall assessment was positive, the initial reviews raised some valid points for clarification:
- Initial reviews questioned the validation process for the new dataset's annotations, particularly for complex properties like kinematics.
- The initial metrics used for evaluating some physical properties were noted as unconventional and could be better aligned with prior work.
- Some reviewers suggested that the experimental comparisons could be strengthened by including a wider range of baselines.

Reasons for Recommendation
The work is well-executed, clearly written, and addresses a timely and important problem. Researchers from various communities (e.g., 3D vision, robotics, and generative models) would benefit from this work.   a spotlight (SAC recommend).
Decision: Accept (spotlight)