NAP: Neural 3D Articulated Object Prior
Authors: Jiahui Lei
Congyue Deng
Bokui Shen
Leonidas Guibas
Kostas Daniilidis
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D articulated objects
diffusion models
generative models

Abstract: We propose Neural 3D Articulated object Prior (NAP), the first 3D deep generative model to synthesize 3D articulated object models. Despite the extensive research on generating 3D static objects, compositions, or scenes, there are hardly any approaches on capturing the distribution of articulated objects, a common object category for human and robot interaction. To generate articulated objects, we first design a novel articulation tree/graph parameterization and then apply a diffusion-denoising probabilistic model over this representation where articulated objects can be generated via denoising from random complete graphs. In order to capture both the geometry and the motion structure whose distribution will affect each other, we design a graph denoising network for learning the reverse diffusion process. We propose a novel distance that adapts widely used 3D generation metrics to our novel task to evaluate generation quality. Experiments demonstrate our high performance in articulated object generation as well as its applications on conditioned generation, including Part2Motion, PartNet-Imagination, Motion2Part, and GAPart2Object.

NAP: Neural 3D Articulated Object Prior
Jiahui Lei!
Congyue Deng"
Bokui Shen?
Leonidas Guibas² Kostas Danillidis
University of Pennsylvania
Stanford University 3 Archimedes, Athena RC
htt.://ssssstaooroooooooooo (leijh, kostas)čcis upenn edu, {congyue, willshen, guibas}0cs. stanford, edu
https:// cis.
Abstract
We propose Neural 3D Articulated object Prior (NAP), the first 3D deep generative model to synthesize 3D articulated object models. Despite the extensive research on generating static objects, compositions, or scenes, there are hardly any ap- proaches capturing the distribution of articulated objects, common object category for human and robot interaction. To generate articulated objects, we first design novel articulation tree/graph parameterization and then apply diffusion- denoising probabilistic model over this representation where articulated objects be generated via denoising from random complete graphs. In order capture both the geometry and the motion structure whose distribution will affect each other, we design graph denoising network for learning the reverse diffusion pro- cess. We propose a novel distance that adapts widely used 3D generation metrics novel task to evaluate generation quality. Experiments demonstrate high performance articulated object generation as well as its applications on condi- tioned generation, including Part2Motion, PartNet-Imaginateee Motion2Part, and GAPar120bject.
DO. NAP
NAP
Part Condition
Static Object
1OLF NAP
NAP generate articulated objects
NAP
NAP
Joint
GAPart Condition
Condition
Figure 1: NAP unconditionally generate articulated objects (left). It can be conditioned on just parts joints (mid), subset of parts plus joints, over-segmented static objects (right).
Introduction
Articulated objects are prevalent in our daily life. As humans, we have strong prior knowledge of both object part geometry and their kinematic structures. Such knowledge is most heavily leveraged when a designer designs a cabinet from scratch, creating both its geometry and motion structure. For learning systems, interesting challenge is to capture such priors generative model that synthesize articulated objects from scratch. While there has been extensive research on generative models for static 3D objects [П-[I,, compositions [[[II8], and scenes [[]] the study of priors regarding closely linked 3D part geometry and 3D motion structures has been relatively neglected. In this work, study how synthesize articulated objects, i.e., how generate full description
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

of an articulated object, an actual URDF [3I] including how each part looks like, which pairs of parts are connected, and with what kind of joint. In contrast to static 3D generation, generating articulated objects involves modeling not only the distribution and composition of geometry but also the motion structures that determine the possible relative movements between rigid parts. Our task is also different than 3D human/hand synthesis [52-5%] where the articulation structure is given and only the degrees of freedom are generated when predicting human poses/sequences. The generation of articulation can further impact simulation and mechanism design and be useful for inference when conditioned on either geometry or kinematic structure.
However, there are several challenges for articulated object generation. Existing datasets of articu- lated objects contain highly irregular data since articulated objects have different numbers of parts, diverse part connectivity topology, and different joint motion types. In order to enable efficient pro- cessing of diverse geometry and structures by a neural architecture, we propose novel unifying articulation tree/graph parameterization (Sec. (.II) to represent articulated objects. We take advan- tage of recent progress in diffusion models for 3D generation and develop a probabilistic model over our parameterization to faithfully model the irregular distribution of artic- ulated objects (Sec. ( Since we are modeling the joint distribution of the part geometry and the inter-part motion constraints, we design a graph denoising network (Sec. (R3) to gradually exchange and fuse information between the edge and nodes on our articulation graph.
To evaluate articulated object generation quantitatively, we adopt the widely used 3D shape genera- tion metric for articulated objects by introducing a novel distance measure between two articulated objects. Through extensive comparisons, we demonstrate high performance in articulated object synthesis and further conduct several ablations to understand the effect of each system component. Using the learned prior knowledge, we demonstrate conditioned generation applications, including Part2Motion, PartNet-Imaginne Imagination,,,,,,,,,, Motion2Part, and GAPart2Object.
In summary, our main contributions are (1.) introducing the articulated object synthesis problem; (2.) proposing simple and effective articulation tree parameterization sufficiently efficient for diffusion denoising probabilistic model that can generate articulated objects; (3.) introducing a novel distance for evaluating this new task; and (4.) demonstrating our high performance in articulated object generation and presenting several conditioned generation applications.
2 Related Work
Articulated object modeling. Modeling articulated objects has been a prolific field with existing works broadly classified into the categories of estimation, reconstruction, simulation, and finally, our generation. Note that there is a wider literature on semi-nonrigid objects, for example, hu- man body [[56]] hands [[7]] and animals [G8]. This paper focuses on everyday articulated objects, like the ones in PartNet-Mobility [3%] which have more diverse and complex structures and are strictly multi-body systems. Estimation focuses on predicting the articulation joint states (joint an- gles and displacements) or joint parameters (type, axis and limits) from sensory observations, using approaches ranging from probabilistic models [_]-24] interactive perception (5-511 and learning- based inference [52-65]]. Reconstruction of articulated objects focuses on reconstructing both artic- ulation and geometry properties of the objects, using techniques ranging from structure from mo- tion [[6i], learning-bant bbased methods [67)-69] and implicit neural representations (I)) While these methods mainly focus on surface reconstruction and joint states/parameter accuracy or are limited to pre-defined simple kinematic structures, we explicitly model the diverse and complex articulation motion structures. One closely related work to ours is [62]], which predicts the joint parameters of articulated objects in PartNet [ by training on MorbiittMooility [[2]. However, [[2]] is a single- point regression that does not capture the generative distribution of joints parts. A growing field in robotics and embodied Al is building interactive environments that support physical interactions between the robot and the scene consisting of articulated objects [/S,, 59 // 82]. Differing from all the above approaches, we build a generative prior of articulated objects, which extends beyond esti- mation and reconstruction. Such learned prior hasential further accelerate the creation of realistic interactive 3D assets.
Generative models for structures. Generating structured articulated objects is closely related to generative models for structured data []3]. Part-based 3D object generation [ [8] has been widely studied with the main modeling target being the hierarchy of static rigid parts with or without se- mantic label. Scene layout generation [[9-30]] utilizes compact scene parameterizations for indoor scene synthesis, where diffusion models on scene graphs have been recently introduced in [21,, 22].
The generative diffusion approach has also been applied widely to 2D floor plans [84], protein struc-

---

tures [85], and graphs [8]] .etc. Unlike all the existing works, our work focuses modeling new type target articulation structure plus part shapes, which requires joint reasoning between 3D geometry and kinematic structure.
Diffusion models in 3D. We mainly focus on the literature on diffusion models for 3D shape and motion generation. Shape: Diffusion models show impressive results in generating point clouds [II- Gl, meshes [X]], implicit surfaces [] [Q], neural radiance fields [88 -I]], or 4D non-rigid shapes [[2]]. However, these methods mostly focus the single object level shape quality and do not pay atten- tion to the kinematic structure. Motion: Diffusion models have seen many recent applications in motion generation given articulation model. Such works generate text-conditioned human tion [32-65],, and physically-vaable [93], audio-driven [94,, BSJ,, scene-aware [96],, multi-human [Р7]] or animation [ trajectories. Diffusion models for motion have also been applied to trajectory planning [09], visuomotor control [аао]], and rearrangement tasks [лτои]]. Different from ours, exist- ing works motion diffusion rely on known geometries with known motion structures. We instead jointly model geometry and motion structure priors to create articulation models.
Method
Articulation Tree (Graph) Parameterization
+ Indicattr [$]
Articulated Olgeet (UD)
lmit pose (6)
K X D. Node anr
Ini 2050
V
Pad
éij
Shape code (128]
Parame
!2
Aediicator [I]
Johge
eij
Pier axis placker (6] Feo Prrdller Range
11/2 D.
Complete Graph
Forward Diffusion: gradually adding random noise
Add noise
Xt
X+-1
X
Parumetertzation
Articulation Tree
Somple random noise Xy
Graph
Sec.3.3
XT
X
Xt-1
Xo
object rextraction
Generation: reverse denoising
Vir for midilee
Figure 2: Method: Parameterization (Top): We parameterize the articulated object as a tree, whose nodes are rigid parts and edges are joints; then pad the tree to complete graph of maximum node number and in the articulation graph attribute list Forward Diffusion (Middle): The parameterized attribute list x is gradually diffused to random noise. Generation (Bottom): Graph Denoiser (Fig Sec. (.33 samples a random articulation graph xy, gradually removes noise, and finally predicts xp- An object extraction stage (Sec. 3.2 including hiiimmum-spanning tree algorithm is applied to the generated graph in the end to find the kinematic tree structure and the output articulated object.
We learn the articulated object priors using diffusion model. However, an articulated structure must first be parameterized into a vector that can be the target of the diffusion. We introduce the parameterization the articulated object as a tree (graph) in Sec. B.I.; the diffusion model on such parameterization in Sec. 122 and the denoising network in Sec. An overview of our method is shown Fig.
3.1 Articulation Tree Parameterization
follow the natural articulated object parameterization as in URDF [31] format where each object is defined by graph with nodes being the parts and edges be- ing the articulation joints. We make two assumptions: (1) Tree assumption: We assume no kinematic motion loop (cycle) exists in the graph and that the graph is connected. (2) Screw joints: We assume
Graph-based representation.

---

each edge is screw [60]] with at most one prismatic translation and one revolute rotation. Most real-world articulated objects [39 776. 82 [02] satisfy the above assumptions and their limitations are further discussed in Sec. M
Nodes. As shown in Fig. tttop. we represent each rigid part as one node in the tree. A node captures its rigid part shape by (I) a shape latent code s; E I (with F' being channels, being the part index) that can decode the SDF [[[о]] of the part surface, and (2) the bounding box edge lengths b, E IR that can scale the decoded SDF properly into the part's 's scale. We pre-train an SDF shape Auto-Encoder [TOAA [C03] for obtaining s;, further details are provided in our supplementary mate- rial. Importantly, one should also specify how to assemble these parts into an object before further modeling the articulation motion. We do so by adding to the node attributes part initial rigid transformation T; E SE(3) in the global object frame representing the part initial configuration. T; comprises an axis-angle rotation and a translation, which can be written as a 6-dimensional vector. As articulated objects can be instantiated to different configurations so any of these configurations can be used as T, in practice, we consistently choose T; by exploiting the canonicalization in the large annotated dataset: we use the part poses in their rest states (zero joint angles and displace- ments) for Tü Since we observe current datasets often align their rest states consistently to static un-articulated objects, for example, PartNet-Mobillty [39] aligns with static PartNet [[Z]] To model the varying number of parts across objects, we define a maximum number of K parts as well as a per-part binary indicator o2 E {0. } of part existence. In summary, node i has an attribute vector 2/- = [o,, T, b,, s,] with dimension D. = 1 + 6 + 3 + F, and the overall node component of an object graph is a feature of dimension K x D...
Edges. The edge in the graph represents the motion constraint of each articulation joint. A joint possesses 3D axis (a directed line), around which the revolute joint can rotate and along which the prismatic joint can translate. Inspired by [60], such a joint axis is represented by Plücker coordinates E 82, m E R²). Here the 3D directional vector is from unit sphere S and the momentum m is perpendicular to I. Such representation avoids defining additional local joint coordinate frames with ambiguity (e.g. translating the joint coordinate frame along revolute axis leads to equivalent joints). Such 6-dimensional Plücker coordinates P(c.) = (la,g) mi.))) for the joint from part to j are defined in the global object frame when parts are in their initial rest configuration T,, Ty. To fully define the joint motion constraint, we also incorporate two joint state ranges T(ij)) E R for both the prismatic translation and the revolute angle components (the left limit of the range can be negative). A purely prismatic joint will have its revolute component range set to [0, 0j and vice versa. Following the node padding, we also pad the edges to a complete graph and use an indicator c(..) denoting the edge existence. Note that the above joint axis and range have a parent-child direction from part i to j. When the direction flips, a notable benefit of expressing P(i.)) in the global object frame, rather than the local part frame, is the inherent relationship P(i.)) -PGaa) and ((..)) "().))- which motivates us to model the padded graph with only K(K - 1)/2 edges for all i < pairs. Since the nodes have no specific order, to avoid the sign flipping in P(c.)) = -P(ja when permuting the nodes and to help the network learn more stable prior, we explicitly embed the sign inside P(c.) = -P(..) into the edge existence indicator C(i,)) E {-1,0, +1J. Here 0 indicates the non-existence of an edge and +1, 1 indicate the chiralities of existing edges, leading to P(..)) = P(..)) (where و denotes the actual prediction target). In essence, each edge is characterized
by an attribute vector ((i))) = [c(K,) P().)) [(.n)) with a dimension of D. = 1 + 6 + 4. The overall edge parameterization of an object has shape K(K- 11/2 x De. We will see later in Sec. how to extract an articulated object model from the diffusion model prediction via the Minimum Spanning Tree.
3.2 iiffusion-Based Articulation Tree Generation
Our goal is to learn the distribution of articulated objects parameterized by the complete articulation graphs x= (fu;). () E IR. ( We apply a diffusion denoising probabilistic model [105] directly over the distribution of x.
Forward diffusion. Given an articulation graph xo from an object distribution g(xo), we gradually add Gaussian noise with variance schedule $1 < ... < Br and obtain a sequence X1," XT ended with a Gaussian distribution p(xT) .V Nx(x; I). The joint distribution of the forward process is:
((XIT (x0) =
x(x,|xx- 3,x(-1,B11).
(I)

---

A notable property is that at arbitrary timestep t can be directly sampled from Xo with g(x,|x0) -N(ā, (1 aā))
where := 1 Bt and ā₁ := -= a'..
Reverse process. Starting from standard Gaussian distribution x7 (O, I), we aim learn series of Gaussian transitions Po (x() parameterized by a neural network with learnable weight 0 that gradually removes the noise. The distribution of the reverse process is:
(X = p(xT) II 1=1
po (xx (x(xxx po(x(-1(xi) == (No (), Eo(x,,))
Following [10]] we set Eg(x,, 1) = of and model this reverse process with Langevin dynamics
1 0z a ex(xxx1)) o:z, N( v1
X
X
Vou
where c(x,,) t) is a learnable network approximating the per-step noise on X₃-
Training objective. We optimize the variational bound on the negative log-likelihood
Po(Xo:T)
Po (x, (x,)
L = Eg log g(x1T(
Egg
log (ogp(x))
Σ log
El - log pa (xo)
q (x:
(5)
With Eq. g. the objective simplifies to
82
Exx.c
||e colVorx v1 v1 v o)C,
(6)
20 (1 a.)
w.r.t learnable network Eg. We refer the readers to [1OS] for more details.
Output Extraction. The above-described Euclidean space denoising intermediate steps as well the final output may strictly lie the manifold of articulated objects parameterized in Sec. 3 (c.g. the generated joint Plücker coordinates may not be valid because and m may not be orthogonal). We found that utilizing the diffusion in Euclidean space and projecting the generated x back to a valid articulation graph in the final step already leads to practically good results. As mentioned Sec. 6.I. once we have completed the denoising process, post-processing step applied to obtain the final articulated object model from the generated x. First, we identify the existing nodes by the generated nodes indicator o with a threshold and make sure there are least two foreground nodes. Then we use the predicted edge chirality |c| as the edge value to find the minimum spanning tree in the generated graph as the output tree topology. As in Sec. B.I. we model half of the edges in the complete graph with node index < and put the edge direction explicitly to the chirality sign. During the object extraction stage, if c < 0, we flip the predicted joint direction on this edge and if 0 keep the joint direction, which is equivalent to flipping the parent and child order of this edge when < O. Finally, the predicted joint coordinates are projected from R to Plücker coordinates by normalizing the predicted to unit vector and subtracting the parallel component of m on I to make predicted m orthogonal to We decode the part shape code to an SDF and extract the part mesh via marching cubes. We alternatively retrieve the nearest part in the training set as most scene generation methods do [23] 22].
Conditioned Generation. A favorable property of diffusion models is that conditions can be di- rectly added to the inference processes using Bayes' rule without any modification to the training process. Here we perform conditioned generation by fixing the known part of variable as in the im- age inpainting [[0G]] and shape completion works. For a variable x = m& known, + (1-m))x unknown (o means element- wise multiplication) with known entries x known fixed unknown entries unkknown to be completed which are separated by mask m, we can sample the known entries directly following Eq. by adding Gaussian noise the known input and generating the unknown entries using reverse diffusion,
(7) Since method directly diffuses in graph space, apply precise control of the parts and joints condition, enabling, thus, a disentanglement. In Sec. !4 we will show applications with different x known and unknown.
q (xx/nown (xo) (VVxx, (1 a, I)) Pe px xx) = (p (ix,,,)),, t). Es t)).
5

---

Duand
E-V
V-F
Graph Leger
Hand
Thwillah
tepa
Graph
Layer
Hook
E-
upineedar kasure
updae lasure
Farkel
Figure 3: Network architecture. Left: input the node and edge list of noisy articulation graph, stack of graph layers will fuse and exchange information on the graph and output the noise that has to be removed. Right: details in the graph layer.
3.3 Denoising Network
Network architecture. Since x represents both the part geometry and the joint motion constraints, utilize a graph denoising network shown in Fig. B. which exchanges and fuses information between parts and joints. The network inputs a noisy articulate graph x and outputs noise in the same shape x. The input attributes for every node are first encoded by input heads shared across nodes to a list node features {fi). Similarly, edge attributes are encoded to list of edge features (g ). Then the node and edge feature lists updated via the graph layers (Fig. E right). Finally, all hidden features, including the input attributes, are concatenated together and decoded to the outputs via shared output heads. Note that the network is shared across all time steps through denoising, and the time step is encoded and concatenated to the hidden features. Similar to recent work on scene graph generation [22]] we also append the positional encoding of the part instance id to the node features to provide stronger guidance in the early denoising stages when the part pose information ambiguous. To learn with positional encoding, we randomly permute the nodes' order during training.
Graph layer. The key building block of the denoising network is the graph layer shown in Fig. right. The edge feature is first updated via an edge MLP by fusing the input edge feature ((.j) and neighboring nodes i, j'J's features: ((.j) = Mf. ((5))) Then, we aggregate the updated edge features to the nodes by attention weights. We compute the query Q(fi) and key ((fss) from the input node features via two MLPs and use their inner product as the attention weights; the graph attention update the node is: f = Σj= PointNet [Toee global pooling over all the graph node features after the attention aggregation to capture more global information.
(fff))(((())))) (f))))((.5)) We additionally do
4 Experiments
We examine 4 important questions with our experiments: (1) How can we evaluate articulated ob- ject generation? (Sec. I. (2) How well does NAP capture the distribution of articulated objects? (Sec. (.2) (3) How effective each of NAP's components? (Sec. (4) What applications can NAP enable? (Sec. H.4)
4.1
Evaluation Metrics
Since the first study artieulated objects in a generative setting, we propose distance metric between two articulated objects for adopting widely used shape generation metrics. Since a generated object's shape and motion structure are dependent, we can evaluate them separately. Such a challenge is poorly addressed in existing works. In articulated object modeling, existing works either consider fixed kinematic structure [ZT], а or given geometry [[ОС]. In graph generation works [и] structures are examined by themselves without need to measure geom- etry. Thus, we propose new distance metric, Instantiation Distance (ID), measure the distance between two articulated objects considering both the part geometry and the overall motion structure.
We treat articulated object O template that, given the joint states q Qo in object's joint range Qo. it returns the overall articulate mesh M(g) and the list part poses T(g) {Tput E (3)).. We compute the distance between two articulated objects in different joint states by
min d(O1,qi, qı, Oz, 92) n.T (T((a)))
M(() (MM))))))
(8)

---

Table Articualted object synthesis comparison with Instantiation Distance
Part SDF Shape
Part Retrieval Shape
Generative Paradiim//ethhdd
MMD 4
COV f
I-NNA 4
MMD 4
COV f
I-NNA +
Auto-Decoding (StructNet)
0.0435
0.1871
0.8820
0.0390
0.2316
0.8675
Variational auto-Encoding (StructNct)
0.0311
0.3497
0.8085
0.0289
0.3363
0.7918
Autoregressive (ATISS- Tree)
0.0397
0.3808
0.6860
0.0333
0.4120
0.6782
Latent Diffusion (StructNet)
0.0314
0.4365
0.6269
0.0288
0.4477
0,6102
Articulation Graph Diffusion (Ours)
0.0268
0.4944
0,5690
0.0215
0.5234
0,5412
where T, M1((1) means canonicalizing the mesh using its ith part pose, and D is standard dis- lance that measures the distance between two static meshes. Specifically,, we sample -N = 2048 points from two meshes and compute their Chamfer Distance. Intuitively, the above distance mea- sures the minimum distance between two posed articulated objects by trying all possible canonical- ization combinations. Then, we define the instantiation distance between O1 and O2
TD(O1, 02) EU((oo)
inf qnERo2 (á(OI. (1,01,)1,02:99))
+ EM(0
inf m1E
91.11,9.00.90 (2)
where q u(Qo) means uniformly sample joint poses from the joint states range. The instantiation distance measures the two-side expectation minimum distance to the other object over all possible joint configurations. However, the inf inside the expectation requires expensive registration between two articulated objects so it is non-trackable practice when computing all distance pairs between the reference and sampled object sets. In practice, approximate the above distance by uniformly sampling M joint poses Q1
{qx gx E uu(o,,), I,.... M and the approximated distance is:
ID(O1,
1 M
Σ EE
((0,,,1,02,42) 42) R2EQ2
min
+
1 M
Σ E
(i(Or,91,02,1)) BIERI
min
O2,
(10)
and we set M 10 in our ID for all evaluations. This pairwise distance can be plugged into the standard metrics for shape generation. Specifically, we adopt the following three metrics [[T2] for evaluation: minimum matching distance (MMD) that measures the generation quality, cov- erage (COV) that tests the fraction the reference set is covered, and 1-nearest neighbor accuracy (1-NNA) that measures the distance between the two distributions by I-nn classification accuracy.
4.2 Articulated Object Synthesis
Baselines. We adapt existing models in related tasks and compare our method with them. Specif- ically, adapt architecture designs from semantic-part-basee shape generation [[]]] and scene- graph-based indoor scene synthesis [233], and equip them with different generative paradigms in- cluding auto-decoding [[ VAE [[ل]], autoregressive models [23]] and latent diffusion [L [J]. We refer to our supplementary for adaptation and details of these baselines.
Setups. We train all the methods on PartNet-Mobility [39] across all categories jointly, with maximum 8 rigid parts (K = 8) and a train-val-test split ratio [0.7, 0.1, 0.2). Since the dataset does not include parts orientation and all initial part poses have rotation I, we ignore the rotation in the paramcterization (Sec. [.] for all the methods. For fair comparison, all the methods are controlled to similar number of learnable parameters. We evaluate the generated articulated object models with both the reconstructed meshes from the generated shape code and the retrieved part meshes from the training set.
Comparison. We report quantitative results in Tab. II and qualitative comparisons in Fig. EL While the Auto-Decoding baseline fits the training set, its generation performance is poor when sampling the latent space as shown Fig. E-AD, which suggests a weak regularization in the latent space. Using VAE to replace auto-decoding brings about better regularization of the latent space, result- ing in an increase in all metrics. Sampling from the prior of VAE also leads to more meaningful generations, as shown in Fig. VAE. For the autoregressive ATISS-Tres -ree bascline, we see relative
7

---

Auto- Daondinndd
Topalogy
Dost
(AD)
Topoke
Be Vix
SUV Mi
Best Vit
Topako
VAE
SPEV
BeeV
(AR) Antoregressive
Retrieval
Trpaker
BenTe
Latent Diffusion
(LD)
Trpskig
Best
Ours
Figure 4: Articulated object generation results. Each generated object is visualized with (1) graph topology (top left), where the edge color means Allue-prismatic red-revolute, and orange-hybrid; (2) the predicted part bounding boxes and joints under different joint joint states (second column), and the overlay of multiple states reflecting the possible motion (bottom left); (3) reconstructed part meshes from the generated shape code (third column); (4) retrieved part meshes (right column).
increase in a majority of metrics compared to VAE. Part shape retrieval leads to further improve- ment in both motion structure and part shape since using retrieval each autoregressive step can decrease the deviation from the training distribution. Interestingly, as shown in Fig. H. we observe that the autoregressive method also has a tendency append too many nodes the tree, resulting in overlapping parts. Latent diffusion works the best among our baselines. We hypothesize that due the superiority of the diffusion model as a sampler in the latent space, mapping the prior Gaussian to reliable regions where the trained decoder performs well near training samples. How- ever, as the generation happens the latent space and the generated latent code have be decoded, slight error or changes in the latent space may lead to unrealistic or wrong articulations, which is shown in Fig. H-LD. Different from the latent diffusion, our method directly applies diffusion articulation tree space, which can generate diverse and high-quality articulation models and achieves a better performance comparing to baselines.

---

Table Ablation studies with Instantiation Distance
Part SDF Shape
Part Retrieval Shape
Ablation
MMD 4
COV f
I-NNA 4
MMD 4
COV
1-NNA
Full
0.0268
0.4944
0.5690
0.0215
0.5234
0.5412
No PE
0.0282
0.4766
0.5490
0.0227
0.5457
0.5557
No Attm, on Edge
0.0286
0.4766
0,5668
0.0232
0.5234
0,5568
No Graph Conv
0.0331
0.4432
0.6570
0.0281
0.4722
0,6481
4.3 Ablation Studies
We verify our denoising network design by ablating components in our full network, and the compar- ison is shown in Tab. 0 Specifically, we examine the effectiveness of positional encoding. attention weights on edges, and graph convolutions. First, removing the positional encoding will slightly worsen performance, since its presence can help to distinguish the nodes at the early stages of re- verse diffusion [22]]. Second, replacing attention weights with mean pooling when aggregating the neighboring nodes' information results drop in performance. Finally, we justify the importance of node-edge information exchange in the graph convolution by removing the graph convolution layer and replacing it with a PointNet [like global pooling layer, where the connectivity of the graph is ignored and the information exchanges through symmetric global pooling. We observe larger performance decrease, which justifies our graph-based processing.
4.4 Applications with Conditioned Generation
Figure 6: PartNet Imagination: Input over- segmented static PartNet shapes (top left) are grouped into rigid parts (bottom left) and be hallucinated with articulations (right). Both training categories (encir- cled by dashed lines) and out-of-distribution objects are shown.
Figure 5: Part2Motion: Known part con- dition on the left, diverse motion proposals on the right.
Once trained, our method can be used directly for conditioned generation. Following Sec. demo applications conditioning on various known attributes x.
we
Part2Motion. We first show that when knowing the static part attributes, how NAP can suggest motion structures, i.e., with known {r;} and umnknown = {( in Eq. Q. We use the unseen object part attributes from the test set as conditions and the generated motion structure is in Fig. B. We observe diverse and plausible motion suggestions that cover the ambiguity of the closed doors. PartNet Imagination. NAP uses PartNet-Mobility [39] for training, which is only small subset of the large-scale but static PartNet [77]]. We show that NAP can be used imagine possible motion structures in static PartNet [Z5]]. Starting from the finest semantic part labeled in PartNet [[5] (as an over-segimentation), we use simple contrastive learned grouping encoder (see Suppl.) to group the fine semantic parts into rigid parts and then apply the same node condition as in Part2Motion. Fig. E shows our predictions. Interestingly, we find the learned articulation prior can be applied to out-of-distribution object categories and generate intriguing motion structures that adhere to some human recognizable logic (e.g.. g., the mug in Fig. bottom, where the handle rotates around the body). This further suggests that our model captures an underlying articulation prior.
Figure 7: Motion2Part: Кnown motion struc- ture (left) and suggested parts (right).
Figure 8: GAPart2Object: GAParts X (left) com-
pleted into full articulated objects (right).

---

Motion2Part. Similar to Part2Motion, we can flip the conditions and suggest possible parts from the motion structure. Given known edge attributes and graph topology x = =e(x)) U{+,} and
x xunknown
(Tgis by, J. ). NAP can generate possible parts that fit the motion structure in Fig. M GAPart2Object. As Part2Motion and Motion2Part are pure edge node conditions, we can also combine the node and edge conditions to complete an articulated object from one part. GAPart [X]]] numan-labeles semantically generalizable parts and one GAPart has part geometry plus a joint type. We use the GAPart [X]]] from our testset as the known condition. Specifically, we set x known {0o. bo. fol, [o.]] U((oo.1) (0.1)) r(o.))]> otherwise unknown, where we ensure that GAPart is part-O and there must be one part-l connected to it. We constrain the joint direction ((0.1) but free the momentum m and also let the non-zero joint range be freely denoised. The completion results are shown in Fig. N
Conclusions
We introduce the articulated object synthesis problem to community and propose the first deep neural network that is able to generate articulated object models. Such generation is achieved via diffusion model over novel articulation graph parameterization with graph denoising network. We further introduced a new Instantiation Distance to adopt the widely used 3D shape generation metries to this task. Our learned neural 3D articulation can be used under different conditions for diverse application settings. We believe this initial step towards articulated object understanding and generation will bring more opportunities to understand and simulate our dynamic and complex real world.
Long tail distribution
Objects violating our assumption
Physically implausible
Figure 9: Limitations: Left The number of rigid parts in the current dataset has a long tail distribu- tion, leading to extremely challenging objects, for example, a remote or a keyboard. Middle: Object violating tree screw joint assumption. Right: The generated objects do not strictly obey physical constraints, for example, self-collision, making them not directly simulatable.
Limitations and future work. Although we have taken initial steps towards this new challenge, there are still many issues that require further exploration.
Long tail distribution and unbalanced structures: We noticed that our methods and baselines work better for small graphs. Large graphs with many parts are rare in the training set, making the generative model biased towards small graphs.
Overfitting: Like many other generative models, we do observe that NAP slightly overfits the training dataset. One bottleneck is that the current dataset still contains too few objects (around 2k) for training generative models. Studying how to learn an articulation prior from the camera observations or building larger dataset will help to address this issue.
3. Tree and screw assumption: As shown in Fig. M-Mid, NAP currently can not model kinematic cycles (pairs) or model several degrees of freedom on a single joint. Introducing dummy nodes and synchronization between joints may solve this limitation.
More structured diffusion: Although demonstrated to be effective, the diffusion we applied is straightforward and may be improved by introducing discretization, for example, the MST in the intermediate steps or by constraining the diffusion to the Plücker manifold.
Physically plausible generation: As shown in Fig. Right, another exciting direction how to generate directly simulatable objects whose joints and parts geometry fulfill physical constraints. This may require novel intersection loss and optimization techniques between parts and joints.
Broader Impacts. We do not see any direct negative ethical aspects or societal consequences of our work. Our paper broadly improve the perception, digitization, and simulation of our physical world.
10

---

Acknowledgments and Disclosure of Funding
The authors appreciate the support of the following grants: NSF FRR 2220868, NSF TIS-RI 2212433, NSF TRIPODS 1934960, NSF CPS 2038873 awarded to UPenn: TRI University 2.0 grant, ARL grant W911NF-21- 2-0104, and a Vannevar Bush Faculty Fellowship awarded to Stanford University.
References
[1] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2837-2845, 2021.
[2] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-5835 2021.
[3] Michal Tyszkiewicz, Pascal Fua, and Eduard Trulls. Gecco: Geometrically-condtiioneell point diffusion models. arXiv preprint orXiv:2303005916 2023.
[4] Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint arXIv:22/2.00842, 2022.
[5] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint arXiv:2210.06978 2022.
[6] Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02444 024663,,, 2023.
[7] Muheng Li. Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion-sdff Text-to-shape via voxelized diffusion. arXiv preprint arXiv:2212.03293, 2022.
[8] Biao Zhang, Jiapeng Tang, Matthias Niessner, and Peter Wonka. 3dshape2veeset: shape represen- tation for neural fields and generative diffusion models, arXiv preprint arX1v:2301.11445, 2023.
[9] Ka-Hei Hui, Ruihui Li. Jingyu Hu, and Chi- Wing Fu. Neural cavelet-domain diffusion for 3d shape generation. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
[10] Xin-Yang Zheng, Hao Pan, Peng-Shuai Wang, Xin Tong, Yang Liu, and Heung- Yeung Shum. Locally attentional sdf diffusion for controllable 3d shape generation. arXiv preprint (tt.:23350044661/ 2023.
[11] Kaichun Mo. Paul Guerrero, turenet: Hierarchical graph networks for 3d shape generation. ACM Transactions on Graphics (TOG), Siggraph Asia 2019, Articee 242, 2019.
Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas. Struc-
[12] Lin Gao, Jie Yang, Tong Wu, Yu-Jic Yuan, Hongbo Fu, Yu-Kun Lai, and Hao Zhang Sdm-net: Deep generative network for structured deformable mesh. ACM Transactions Graphics (TOG), 1161111 2019.
[13) Rundi Wu, Yixin Zhuang, Kai Xu. Hao Zhang, and Baoquan Chen. Pq-net: A generative part seq2seq network for 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 829-838, 2020.
[14] Guanqi Zhan, Qingnan Fan, Kaichun Mo, Lin Shao, Baoquan Chen, Leonidas J Guibas, Hao Dong, et al. Generative 3d part assembly via dynamic graph learning. Advances in Neural Information Processing Systems, 33:6315-6326, 2020.
[15] Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi-Wing Fu, Neural template: Topology- reconstruction and disentangled generation of 3d meshes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 18572-18582, 2022.
[16] Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry S Davis, Vijay Mahadevan, and Abhinav Shri- vastava. Layouttransformer: Layout generation and completion with self-artention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1004-1014, 2021.
[17) Yichen Li, Kaichun Mo, Yueqi Duan, He Wang. Jiequan Zhang, Lin Shao, Wojciech Matusik, and Leonidas Guibas. letegory-level multi-part multi-joint 3d shape assembly. arXiv preprint wrXiv:2303.06163 2023.

---

[18] Jun Li, Chengjie Niu, and Kai Xu. Learning part generation and assembly for structure-aware shape synthesis. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11362- 11369, 2020.
[19] Zaiwei Zhang, Zhenpei Yang, Chongyang Ma, Linjie Luo, Alexander Huth, Etienne Vouga, and Qixing Huang. Deep generative modeling for scene synthesis via hybrid representations. ACM Transactions on Graphics (TOG), 39(2):1-21, 2020.
[20] Helisa Dhamo, Fabian Manhardt, Nassir Navab, and Federico Tombari. Graph-to-34 3d: End-to-end gener- ation and manipulation of 3d scenes using scene graphs. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16352- 16361, 2021.
[21] Qiuhong Anna Wei, Sijie Ding, Jeong Joon Park, Rahul Sajnani, Adrien Poulenard, Srinath Sridhar, and Leonidas Guibas. Lego-net: Learning regular rearrangements of objects in rooms. arXiv e-prints, pages mrXiv-2301. 2023.
[22] Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Dif- fuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis. arXiv preprint arXiv:2303. 14207, 2023.
[23] Despoina Paschalidou, Amlan Kar. Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Information Process- ing Systems, 34:12013 12200111200 12026, 2021.
[24] Pulak Purkait, Christopher Zach, and Ian Reid. Sg-vae: Scene grammar variational autoencoder to generate new indoor seenes. In Computer Vision-ECC ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIV 16, pages 155-171. Springer, 2020.
[25] Haitao Yang. Zaiwei Zhang, Siming Yan, Haibin Huang, Chongyang Ma, Yi Zheng, Chandrajit Bajaj. and Qixing Huang. Scene synthesis via uncertainty- attribute synchronization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5630-5640, 2021.
[26] Ming-Jia Jia Yang. Yu-Xiao Guo, Bin Zhou, and Xin Tong. Indoor scene generation from a collection of semantic-segmenned depth images. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 15203-15212, 2021.
[27] Yinyu Nie, Angela Dai. Xiaoguang Han, and Matthias Nießner. Learning 3d scene priors with 2d supervision. arXiv preprint arXiv:2211. 14157, 2022.
(28) Xinpeng Wang. Chandan Yeshwanth, and Matthias Nicßner. Sceneformer: Indoor seene generation with transformers. In 2021 International Conference on 3D Vision (3DV), pages 106-115, IEEE, 2021.
[29] Daniel Ritchie, Kai Wang, and Yu-an Lin. Fast and flexible indoor scene synthesis via deep convolu- tional generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6182-6190, 2019.
[30] Kai Wang, Yu-An Lin, Ben Weissmann, Manolis Savva, Angel X Chang, and Daniel Ritchie, Planit: Planning and instantiating indoor scenes with relation graph and spatial prior networks. ACM Transac- tions on Graphics (TOG), 38(4):1-15, 2019.
[31] Morgan Quigley, Brian Gerkey, and William D Smart. Programming Robots with ROS: a practical introduction to the Robot Operating System. " Reilly Media, Inc.", 2015.
[32] Jihoon Kim. Jiseob Kim, and Sungjoon Choi. Flame: Free-form language- based motion synthesis & editing. arXiv preprint arXiv:2209.003: 00349, 2022.
[33] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir. Amit II Bermano, and Daniel Cohen-Or. Human motion diffusion model. arXiv preprint 2209v:2209.14916 2022.
[34] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong. Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208 15001, 2022.
[35] Zhiyuan Ren, Zhihong Pan, Xin Zhou, and Le Kang. Difffusion motion: Generate text-guided 3d human motion by diffuusion model. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[36] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Moll, and Michael J. Black. SMPL: A skinned multi- person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6))248:1-244 October 2015.
12

---

[37] Javier Romero. Dimitrios Tzionas. and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies logether. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), November 2017.
[38] Silvia Zaffi, Angjoo Kanazawa, David Jacobs, and Michael Black. 3D menagerie: Modeling the 3D shape and pose of animals. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), July 2017.
[39] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangehen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part- based interactive environment. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11097-11107, 2020.
[40] Anthony Dearden and Yiannis Demiris. Learning forward models for robots. In IJCAI, volume S, page 1440, 2005.
[41] Jürgen Sturm, Christian Plagemann, and Wolfram Burgard. Adaptive body scheme models for robust rohotic manipulation. Robotics: Science and systems. Zurich, 2008
[42] Jurgen Sturm, Christian Plagemann, Wolfram Burgard. Unsupervised body scheme learning through self-perception. In 2008 IEEE International Conference on Robotics and Automation, pages 3328-3333.
IEEE, 2008.
[43] Jürgen Sturm, Vijay Pradeep, Cyrill Stachniss, Christian Plagemann, Kurt Konolige, and Wolfram Bur- gard. Learning kinematic models for articulated objects. In Twenty-First International Joint Conference on Artificial Intelligence. Citeseer, 2009.
[44] Jürgen Sturm, Cyrill Stachniss, and Wolfram Burgard. A probabilistic framework for learning kinematic models of articulated objects. Journal of Artificial Intelligence Research, 11:4777-26, 2011.
[45] Neil Nie, Samir Yitzhak Gadre, Kiana Ehsani, and Shuran Song. Structure from action: Learning interactions for articulated object 3d structure discovery. arXiv preprint wrXiv22207.08997 2022.
[46] Dov Katz and Oliver Brock Manipulating articulated objects with interactive perception. 2008 IEEE International Conference on Robotics and Automation, pages
IEEE, 2008.
[47] Roberto Martin Martin and Oliver Brock, Online interactive perception articulated objects with multi- level recursive estimation based on task-specific priors. In 2014 IEEF/RSJ International Conference on Intelligent Robots and Systems, pages 1404-2201. IEEE, 2014.
[48] Karol Hausman, Scott Niekum, Sarah Osentoski, and Gaurav S Sukhatme. Active articulation model estimation through interactive perception. In 2015 IEEE International Conference Robotics and Automation (ICRA), pages 3305-3312. IEFF, 2015.
[49] Roberto Martin-Martin, Sebastian Höfer, and Oliver Brock. An integrated approach to visual perception articulated objects. In 2016 IEEE international conference on robotics and automation (ICRA), pages 5091-5097. IEEE, 2016.
[50] Jeannette Bohg. Karol Hausman, Bharath Sankaran, Oliver Brock, Danica Kragic, Stefan Schaal, and Gaurav S Sukhatme. Interactive perception: Leveraging action in perception and perception in action. IEFF Transactions Robotics, 3376))127311291 2017.
[51] Cheng-Chun Hsu, Zhenyu Jiang, and Yuke Zhu. Ditto in the house: Building articulation models of indoor scenes through interactive perception. arXiv preprint arXiv:2302.01295. 2023.
[52] Ruizhen Hu, Wenchao Li, Oliver Van Kaick, Ariel Shamir, Hao Zhang, and Hui Huang. Learning to predict part mobility from a single static snapshot. ACM Transactions on Graphics (TOG), 36(6):1-13, 2017.
[53] Yi, Haibin Huang. Difan Liu, Evangelos Kalogerakis, Hao Su, and Leonidas Guibas, Deep part induction from articulated object pairs. arXiv preprint arXiv:1809.07414 2018.
[54] Ben Abbatematteo, Stefanic Tellex, and George Konidaris, Learning to generalize kinematic models to novel objects. In Proceedings of the 3rd Conference on Robot Learning, 2019.
[55] Xiaolong Li, He Wang, Li Yi, Leonidas J Guibas, A Lynn Abbott, and Shuran Song. Category- articulated object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3706 7715, 2020.
13

---

[56] Qihao Liu, Weichao Qiu, Weiyao Wang, Gregory D Hager, and Alan L Yuille. Nothing but ge- ometric constraints: A model-free method for articulated object pose estimation. arXiv preprint urXiv:2012.00088, 2020.
[57] Yijia Weng, He Wang. Qiang Zhou, Yuzhe Qin, Yueqi Duan, Qingnan Fan, Baoquan Chen, Hao Su, and Leonidas J Guibas. Captra: Category-leys pose tracking for rigid and articulated objects from point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13209- 13218, 2021.
[58] Vicky Zeng, Tabitha Edith Lec, Jacky Liang, and Oliver Kroemer. Visual identification of articulated object parts. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2021.
[59] Yuki Kawana, Yusuke Mukuta, and Tatsuya Harada. Unsupervised pose-aware part decomposition for 3d articulated objects. arXiv preprint arXiv:2 10.04411 2021.
[60] Ajinkya Jain, Rudolf Lioutikov, Caleb Chuck, and Scott Nickum. Screwnet: Category artic- ulation model estimation from depth images using screw theory. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 13670-13600 13677. IEÉE, 2021.
[61] Ajinkya Jain, Stephen Giguere, Rudolf Lioutikov. and Scott Niekum. Distributional depth-based es- timation of object articulation models. In Conference on Robot Learning, pages 1611-1621. PMI.R, 2022.
[62] Xueyi Liu, Ji Zhang, Ruizhen Hu, Haibin Huang, He Wang, and Li Yi. Self-supervieed category-level articulated object posc estimation with part-level se (3) equivariance. arXiv preprint 142668,,, 2023.
[63] Zihao Yan, Ruizhen Hu, Xingguang Yan, Luanmin Chen, Oliver Van Kaick, Hao Zhang, and Hui Huang. Rpm-net: recurrent prediction motion and parts from point cloud. arXiv preprint arXiv:2006. 14865, 2020.
[64] Ben Eisner, Harry Zhang, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. arXiv preprint 2rXhv:2005.04382, 2022.
[65] Xianghao Xu, Yifan Ruan, Srinath Sridhar, and Daniel Ritchie. Unsupervised kinematic motion detec- tion for part-segmented 3d shape collections. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-9, 2022.
[66] Xiaoxia Huang, Ian Walker, and Stan Birchfield. Occlusion-aware reconstruction and manipulation of 3d articulated objects. In 2012 IEEE International Conference on Robotics and Automation, pages 1365-1371. IEEE, 2012.
[67] Aljaz Bozie, Pablo Palafox, Michael Zollhofer, Justus Thies, Angela Dai, and Matthias Nießner. Neural deformation graphs for globally-consisentt non-rigid reconstruction. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1450-1459, 2021.
[68] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ra- manan, William T Freeman, and Ce Liu. Last: Learning articulated shape reconstruction from a monoc- ular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15980- 15989, 2021.
[69] Gengshan Yang. Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2863-2873, 2022.
[70] Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vascomcelos, and Xiaolong Wang. A- sdf: Learning disentangled signed distance functions for articulated shape representation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13001 13011, 2021.
[71] Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 5616-5626, 2022.
[72) Wei- Cei-Cheng Tseng, Hung-Ju Ju Liao, Lin Yen-Chen, and Min Sun. Cla-nerf: Category- articulated neural radiance field. In 2022 International Conference on Robotics and Automation (ICRA), pages 4454-8460 IEEE, 2022.
14

---

[73] Fangyin Wei, Rohan Chabra, Lingni Ma, Christoph Lassner, Michael Zollhöfer. Szymon Rusinkiewicz. Chris Sweeney, Richard Newcombe, and Mira Slavcheva. Self-supervieed neural articulated shape and appearance models. IEEL/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[74] Jiahui Lei and Kostas Daniilidis. Cadex: Learning canonical deformation coordinate space for dynamic surface representation via neural homeomorphism. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[75] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 909-
918, 2019.
[76] Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu. Shape2motion: Joint analysis motion parts and altribules from 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8876-8884, 2019.
[77] Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. Robothor: An open simulation- to-real embodied ai platform. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pages 3 3164-3174, 2020.
[78] Bokui Shen, Fei Xia, Chengshu Li, Roberto Martin-Martín, Linxi Fan, Guanzhi Wang, Claudia Pérez- DArpino, Shyamal Buch, Sanjana Srivastava, Lyne Tchapmi, et al. igibson 1.0: A simulation envi- ronment interactive tasks in large realistic scenes. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7520-7527. IEEE, 2021.
[79] Andrew Szot, Alexander Clegg. Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34:251-266. 2021.
[80] Haoran Geng, Ziming Li, Yiran Geng, Jiayi Chen, Hao Dong, and He Wang. Partmanip: Learning cross-category generalizable part manipulation policy from point cloud observations. arXiv preprint arXiv:2303. 169588 2023.
[81] Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. Gapartnet: Cross-category domain-generalizzie object perception and manipulation via generalizable and action- able parts. arXiv preprint arXiv:221 1.05272, 2022.
[82] Liu Liu, Wenqiang Xu, Haoyuan Fu, Sucheng Qian, Qiaojun Yu, Yang Han, and Cewu Lu. Akb-48: a real-world articulated object knowledge base. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14809-14818, 2022.
[83] Siddhartha Chaudhuri, Daniel Ritchie, Jiajun Wu, Kai Xu, and Hao Zhang. Learning generative models of 3d structures. In Computer Graphics Forum, volume 39, pages 643-666, Wiley Online Library, 2020.
[84] Mohammad Amin Shabani, Sepidehsadat Hosseini, and Yasutaka Furukawa. Housediffusion: Vector floorplan generation via diffusion model with discrete and continuous denoising. arXiv preprint arXiv:2211.12887 2022.
[85] Jason Yim, Brian L Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. Se (3) diffusion model with application to protein backbone generation. arXiv preprint 23i2:2302.02277 2023.
[86] Wenqi Fan, Chengyi Liu, Yunqing Liu, Jiatong Li, Hang Li, Hui Liu, Jiliang Tang. and Qing Li. Gener- ative diffusion models on graphs: Methods and applications. arXiv preprint arXiv:2302.02500 2023.
[87] Zhen Liu, Yao Feng, Michael J Black, Derek Nowrouzezahrai, Liam Paull, Weiyang Liu. Meshdif- fusion: Score-based generative 3d mesh modeling. arXiv preprint arXiv: 2023.
[88] Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209, 14988, 2022.
[89] Hounn Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler. Ming- Yu Liu, and Tsung-Yl Yi Lin. Magic3d: High-resolution text-to-3d 3d content creation. arXiv preprint axXiv2221..10444 2022.
15

---

[90] Miguel Angel Bautista, Pengsheng Guo, Samira Abnar. Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, al. Gaudi: A neural architect for immersive 3d scene generation, Advances in Neural Information Processing Systems, 35:25102-25116. 2022.
[91] Congyue Deng. Chiyu Jiang, Charles R Qi Xinchen Yan. Zhou, Leonidas Guibas, Dragomir Anguelov, et al. Nerdi: Single-view nerf synthesis with language- guided diffusion as general image priors. arXiv preprint arXiv:2212.03266 2022.
[92] Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias NieBner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. arXiv preprint arXiv2233./70115. 2023.
193] Ye Yuan, Jiaming Song, Umar lqbal, Arash Vahdat, and Jan Kautz, Physdiff: Physics-guided human motion diffusion model. arXiv preprint arXiv:2212.025500 2022.
[94] Jonathan Tseng, Rodrigo Castellon, and C. Karen Liu. Edge: Editable dance generation from music.
arXiv preprint S..iv:22111.06544. 2022.
[95] Simon Alexanderson, Rajmund Nagy, Jonas Beskow, and Gustav Eje Henter. Listen, denoise, action! audio-driven motion synthesis with diffusion models. arXiv preprint arXiv:2211009707, 2022.
[96] Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song- Chun Zhu. Diffusion-based generation, optimization, and planning in 3d scenes. arXiv preprint arXiv:2301.000 5, 2023.
[97] Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as a generative prior. arXiv preprint arXiv:2303.01418, 2023.
[98] Sigal Raab, Inbal Leibovitch, Guy Tevet, Moab Arar, Amit H Bermano, and Daniel Cohen-Or. Single motion diffusion. arXiv preprint arXiv:2302.05000 2023.
[99] Michael Janner, Yilun Du Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint 22Xi::2205.09991,, 2022.
[100] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
[101] Weiyu Liu, Tucker Hermans, Sonia Chernova, and Chris Paxton. Structdiffusion: Object-centric diffu- sion for semantic rearrangement of novel objects. arXiv preprint 22 1,06404 2022.
[102] Roberto Martin-Martin, Clemens Eppner. and Oliver Brock. The rbo dataset of articulated objects and interactions. The International Journal of Robotics Research, 1809))00131100 2019.
[103] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[104] Lars Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019.
[105] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion prohabilistic models. Advances Neural Information Processing Systems, 33:6840-6851, 2020.
[106] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion prohabilistic models. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 11461-11471, 2022.
[107] Charles R Qi, Hao Su, Kaichun Mo, Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652 660, 2017.
[108] Gengxin Liu, Qian Sun. Haibin Huang, Chongyang Ma, Yulan Guo, Li Yi, Hui Huang, and Ruizhen Hu. Semi-weakly supervised object kinematic motion prediction. arXiv preprint 2rXiv:2303.1777 177744 2023.
[109] Chenhao Niu, Yang Song, Jiaming Song. Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permu- tation invariant graph generation via score-based based generative modeling. In International Conference on Artificial Intelligence and Statistics, pages 4474 4484. PMLR, 2020.
16

---

[110] Jachyeong Jo, Seul Lee, and Sung Ju Hwang. Scsre-based generative modeling of graphs via the system of stochastic differential equations. In International Conference Machine Learning, pages 10362- 10383. PMLR, 2022.
[111] Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.1477 14734, 2022.
[112] Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Point- flow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2019.
[113] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
17


Reviews:
[{"review_id": "g8StRj2W6D", "reviewer": "Reviewer_zyR9", "summary": "This paper introduces the novel task of articulated 3D object generation. The method devises a parameterization of an articulated 3D object by representing it as a complete graph with nodes corresponding to parts and edges corresponding to joints. The method then trains a diffusion model on this parameter space. Once parameters are generated with the diffusion model, they can be converted back to an articulated 3D object. This paper also proposes a novel metric for evaluation that considers both the geometry and motion of the object. The paper compares to adapted baselines both quantitatively (using the new metric) and qualitatively. Results suggest that this approach outperforms the adapted baselines.", "questions": "Please further clarify the contributions regarding representation parameterization and graph-attention architecture as compared to existing methods. My understanding for the parameterization is that the novelty comes from choosing what features need to be contained in nodes (indicator, pose, bbox, shape code) and edges (chirality [indicator], Plucker, joint limits).", "limitations": "Authors discuss the limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) Novel task. NAP solves the task of articulated 3D object generation. The task is useful for creating data that incorporates motion and this work could inspire future research in this new area.\n\n2) This paper devises a parameterization of 3D articulated objects that can be integrated with a diffusion model.\n\n3) The paper takes a thorough approach to evaluation creating a novel distance metric that considers both geometry and motion. Results outperform adapted baselines both quantitatively and qualitatively.", "weaknesses": "1) This paper could benefit from more clarity on distinctions between contributions from this work versus existing techniques. Specifically, consider adding more detail differentiating the parameterization contributions of this paper that are distinct from prior work as it seems that the parameterization is closely based on URDF [1] for converting the articulated object into the graph representation that can then be used with diffusion. Additionally, further distinction between general graph-attention denoising networks and the contribution of this work’s architecture would be helpful.\n\n2) This method relies heavily on existing techniques such as diffusion models and graph representations of articulated objects. Significant contribution seems to come from the novelty of the task itself and the way in which the paper combines these existing techniques to solve it.\n\nMinor comments:\nFigure 4 showing the qualitative comparisons is slightly confusing. More labels on the image itself would be helpful for clarity.\n\nReferences:\n[1] Morgan Quigley, Brian Gerkey, and William D Smart. Programming Robots with ROS: a practical 407 introduction to the Robot Operating System. \" O’Reilly Media, Inc.\", 2015.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback! Here are our responses to your questions and comments and we hope that they could help to address your concerns:\n\n- We agree that the introduction of a new task and the establishment of a benchmark metric are pivotal contributions of this paper. We also make the first attempt to provide a deep-learning solution for the articulated object synthesis problem:\n\n  - **Parameterization/Representation contribution**: The URDF format, widely adopted for modeling robots and articulated objects, served as an inspiration for our representation. However, it's crucial to understand that our representation is tailored for neural network compatibility – designed to be fed into and extracted from a learned model, ensuring it can seamlessly represent a diverse range of different structured articulated objects in a dataset. While URDF is intuitive, our representation has several differences:\n\n    - (1.) We define the joint and part relationships in a more unified and elegant way: in URDF, one must specify the joint type as revolute or prismatic and define the parent-to-joint and optionally the joint-to-child transformation between two parts, which leads to redundancy (e.g. translating joint coordinate frame along the joint axis won’t change the joint). Instead, we model the joint in the global object coordinate system with screw representations and use per part pose in the global coordinate frame to define the initial relative transformations between the parent and child, which unifies the revolute and prismatic joint and is more stable and cleaner for the network to learn.\n    - (2.) We pad the tree to a complete graph, adding indicators to the complete graph, and use MST postprocessing to extract back the tree from the network output. With all these careful designs of representation that do not come with URDF, we enable the processing of deep networks on this highly irregular collection of different articulated objects.\n\n     Compared to existing methods of modeling articulated objects, they either model a fixed/simple structure of articulation, a fixed number of parts (usually two), or implicitly plug the joint states into the latent code instead of explicitly modeling the structure. Our representation, inspired by URDF and with the above-mentioned difference to URDF, is designed for neural processing and is an explicit and holistic description of various structured objects. In short, the representation contribution is a complete description of various articulated objects across the dataset that is oriented to the neural network.\n\n  - **Deinosing network distinction**: Indeed, there are many graph-based networks for denoising and some diffusion models for generating large and general graphs. We leverage the general attention mechanism on graphs and tailor it to our unique task. In our problem, we pay more attention to the node and edge attributes and the key observation is that the information exchange between edge and nodes is important (joints and parts talk). Thus we utilize a graph attention network as in Sec.3.3 that explicitly fuses information of both node and edge. We also hybridize a pointnet-like global pooling over all node features to gain more global information. While we don't highlight this network as our primary contribution, its reasonable design is rooted in our deep understanding of the task at hand.\n\n  In summary, our work is not just a combination of existing methods: besides the significant contribution of our novel task, we also contribute a novel representation that is tailored to neural network diffusion and design a denoising network with our insight into this task. We hope our first attempt toward this brand-new direction may inspire future works to propose better solutions.\n\n- **Figure suggestion**: We appreciate the constructive feedback. To enhance clarity, we will incorporate labels into the sub-images in our revised version.\n\n"}, {"review_id": "5fsPukf43k", "reviewer": "Reviewer_pAau", "summary": "The paper presents the first generative model over articulated shapes. The paper presents several contributions: (1) a parameterization of articulated shapes that are easy to be used with neural networks, (2) design of denoising diffusion architectures that are structured and can effectively denoise the shapes in the introduced parameterization, and (3) extensive evaluations using novel and meaningful metrics. ", "questions": "Listed in weaknesses. ", "limitations": "Limitations are adequately discussed. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper presents the first such generative model of articulated shape, and demonstrates excellent results. The extensive baseline comparisons support the claims. While latent diffusion comes close in performance, I believe the structured models introduced in this paper would become even more relevant with more complex shapes. \n\nThe ideas presented are original and are very well explained. I appreciated the supplemental video visualizations. ", "weaknesses": "- The model is trained on a dataset of around 2k shapes. I wonder if there is significant overfitting. Visualizing the nearest training shape from the generated samples would be helpful.\n- The forward process adds noise to discrete indicator variables (both for nodes and edges). How is the binning done during sampling, i.e., from a floating point value, how is the indicator value computed? \n- Adding noise to the introduced parameterization leads to intermediate noisy states that do not correspond to any valid 3D shape. Is that true? If not, it would be great to add a visualization of the diffusion process. \n- The baseline details are not all obvious. The technical exposition is at a high level. Without publicly available code, these evaluations would not be reproducible. \n-  How is the random sampling over joint poses done for implementing the ID metric? Is the distribution over poses known for each shape?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback! Here are our responses to your questions and comments and we hope that they could help to address your concerns:\n\n- **Overfitting**: Please see [G2] in our global response block.\n- **Indicator variables**: We apologize for any confusion caused. As detailed in Suppl. L39, we initially determine node existence by thresholding the node indicator o at 0.5. For all identified existing nodes (which form a sub-complete graph), we utilize the edge indicator/chirality c value, specifically -|c|, to construct a Minimum Spanning Tree (MST). The binarization process occurs within the MST during the selection of tree edges. We will clarify this procedure in our revised manuscript.\n - **Diffusion process**: Yes, when adding noise, for example, to the plucker coordinates, the intermediate steps may not correspond to valid joint parameters anymore. While we currently treat the parameterization as if it exists in continuous Euclidean space and employ standard Euclidean diffusion (which has proven effective), we concur that leveraging advanced manifold diffusion techniques (like Grassmannian for Plucker) or discrete diffusion methods for indicators could enhance the generation process. This point is highlighted in our limitations section (L316). We also visualize the intermediate steps by projecting the noisy parameterization back to the nearest valid parameterization, the bottom three small figures in Fig.2 and the small animation in our video starting from 2:51 are these visualizations. We will enhance the clarity of these visualizations in our revised version.\n- **Baselines**: We provide more details of the baselines in Suppl. L53 Sec.S1.2, and we will release the code once the paper is published. Hope these baselines would also help the new area.\n- **ID Pose sampling**: As outlined in L222, each joint possesses working ranges (predicted for generated objects and sourced from ground truth for references) for both prismatic and revolute modes. We uniformly sample within these ranges to obtain the sampled joint states, implying that the distribution is simply a uniform distribution, parameterized by the range's endpoints.  We will make this more clear in our revision.\n"}, {"review_id": "ke7Fo06MJf", "reviewer": "Reviewer_ncuk", "summary": "This paper proposes a diffusion model based 3D generative model for articulated objects. The major contribution is the 1) tree representation of the articulated objects, 2) the corresponding graph-based diffusion models, and 3) a distance metric for evaluation. The proposed framework works well on PartNet-Mobility objects, beating baselines based on other generative models, and enables a series of conditional generation applications. ", "questions": "There are some unclear writings which can be clarified:\n- L109: is the initial pose $T_{gi}$ a node property? This is different to the statement in L106 \"Every joint (edges) has an initial pose\"\n- L112: why a 3D bbox $b_i$ is in $\\mathbb{R}^3$? Three numbers represents a point in 3D space.\n- L115: $o_i$ is a **per-part** binary indicator of part existence. If there are K parts, I think $o_i$ should be in dimension {0,1}$^K$?\n- L116: $f_i$ is a new notation and not introduced before. I assume this is a typo, which should be $s_i$?\n- L121: the domain of $l$ is $\\mathbb{S}^2$: what is $\\mathbb{S}$?\n- L124: \"... to avoid local coordinate changes caused by parent-child order flips\" -- > when will parent-child order flips happen? How bad is this problem in practice?\n- L125: for two ranges $r$ domain $\\mathbb{R}^{2\\times2}$: can the lower range be negative values?\n- L128: joint states $(\\theta, d)$ is not defined. What are the meanings of these two variable?\n- L128: the relative transformations $T(\\theta, d)$ is between two parts, let's say i and j. Then is the joint state $(\\theta, d)$ of i or j? \n- L129: global joint axis $(l_g, m_g)$ is undefined before using. How are this global axises defined?\n- L129: $R_{ig}$ and $t_{ig}$ are undefined before using. The authors might want to state in L109 that $T_{gi} = [R_{gi};t_{gi}]$. In addition, the subscripts are not consistent.\n- L136: based on the description in L121, $l\\in\\mathbb{S}^2$ and $m\\in\\mathbb{R}^3$. Then $p_{i,j}$ should be of dimension 5 not 6?\n- L136: $c$ is a discrete value and hence not in $\\mathbb{R}$.\n- L146: I find this domain definition strange. v and e are different variable and you cannot simply adding their dimension as the graph dimension. \n- L155/Eq.4: $\\theta$ is already defined in Eq1. Please change variable name. \n- Occupancy Networks predicts occupancy score not SDF value. In many places of this work, SDF is mentioned as OccNet output. Just want to clarify, which one is used?\n\nOther more conceptual questions:\n- Following my point regarding memory consumption in weakness section, what's the average graph size and the corresponding memory usage and training time for current framework? What's the biggest graph size the current solution can afford on a standard GPU during inference time and what's the corresponding memory usage? The illustrated graphs in this work are all relatively easy with few nodes and edges (L240 maximum 8 rigid parts). I saw that the limitation section first point mentioned this, but please be more detailed. \n- It's unclear to me why doing a graph diffusion on a complete graph is better than doing in latent space. What advantages do we gain for using this graph diffusion process? On the other hand, the disadvantages are obvious: high memory consumption and tricky graph-network design.\n", "limitations": "The authors discussed the limitations in L311-318. I agree with these points in general but I do think one important point is missing. The approach is based on two assumptions: Tree assumption and Screw joints. The limitation of these two assumptions should be mentioned. For what kind of objects will they break?\n\nI agree that the potential negative societal impact isn't an concern for this work as mentioned in L325. ", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The motivation is clear. To catch up the latest trends and use the diffusion models, the tree/graph parameterization is developed and implemented in a neat way. \n- The proposed solution works well on the tested dataset and outperforms other generative baselines including a latent diffusion model.\n- The downstream applications are interesting and demonstrates one important advantage of this framework: conditional generation is easy.\n- The graph attention denoiser is reasonable and serves well in the entire framework. Usually graph-based network doesn't generalize very well. \n- The proposed distance metric makes lots of sense. ", "weaknesses": "### Major: (my current rating is based on my concerns as listed below)\n- Doing diffusion process on a complete graph is computationally heavy and slow. I'm not sure how well this method scale. The illustrated graph are all relatively easy with few nodes and edges. Please see below section for my detailed question regarding this point.\n- The writing is unclear and I quickly lost in the writing, especially in the approach section. Some of the issues can still be figured out through context, but others really hurts the readability (eg., I had a hard time understanding the edges section). Please see my detailed comments in Question section. I try to list as many as I can but probably still miss a lot.\n- In many places, this paper claims they are \"the first 3D deep generative model to synthesize 3D articulated objects\" or \"introducing the articulated object synthesis problem\". This is a bit over-claimed since 3D articulated object is a very broad domain, which also includes animals, human, and many other things. There is big literature on 3D human/animal generation in computer vision field. Meanwhile, the motion patterns and the graph-scale studied in this work probably only works for robots and simple objects. Therefore, it's better to tone down a bit.\n- Is PartNet-Mobility the only dataset can be used? The objects in this dataset are relatively simple in geometry and motion. Is it possible to generalize to more diverse and realistic dataset containing more articulated objects like animals/human or more challenging objects?\n\n### Misc:\n- In the official Formatting Instructions Line101-102, it stats that \"The table number and title always appear before the table\". The submitted paper doesn't follow this.\n- Approaches [3.1]: Why using Plücker coordinate to represent joints? Is this the only option? If not, what's the advantage of using this over the other alternatives? The motivation is a bit unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback! Here are our responses to your questions and comments and we hope that they could help to address your concerns:\n\n- **Graph size and resource**:\n\n  (1.) **Dataset property**: PartNet-Mobility(Sapien) stands as one of the largest and most widely used and compared datasets for everyday articulated object modeling. We provide statistics on the number of rigid parts from this dataset in the attached Fig.R4, where most objects have less than 8 rigid parts and the distribution has a long tail. Some extreme examples are the remote controller and the keyboard. We hope future work can address this challenge as in L313.\n\n  (2.) **Larger K and resource consumption**:  As detailed in Suppl. Tab.S2, scaling NAP’s max number of nodes from 8 to 15 and 20 still yields effective results (note that 20 nodes almost cover most objects in the dataset). Because the parameterization is compact (different from image generation) NAP can be trained extremely fast (L26 in Supp): when the max number of nodes K=8, the training batch size=64, and supervise 10 diffusion steps in a forward, we only have to consume 8465MB vRAM. The training can be done in 9 hours on one RTX3090 GPU. For inference: when set K=20, generating 10 objects in parallel on an RTX3080-Laptop GPU only takes 20 seconds with 2545MB vRAM cost.\n- **Articulated objects claim**: Yes, there is a huge literature on humans and animals, but this paper aims to focus on synthesizing, specifically, the daily articulated objects. We can change our title from “Neural 3D Articulation Prior” to “Neural 3D Articulated Object Prior” and explicitly emphasize the scope precisely in the paper. While humans and animals usually have some template parametric models, it’s very interesting to study how to generate these deformable objects. We will update our related work section with more literature about humans and animals in our revision.\n- **Why is latent diffusion worse**? As in L259, latent diffusion first generates a latent code of the object. However, since the decoder is learned, maybe a slight error made by the generated code in latent space will lead to the very wrong decoded structure in the 3D space. On the contrary, NAP directly diffuses the whole graph and its attributes in the 3D space so the errors will be controlled better from the loss. Another advantage is that all the convenience and flexibility of conditional generation come from this explicit graph space diffusion.\n- **Assumptions as limitations**: Thanks, we will append a new item to the limitation. As in the attached Fig.R5, two examples violating the assumptions are shown. The left object has a chain in the middle, which can be converted to a tree with one dummy node plus motion synchronization; the Airbus handle on the right violates the screw assumption where the joint can have two revolute DoF, which can also be resolved by adding a dummy node for the second DoF. We leave for future works to study these more complex kinematics.\n- **Why Plucker?** Plucker is not the only way to represent joints but an elegant and simple way. As studied in [56], such representation unifies the revolute and prismatic joints and avoids the redundancy caused by defining additional joint coordinate systems (e.g. moving the joint frame along the joint axis still represents the same axis).\n- **Clarification, writing, and format**: Thanks for all these careful notes and checks! We take all these seriously. Please see also the global response [G1]:\n  - *L109:* Yes $T_{gi}$ is a node property, which is computed from the joint initial state ($\\theta=0,d=0$).\n  - *L112:* The $\\mathbb R^3$ means the concatenation of three real numbers, representing the lengths of the bbox.\n  - *L115:* The subscript $i$ for $o_i$ means for one node, which either exists or does not, but if we concatenate all nodes together then $o$ should be in {0,1}$^K$.\n  - *L116:* Yes, sorry for this typo, $f_i$ should be $s_i$\n  - *L121:* The $\\mathbb S^2$ means a unit sphere, which means that $l$ should be a unit-length vector.\n  - *L124:* L124 is to motivate defining the joint parameters in the global frame. For the same joint, its parameters written in the child and the parent frames are quite different (since the axis line is located differently). If defined locally in either the child or parent frame, when the parent and child node order flips, the edge has to change significantly, which may be harmful to the later learning process.\n  - *L125:* Yes, it can be negative\n  - *L128-1:* $\\theta$ means the revolute joint angle and $d$ is for prismatic displacement.\n  - *L128-2:* Sorry for this confusion, we realized that the text definition of $T(\\theta, d)$ is not precise. Let’s say i is the parent part and j is the child part. $T(\\theta, d)$ is the motion of the child part from its initial rest state caused by joint states $(\\theta, d)$ expressed in the parent frame. So $(\\theta, d)$ can be regarded as defined in the reference of parent i. This is illustrated in the attached Fig.R8 (right) and we will append a detailed text explanation to our revision.\n  - *L129-1:* Here the subscript means: written in which coordinate frame. $(l_g, m_g)$ is the plucker coordinate of the joint written in the global object frame.\n  - *L129-2:* Yes, $T_{gi}=[R_{gi};t_{gi}]$, we’ll clarify our subscripts convention as in L110 in our revision.\n  - *L136-1:* We represent the unit vector $l$ with 3 numbers, and $p_{i,j}$ has a dimension of 6.\n  - *L136-2:* Will update this. But note that during prediction, the network still predicts a continuous scalar.\n  - *L146:* To better link to the diffusion formula (all stuff as one $x$ in later equations), we can change the $+$ to, for example, $\\oplus$.\n  - *L155/Eq.4:* We will change the notation.\n  - *Occupancy or SDF?:* The SDF is used for prediction and supervision, but the network architecture is an OccNet encoder-decoder.\n"}, {"review_id": "eyvykON5MG", "reviewer": "Reviewer_17WX", "summary": "This paper proposes Neural 3D Articulation Prior (NAP) to synthesize 3D articulated object models. The key contributions include (a) an articulation tree parameterization for the diffusion denoising probabilistic model and (b) a new distance function for evaluation. The paper also shows quantitative and qualitative improvements over prior methods.", "questions": "One suggestion is to improve Fig. 3. Currently, it seems too packed and the text is too small to be read.", "limitations": "The authors have adequately addressed the limitations and broader impacts.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper introduces a new problem of articulated object synthesis. While there are some prior works for (mostly) category-specific articulated object modeling, this paper focuses on generative modeling across different categories.\n\n2. There are a few key innovative components in the proposed methods such as the articulation tree parameterization and graph-attention denoising network. Experiments (Tab. 2) have also shown the effectiveness of these new components.\n\n3. This paper has also proposed new metrics (distance) to evaluate this new task, which may benefit future works for a fair and efficient comparison.\n\n4. Implementation details are well-documented.", "weaknesses": "1. A neural implicit surface is used to fit each part. From figures such as Fig. 4, it seems the geometry of the proposed method is not very high-quality. However, it seems the geometry of single parts is simpler than the geometry of the entire shape, and prior work (eg, [100, 110]) has shown much better geometric results even for an entire object. Could the authors explain why the synthesized neural part surface looks worse than synthesized neural single-object surface from prior work?\n\n2. The entire pipeline and the key components all seem very valuable to the community. To help us better understand what is easier for the network to learn, could the authors share some examples of failure cases / more challenging types or categories?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback! Here are our responses to your questions and comments and we hope that they could help to address your concerns:\n\n- **Part shape quality**: We appreciate this insightful observation. Here are some potential explanations:\n\n  (1.) **Focus on articulated object generation**: Many scene synthesis methods, similar to ours, don't primarily focus on individual object shapes but rather retrieve them from a database. As our main emphasis is on generating structured objects instead of single-part shapes, we take the most naive way to model the part shape, an Auto-Encoder, even with NO regularization in the latent space. Here we do an additional experiment: when learning the pre-trained part shape module, we add regularization to the latent space (VAE loss) and reduce the latent dim from 128 to 64, aiming to cultivate a more meaningful part shape latent space. After learning NAP on this updated part shape prior, we do observe the part quality is improved a little, and the quantitative results evaluated with the predicted part SDF geometry improve as shown in the table:\n  \n    |                            | MMD ↓      | COV ↑      | 1-NNA ↓    |\n    |----------------------------|------------|------------|------------|\n    | Paper                      |     0.0268 |     0.4944 |     0.5690 |\n    | With VAE regularized prior | **0.0229** | **0.5167** | **0.5490** |\n\n  We believe that leveraging advanced part representations, like tri-planes or local feature grids, can further enhance reconstruction quality. We will highlight this in our limitation section. \n\n  (2.) **Thin structures**: Unlike shapes in ShapeNet, the rigid parts of an object usually exhibit very thin structures. In the PartNet(Mobility), a lot of parts are non-watertight and have large areas of single side surface: for example, a door may be just one plane, with zero thickness. To learn an SDF, we apply heavy pre-processing on these ill-defined thin structures. We know that vanilla DeepSDF or OccNet struggle with such thin structures (e.g. the airplane wings), which might be the biggest reason for the quality drop. Potentially, more expressive local methods or unsigned distance functions may help to solve these problems.\n\n  (3.) **Part shape variance**: We are modeling rigid parts, whose shape variance may not be as small as we first thought. For example, the network has to learn very simple shapes – a door, a round button, or a drawer that may appear repeatedly in the dataset, and simultaneously learn relatively complex shapes – a whole chair seat back plus arms and the bottom, or the whole body plus base of the storage furniture that are relatively rare in the dataset. What’s worse, as the rigid part in our dataset has no semantic labels or the concept of categories, it’s not easy to balance the part samples during training between these simple and hard shapes. As a result, the network may be biased to small and simple repeating parts but have difficulty capturing more complex ones. This issue may be resolved by a smarter training strategy or network design for the part shape prior.\n\n  (4.) **Prediction**: as the shape latent code is predicted from the denoising, we can not guarantee that the accuracy in the latent space is perfect. As small changes in the latent space may lead to un-plausible changes in the output mesh, suggesting room for improvement in this task.\n\n- **Failure cases and more challenging types or categories**: We've included visualizations of failure cases in the attached Fig.R7. Fig.R5 showcases examples that currently violate our assumptions, which might be addressed with dummy nodes. Fig.R4 highlights the long-tail distribution of articulated objects in our dataset, with items like remotes and keyboards being particularly challenging. We'll incorporate these insights into our revision.\n- **Figure suggestion**: Thanks for this suggestion. Due to page constraints, the figure was downsized. We'll restructure the figure layout in our revision for better clarity.\n"}, {"review_id": "9IlhC3zOae", "reviewer": "Reviewer_hNRt", "summary": "The paper presents the task of generating articulated objects, encompassing the generation of both structurally and geometrically plausible objects. To address this task, the authors propose a novel articulation \"complete-graph\" parameterization. This parameterization encodes the geometry and part poses within the nodes, while representing the joint constraints through the edges. To implement the generation process, the authors employ a diffusion model with their designed graph fusion module. The paper also introduces a novel distance metric for evaluating the distribution of the generated articulated objects. Additionally, the paper delves into various applications leveraging conditioned generation.", "questions": "* For the evaluation, the part geometry uses the retrieved ones or directly the results from the occupancy network? Because from the rendered results used for evaluation in the supplemental material, it seems that the part geometry is more likely the retrieval results.\n* For the conditioned generation, it’s easier to compare the results with some existing work. For example, for the part2motion, is it possible to quantitatively evaluate the generated results? How many generated models will cover the GT motion? And based on the results from Part2Motion, it seems that the handle hints in the part are not fully understood by the network. Are there some results to show that the NAP can really understand the joint constraints based on the part geometry?\n* For the generated structures, is there some statistics on the number of nodes and number of edges of the generated set and the train/test set?\n* For the parameterization of the articulated objects, the order of the node seems also important to be somehow consistent. Are there some operations for the node order when parameterzing the articulated objects (e.g. part from top to down)", "limitations": "The authors mention the limitation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed task to generate articulated objects is interesting and the motivation to learn the distribution of the articulated objects is intuitive.\n* The “complete graph” representation for the articulated objects is effective to decompose the geometry and structures into the nodes and edges of the graph. Through setting a max number of nodes K, the parameterization can successfully encode most of the articulated objects into a consistent complete graph representation for the following generation process.\n* The graph layer to fuse and update the information in the node and edges is helpful based on the results of the ablation studies.", "weaknesses": "* For a generation task, it’s hard to evaluate the novelty of the generated objects with existing evaluation metric mentioned in the paper or the user study to compare the results from different baselines. The dataset used in this paper is PartNet-Mobility only contains 2346 objects, after filtering objects with more than 8 parts, there are about 2000 objects. Based on the split mentioned in the paper, there will only be about 1400 articulated objects in the train set used to train the diffusion model. It’s highly easy for the diffusion model to overfit on such a small set of articulated objects. Some potential way to evaluate the novelty and uniqueness of the generated model is to fetch the most similar objects in the train set and fetch the most similar objects in the generated set to qualitatively see the difference. Current evaluation and qualitative results cannot show if the generated models are a copy of the models in the train set.\n* In this paper, actually the task simplify the general articulated object generation task, which only generate articulated objects whose all parts are in the rest state. The authors also assume the initial state of the articulated models in the PartNet-Mobility dataset is close state, which doesn’t hold true for all models in the dataset. For example, some cabinets in the PartNet-Mobility dataset have some door open, while some are close as the initial state.\n* The generation task of the articulated objects is demonstrated to focus on both geometry and the motion structure. However, for the part geometry, the authors choose to retrieve the most similar parts in the latent space for some visualization and potentially for the evaluation. In this way, actually the generative task focuses on generate the combinations of the parts instead of really generate the articulated models.\n* The direction of the edge is explained in the parameterization, however, not used when constructing the minimum-spanning tree. It’s unclear how the edge chirality (+1, -1) are used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback! Here are our responses to your questions and comments and we hope that they could help to address your concerns:\n\n- **Overfitting?**: Please see response G2 in the global block.\n\n- **Rest state**: We appreciate the reviewer’s careful observation of PartNet-Mobility. We will explain this more clearly in Sec 3.1:\n\n  (1.) NAP does not require a strict restriction of what is “rest”: NAP chooses to define the parts and joints in a global object coordinate system for simplicity and any well-defined articulated object can be instantiated and represented in such an object frame. The object articulation pose does not have to be what human thinks is “rest”.\n\n  (2.) A relatively “canonical” rest state is a reasonable situation in the current existing datasets, from which NAP can take advantage: As articulated object annotation is expensive, most of the objects have some consistent state presented when labeling the dataset. For example, PartNet-Mobility is mostly from PartNet which is eventually from ShapeNet. No matter whether the door is closed or open, there does relatively exist global consistency in the dataset that forms relative canonical states (even if half of the doors are open and half of the doors are closed, there is a two-mode canonical state), from which NAP can take advantage and learn a more canonical and stable prior.\n\n- **Retrieval or reconstruction?** We will emphasize this more obviously in our revision.\n\n  (1.) We evaluate with both the implicit part reconstruction and the retrieval in Tab.1 and Tab.2 (left is recon and right is retrieval). Fig.4 (the third column of each object viz group) in our main paper also visualizes both cases.\n\n  (2.) Our primary emphasis is on the generation of structured objects. Like many scene synthesis methods [19-30], we retrieve the nearest part shape from the training set to enhance quality. We anticipate that future works can further refine the quality of individual parts using advanced shape-generation techniques.\n\n- **Chirality:** Sorry for this confusion, and we will provide a clearer explanation in our revision:\n\n  (1.) In L132, we highlight the simplicity of global plucker coordinates (against defining in local parent or child coordinates), i.e. $p_{(i,j)} = -p_{(j,i)}$ and $r_{(i,j)}=r_{(j,i)}$, which motivate us to only represent compactly $K(K-1)/2$ edges as in L133. But note that the nodes have no specific order (randomly permuted) so that the edge between node i and j in these $K(K-1)/2$ may sometimes has direction $(i,j)$ but sometimes $(j,i)$. For a more elegant representation, we explicitly model the negative sign of $p_{(i,j)} = -p_{(j,i)}$ into chirality as $+1,-1$ and let the representation always have one consistent plucker $p$ for the edge between node i and j ignoring the edge direction.\n\n  (2.) In practice, after applying the MST based on negative chirality $-|c_{(i,j)}|$, we determine the direction of each existing edge: if $c_{(i,j)}>0$ we do nothing; if $c_{(i,j)}<0$ we multiply the predicted plucker coordinates with $-1$, which is equivalent to claiming that this predicted joint parameter is actually with direction $(j,i)$ instead of $(i,j)$.\n\n- **Part2Motion**: Our primary focus in this first step towards deep articulated object generation is unconditional synthesis, and the applications further demonstrate our prior is useful and flexible. \n\n  (1.) As requested, in Part2Motion, we count how frequently the ground truth joint axes are covered by the generated objects (within 5 degrees, 0.05 distance, 20 generations per GT object) and the results are reported in Fig.R2. On the testing set, we observe an average of 38.64% probability that the ground truth joint will be covered by the conditional generated joints. As these application tasks are quite new and their evaluation and task definition are non-trivial, we leave them for future work in the area to further explore.\n\n  (2.) Indeed, we found that the small handle hints are not fully parsed in Fig.5 in our main paper: the generated motion includes joints near the handle side. This is a limitation of our current method. One potential reason is that the simple pretrained shape Encoder-Decoder (PointNet+OccNet) is not detailed enough and the learned part shape latent space is not sensitive enough to these small geometry details. We hope this would be improved by future approaches and we will append this to our limitation, thanks!\n\n- **Statistics on Nodes and Edges**: These are usually reported when studying the generation of general graphs[83]. We report the statistics of the number of nodes and node degrees (reflecting some structure) in attached Fig.R3. Note that since the generation always produces a tree, there is no need to count the number of edges. We observe that NAP can generate close distribution to the training and test set.\n- **Order of nodes**: We don’t have any specific spatial order of the node in the parameterization. However, in order to make the network more expressive, especially during the early denoising stages, we do add the positional encoding of the order (the #th position in the list) to the network (inspired by [22]), and during training, we randomly permute the node order in the graph.\n"}, {"review_id": "Nyosm7wtvp", "reviewer": "Reviewer_WyGQ", "summary": "This paper introduces a diffusion-based generative approach targeting daily articulated objects as a novel target category. The model outputs part shapes and joint configurations based on the proposed graph representation. The paper also proposes a novel transformer network to accommodate the graph structure in the generative process. The proposed method is evaluated on the Part-Mobility dataset using the newly proposed evaluation metric for the task, and it demonstrates superior performance over self-made baselines of non-diffusion-based and diffusion-based generative model architectures. The paper also demonstrates several applications of conditional generation on seen and unseen synthetic datasets.", "questions": "# Questions\n* How is either prismatic or revolute enforced? In L125, how is a node’s joint type decided as prismatic, revolute, or hybrid using r(i,j)? Is there some threshold?\n* Why not use an indicator variable to determine either prismatic or revolute?\n* In S.1.1, how do you reflect node existence o in MST over chirality? Do you make the chirality of the edges having the node zero?\n\n# Suggestions\nI suggest adding the following visualizations for a better understanding of the paper:\n* Qualitative visualization of the ablation\n* Visualization of failure cases\n* Schematic visualization of the variables described in Sec. 3.1.", "limitations": "Limitations are explained in the conclusion of the main paper. However, visualizations of some of the limitations and failure cases are missing, such as physically implausible generation. Adding those visualizations would help further understand the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "* The paper tackles the previously unaddressed setting of generating daily articulated objects and the proposed approach seems promising.\n* The approach demonstrates superior generative performance over the baselines both qualitatively and quantitatively.\n* The paper demonstrates novel conditional generative applications and demonstrates generalizability to unseen datasets.", "weaknesses": "# Major\nAlthough I found this work exciting and potentially interesting to the related audience, I feel the current writing quality is not sufficient for a conference paper.\n\n* Inconsistent/confusing/unexplained notations\n  * L111,116: The bold font of T_{gi} is inconsistent.\n  * L118,216: The italic font of SE is inconsistent.\n  * L113,116, and supp. L9: The implicit shape latent code s_i is written as f_i or f, which is used as a node feature in L194.\n  * In L109, L129: The transformation from local to global coordinate is implicitly expressed as switched subscript: _{ig} and_{gi}. But it’s hard to understand at first glance.\n  * L109,217: \"i-th\" and \"ith\" are inconsistent.\n  * L215: The definition of T_{part} is missing.\n* Missing reference to supp.: The implementation details of the Graph layer are missing in the supp., although it’s referred to in L199. How global pooling is applied is unclear.\n\n# Minor\nThe training detail of the pre-trained shape prior network is missing in the supp.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback! Here are our responses to your questions and comments and we hope that they could help to address your concerns:\n\n- **Writting and schematic visualization**: We really appreciate these careful checks and suggestions and we take all these seriously. Please see [G1] in the global response as well. As suggested, we append a draft of the schematic visualization in the attached Fig.R8.\n\n  In L215, $T_{part}$ represents the part pose in the global object frame. We will correct all the typos, inconsistent notations, and confusing subscripts as mentioned, and complete the missing details in our revision.\n\n- **Graph layer and shape prior training details**: We apologize that some graph layer details are missing in Suppl. Sec.S.1: The global pooling in the graph layer is similar to a point-net – doing a max pooling over all node features and concatenating this global pooled feature back to each node again. \n\n  Regarding the shape prior training, besides L8 in Suppl. here are more details: the optimizer is a standard Adam with lr=0.0001 and the learning rate has a step decay at [100000, 150000, 200000] iterations with factor 0.3. The batch size is 32 and we use the model checkpoint at epoch 737. We will add all these details to the supplementary, and the code that contains all implementation details will also be released once the paper is published.\n\n- **Joint type**:  Leveraging the screw representation [56], as in L126, if a joint is revolute, its ground truth prismatic working range will be [0,0] and vice versa. This is softly enforced by the loss that learns to predict [0,0] through denoising. To decide the joint type for visualization purposes (in Fig.4 left top corner), yes, there is a threshold to test whether the working range is large enough so the corresponding mode really exists. We use th=0.003 in the paper visualization for both prismatic and revolute modes. “Why not use an indicator for these?” We agree that this is a valid design but it would potentially add two additional variables that need to be predicted, so we directly decide the joint type from the joint range prediction, fully exploiting the simplicity in the screw representation.\n- **Existence of nodes:** Sorry for this confusion. There are two “indicators” in our representations, The variable o serves as the node existence indicator, while c indicates edge existence. During the final output extraction (as described in L163 and Suppl. L39), we first identify node existence by thresholding the node indicator o at 0.5. All existing nodes at this point form a complete graph (a sub-graph of the original padded one). We then utilize the chirality absolute value −∣c∣ of each edge to perform the MST on this sub-complete graph, determining which edges exist.\n- **Visualization of ablation and failure cases:** We append the visualization of ablation in the attached Fig.R6 and of failure cases in Fig.R7. Note that in Fig.R7 highlighted area, the drawer is not physically plausible, which means it can not be simulated with self-collision in a physical simulator, we hope future work will tackle this problem. We’ll append these figures to our revision."}]

Author rebuttal:
We thank the reviewers for their constructive comments and are glad that they all have a consensus on our contributions/novelty:

1. **Contribution and novelty of studying a new problem**
2. **Contribution of new evaluation metrics to this novel task**
3. **Technical contribution of the first deep-learning framework on this problem**
4. **Performance and effectiveness of our approach including novel applications**

We believe that, despite some detailed imperfections, our work has the potential to inspire future works and pave the way for new research areas.

We will answer the review comments and questions and hope this could help address some concerns. Here we will first respond to some common questions in this block and then provide individual responses to each reviewer. To streamline the discussion, we will reference this global response in the individual sections when necessary. An accompanying PDF with figures for this rebuttal is also attached.

**Global response to common questions:**

- **[G1]** **Paper writing**: We are thankful to the careful and constructive feedback from all the reviewers including typos, inconsistencies, missing details, figure labels, and table layout. Indeed, our presentation, especially in Sec.3.1, is not that easy to follow due to the compression of the original manuscript to fit the page limits. We will revise paragraphs, update all the issues the reviewers bring up, and check all the text carefully. The discussions in this response will also be incorporated into our revised paper. As suggested by WYGQ, a draft schematic visualization of the variables described in Sec. 3.1 is provided in Fig.R8 in the attached PDF.

- **[G2]** **Overfitting**(PAAU, HNRT): Regarding this problem, we would like to argue the following points:
  1. We retrieve the nearest training object (w.r.t. our ID metric) and visualize some samples in the attached Fig.R1. As it shows, both geometric and structural differences can exist in the (generated objects, nearest neighbors) pairs.
  2. We mainly demonstrate our effectiveness in modeling articulated-object distributions because this is the most important feature of generative models. Other baselines, on the contrary, cannot efficiently fit the distribution with similar or even larger network capacities.
  3. The overfitting problem is also an active research problem for diffusion models in general [1]. Even the well-known image diffusion models show overfitting properties, yet they are widely used for their effectiveness in many applications – this also applies to our method. For example, in Sec.4.4 we show that the learned articulation prior can be used for articulating static models (Fig.6) or doing part-to-articulated-object completion with multiple proposals using conditioned generation.
  4. One bottleneck here compared to the widely-existing image diffusion models, as pointed out by HNRT, is the size of the dataset for training. Potentially, collecting larger datasets would be very beneficial to the community. On the other side, new techniques that learn articulation prior from images or videos may also help to address this issue.
  5. For evaluation, we have different training and testing dataset splits, ensuring the evaluations of the learned distribution are valid.

[1] Carlini, Nicholas, et al. "Extracting training data from diffusion models." arXiv preprint arXiv:2301.13188 (2023).


Meta-review: This paper initially received mixed reviews with concerns revolving around the novelty of the task definition, some limitations of the approach not being clearly explained, concern that the presented results may not generalize due to over-reliance on a specific dataset, and lastly some clarity issues with the writing in the paper.  During the rebuttal and discussion period, the authors responded to these concerns and largely alleviated them.  With the exception of one borderline-negative reviewer who is not strongly opposed, all reviewers would like to see the paper accepted.  The AC finds no basis to overturn this near-consensus and therefore recommends acceptance.

The reviewer who remained negative (ncuk) had as a primary concern issues with the clarity of exposition.  The authors responded with rewritten parts of the paper that demonstrate a commitment to appropriately revising the draft to incorporate reviewer comments and suggestions.  The AC strongly recommends a thorough editing pass when preparing the camera ready version to incorporate the clarifications and improvements from this discussion period.
Decision: Accept (poster)