SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiahui Lei1 Congyue Deng2 Bokui Shen2 Leonidas Guibas2 Kostas Daniilidis13
<sup>1</sup> University of Pennsylvania
2 Stanford University
3 Archimedes, Athena RC
{leijh, kostas}@cis.upenn.edu, {congyue, willshen, guibas}@cs.stanford.edu
https://www.cis.upenn.edu/~leijh/projects/nap
We propose Neural 3D Articulated object Prior (NAP), the first 3D deep generative model to synthesize 3D articulated object models. Despite the extensive research on generating 3D static objects, compositions, or scenes, there are hardly any approaches on capturing the distribution of articulated objects, a common object category for human and robot interaction. To generate articulated objects, we first design a novel articulation tree/graph parameterization and then apply a diffusion-denoising probabilistic model over this representation where articulated objects can be generated via denoising from random complete graphs. In order to capture both the geometry and the motion structure whose distribution will affect each other, we design a graph denoising network for learning the reverse diffusion process. We propose a novel distance that adapts widely used 3D generation metrics to our novel task to evaluate generation quality. Experiments demonstrate our high performance in articulated object generation as well as its applications on conditioned generation, including Part2Motion, PartNet-Imagination, Motion2Part, and GAPart2Object.
Figure 1: NAP can unconditionally generate articulated objects (left). It can be conditioned on just parts or joints (mid), a subset of parts plus joints, or over-segmented static objects (right).
Articulated objects are prevalent in our daily life. As humans, we have strong prior knowledge of both object part geometry and their kinematic structures. Such knowledge is most heavily leveraged when a designer designs a cabinet from scratch, creating both its geometry and motion structure. For learning systems, an interesting challenge is to capture such priors as a generative model that can synthesize articulated objects from scratch. While there has been extensive research on generative models for static 3D objects [1-10], compositions [11-18], and scenes [19-30], the study of priors regarding closely linked 3D part geometry and 3D motion structures has been relatively neglected. In this work, we study how to synthesize articulated objects, i.e., how to generate a full description
of an articulated object, an actual URDF [31], including how each part looks like, which pairs of parts are connected, and with what kind of joint. In contrast to static 3D generation, generating articulated objects involves modeling not only the distribution and composition of geometry but also the motion structures that determine the possible relative movements between rigid parts. Our task is also different than 3D human/hand synthesis [32-35] where the articulation structure is given and only the degrees of freedom are generated when predicting human poses/sequences. The generation of articulation can further impact simulation and mechanism design and be useful for inference when conditioned on either geometry or kinematic structure.
However, there are several challenges for articulated object generation. Existing datasets of articulated objects contain highly irregular data since articulated objects have different numbers of parts, diverse part connectivity topology, and different joint motion types. In order to enable efficient processing of diverse geometry and structures by a neural architecture, we propose a novel unifying articulation tree/graph parameterization (Sec. 3.1) to represent articulated objects. We take advantage of recent progress in diffusion models for 3D generation and develop a diffusion-denoising probabilistic model over our parameterization to faithfully model the irregular distribution of articulated objects (Sec. 3.2). Since we are modeling the joint distribution of the part geometry and the inter-part motion constraints, we design a graph denoising network (Sec. 3.3) to gradually exchange and fuse information between the edge and nodes on our articulation graph.
To evaluate articulated object generation quantitatively, we adopt the widely used 3D shape generation metric for articulated objects by introducing a novel distance measure between two articulated objects. Through extensive comparisons, we demonstrate high performance in articulated object synthesis and further conduct several ablations to understand the effect of each system component. Using the learned prior knowledge, we demonstrate conditioned generation applications, including Part2Motion, PartNet-Imagination, Motion2Part, and GAPart2Object.
In summary, our main contributions are (1.) introducing the articulated object synthesis problem; (2.) proposing a simple and effective articulation tree parameterization sufficiently efficient for a diffusion denoising probabilistic model that can generate articulated objects; (3.) introducing a novel distance for evaluating this new task; and (4.) demonstrating our high performance in articulated object generation and presenting several conditioned generation applications.
Articulated object modeling. Modeling articulated objects has been a prolific field with existing works broadly classified into the categories of estimation, reconstruction, simulation, and finally, our generation. Note that there is a wider literature on semi-nonrigid objects, for example, human body [36], hands [37], and animals [38]. This paper focuses on everyday articulated objects, like the ones in PartNet-Mobility [39], which have more diverse and complex structures and are strictly multi-body systems. Estimation focuses on predicting the articulation joint states (joint angles and displacements) or joint parameters (type, axis and limits) from sensory observations, using approaches ranging from probabilistic models [40-44], interactive perception [45-51] and learning-based inference [52-65]. Reconstruction of articulated objects focuses on reconstructing both articulation and geometry properties of the objects, using techniques ranging from structure from motion [66], learning-based methods [67-69] and implicit neural representations [70-74]. While these methods mainly focus on surface reconstruction and joint states/parameter accuracy or are limited to pre-defined simple kinematic structures, we explicitly model the diverse and complex articulation motion structures. One closely related work to ours is [62], which predicts the joint parameters of articulated objects in PartNet [75] by training on PartNet-Mobility [39]. However, [62] is a single-point regression that does not capture the generative distribution of joints or parts. A growing field in robotics and embodied AI is building interactive environments that support physical interactions between the robot and the scene consisting of articulated objects [75, 39, 76-82]. Differing from all the above approaches, we build a generative prior of articulated objects, which extends beyond estimation and reconstruction. Such learned prior has essential further to accelerate the creation of realistic interactive 3D assets.
Generative models for structures. Generating structured articulated objects is closely related to generative models for structured data [83]. Part-based 3D object generation [11-18] has been widely studied with the main modeling target being the hierarchy of static rigid parts with or without a semantic label. Scene layout generation [19-30] utilizes compact scene parameterizations for indoor scene synthesis, where diffusion models on scene graphs have been recently introduced in [21, 22]. The generative diffusion approach has also been applied widely to 2D floor plans [84], protein struc
tures [85], and graphs [86].etc. Unlike all the existing works, our work focuses on modeling a new type of target - articulation structure plus part shapes, which requires joint reasoning between 3D geometry and kinematic structure.
Diffusion models in 3D. We mainly focus on the literature on diffusion models for 3D shape and motion generation. Shape: Diffusion models show impressive results in generating point clouds [1-3], meshes [87], implicit surfaces [4-10], neural radiance fields [88-91], or 4D non-rigid shapes [92]. However, these methods mostly focus on the single object level shape quality and do not pay attention to the kinematic structure. Motion: Diffusion models have seen many recent applications in motion generation given an articulation model. Such works generate text-conditioned human motion [32-35], and physically-viable [93], audio-driven [94, 95], scene-aware [96], multi-human [97] or animation [98] trajectories. Diffusion models for motion have also been applied to trajectory planning [99], visuomotor control [100], and rearrangement tasks [101]. Different from ours, existing works in motion diffusion rely on known geometries with known motion structures. We instead jointly model geometry and motion structure priors to create articulation models.
Figure 2: Method: Parameterization (Top): We parameterize the articulated object as a tree, whose nodes are rigid parts and edges are joints; we then pad the tree to a complete graph of maximum node number and store it in the articulation graph attribute list x. Forward Diffusion (Middle): The parameterized attribute list x is gradually diffused to random noise. Generation (Bottom): A Graph Denoiser (Fig 3, Sec. 3.3) samples a random articulation graph  $\mathbf{x}_T$ , gradually removes noise, and finally predicts  $\mathbf{x}_0$ . An object extraction stage (Sec. 3.2) including a minimum-spanning-tree algorithm is applied to the generated graph in the end to find the kinematic tree structure and the output articulated object.
We learn the articulated object priors using a diffusion model. However, an articulated structure must first be parameterized into a vector that can be the target of the diffusion. We introduce the parameterization of the articulated object as a tree (graph) in Sec. 3.1; the diffusion model on such parameterization in Sec. 3.2; and the denoising network in Sec. 3.3. An overview of our method is shown in Fig. 2.
3.1 Articulation Tree Parameterization
Graph-based representation. We follow the natural articulated object parameterization as in URDF [31] format where each object is defined by a graph with nodes being the parts and edges being the articulation joints. We make two assumptions: (1) Tree assumption: We assume no kinematic motion loop (cycle) exists in the graph and that the graph is connected. (2) Screw joints: We assume
each edge is a screw [60] with at most one prismatic translation and one revolute rotation. Most real-world articulated objects [39, 76, 82, 102] satisfy the above assumptions and their limitations are further discussed in Sec. 5.
Nodes. As shown in Fig. 2-top, we represent each rigid part as one node in the tree. A node captures its rigid part shape by (1) a shape latent code  $s_i \in \mathbb{R}^F$  (with  $F$  being channels,  $i$  being the part index) that can decode the SDF [103] of the part surface, and (2) the bounding box edge lengths  $b_i \in \mathbb{R}^3$  that can scale the decoded SDF properly into the part's scale. We pre-train an SDF shape Auto-Encoder [104, 103] for obtaining  $s_i$ , further details are provided in our supplementary material. Importantly, one should also specify how to assemble these parts into an object before further modeling the articulation motion. We do so by adding to the node attributes a part initial rigid transformation  $T_i \in SE(3)$  in the global object frame representing the part initial configuration.  $T_i$  comprises an axis-angle rotation and a translation, which can be written as a 6-dimensional vector. As articulated objects can be instantiated to different configurations so any of these configurations can be used as  $T_i$ , in practice, we consistently choose  $T_i$  by exploiting the canonicalization in the large annotated dataset: we use the part poses in their rest states (zero joint angles and displacements) for  $T_i$ . Since we observe current datasets often align their rest states consistently to static un-articulated objects, for example, PartNet-Mobility [39] aligns with static PartNet [75]. To model the varying number of parts across objects, we define a maximum number of  $K$  parts as well as a per-part binary indicator  $o_i \in \{0,1\}$  of part  $i$  existence. In summary, a node  $i$  has an attribute vector  $v_i = [o_i,T_i,b_i,s_i]$  with dimension  $D_v = 1 + 6 + 3 + F$ , and the overall node component of an object graph is a feature of dimension  $K \times D_v$ .
Edges. The edge in the graph represents the motion constraint of each articulation joint. A joint possesses a 3D axis (a directed line), around which the revolute joint can rotate and along which the prismatic joint can translate. Inspired by [60], such a joint axis is represented by Plücker coordinates  $(l \in \mathbb{S}^2, m \in \mathbb{R}^3)$ . Here the 3D directional vector  $l$  is from a unit sphere  $\mathbb{S}^2$  and the momentum  $m$  is perpendicular to  $l$ . Such representation avoids defining additional local joint coordinate frames with ambiguity (e.g. translating the joint coordinate frame along a revolute axis leads to equivalent joints). Such 6-dimensional Plücker coordinates  $p_{(i,j)} = (l_{(i,j)}, m_{(i,j)})$  for the joint from part  $i$  to  $j$  are defined in the global object frame when parts are in their initial rest configuration  $T_i, T_j$ . To fully define the joint motion constraint, we also incorporate two joint state ranges  $r_{(i,j)} \in \mathbb{R}^{2 \times 2}$  for both the prismatic translation and the revolute angle components (the left limit of the range can be negative). A purely prismatic joint will have its revolute component range set to  $[0,0]$  and vice versa. Following the node padding, we also pad the edges to a complete graph and use an indicator  $c_{(i,j)}$  denoting the edge existence. Note that the above joint axis and range have a parent-child direction from part  $i$  to  $j$ . When the direction flips, a notable benefit of expressing  $p_{(i,j)}$  in the global object frame, rather than the local part frame, is the inherent relationship  $p_{(i,j)} = -p_{(j,i)}$  and  $r_{(i,j)} = r_{(j,i)}$ , which motivates us to model the padded graph with only  $K(K-1)/2$  edges for all  $i < j$  pairs. Since the nodes have no specific order, to avoid the sign flipping in  $p_{(i,j)} = -p_{(j,i)}$  when permuting the nodes and to help the network learn a more stable prior, we explicitly embed the sign inside  $p_{(i,j)} = -p_{(j,i)}$  into the edge existence indicator  $c_{(i,j)} \in \{-1,0,+1\}$ . Here  $0$  indicates the non-existence of an edge and  $+1,-1$  indicate the chiralities of existing edges, leading to  $\hat{p}_{(i,j)} = \hat{p}_{(j,i)}$  (where  $\hat{p}$  denotes the actual prediction target). In essence, each edge is characterized by an attribute vector  $e_{(i,j)} = [c_{(i,j)}, p_{(i,j)}, r_{(i,j)}]$  with a dimension of  $D_e = 1 + 6 + 4$ . The overall edge parameterization of an object has shape  $K(K-1)/2 \times D_e$ . We will see later in Sec. 3.2 how to extract an articulated object model from the diffusion model prediction via the Minimum Spanning Tree.
3.2 Diffusion-Based Articulation Tree Generation
Our goal is to learn the distribution of articulated objects parameterized by the complete articulation graphs  $\mathbf{x} = (\{v_{i}\},\{e_{(i,j)}\})\in \mathbb{R}^{KD_v + K(K - 1)D_e / 2}$ . We apply a diffusion denoising probabilistic model [105] directly over the distribution of  $\mathbf{x}$ .
Forward diffusion. Given an articulation graph  $\mathbf{x}_0$  from an object distribution  $q(\mathbf{x}_0)$ , we gradually add Gaussian noise with variance schedule  $\beta_1 < \dots < \beta_T$  and obtain a sequence  $\mathbf{x}_1, \dots, \mathbf{x}_T$  ended with a Gaussian distribution  $p(\mathbf{x}_T) = \mathcal{N}(\mathbf{x}_T; \mathbf{0}, \mathbf{I})$ . The joint distribution of the forward process is:
$$
q \left(\mathbf {x} _ {1: T} \mid \mathbf {x} _ {0}\right) := \prod_ {t = 1} ^ {T} q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t - 1}\right), \quad q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t - 1}\right) := \mathcal {N} \left(\sqrt {1 - \beta_ {t}} \mathbf {x} _ {t - 1}, \beta_ {t} \mathbf {I}\right). \tag {1}
$$
A notable property is that  $\mathbf{x}_t$  at arbitrary timestep  $t$  can be directly sampled from  $\mathbf{x}_0$  with
$$
q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {0}\right) = \mathcal {N} \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0}, (1 - \bar {\alpha} _ {t}) \mathbf {I}\right) \tag {2}
$$
where  $\alpha_{t} := 1 - \beta_{t}$  and  $\bar{\alpha}_{t} := \prod_{s=1}^{t} \alpha_{s}$ .
Reverse process. Starting from a standard Gaussian distribution  $\mathbf{x}_T\sim \mathcal{N}(\mathbf{0},\mathbf{I})$ , we aim to learn a series of Gaussian transitions  $p_{\theta}(\mathbf{x}_{t - 1}|\mathbf{x}_t)$  parameterized by a neural network with learnable weight  $\theta$  that gradually removes the noise. The distribution of the reverse process is:
$$
p _ {\theta} \left(\mathbf {x} _ {0: T}\right) := p \left(\mathbf {x} _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right), \quad p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right) := \mathcal {N} \left(\mu_ {\theta} \left(\mathbf {x} _ {t}, t\right), \Sigma_ {\theta} \left(\mathbf {x} _ {t}, t\right)\right). \tag {3}
$$
Following [105], we set  $\Sigma_{\theta}(\mathbf{x}_t,t) = \sigma_t^2\mathbf{I}$  and model this reverse process with Langevin dynamics
$$
\mathbf {x} _ {t - 1} = \frac {1}{\sqrt {\alpha_ {t}}} \left(\mathbf {x} _ {t} - \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \epsilon_ {\theta} (\mathbf {x} _ {t}, t)\right) + \sigma_ {t} \mathbf {z}, \quad \mathbf {z} \sim \mathcal {N} (\mathbf {0}, \mathbf {I}) \tag {4}
$$
where  $\epsilon (\mathbf{x}_t,t)$  is a learnable network approximating the per-step noise on  $\mathbf{x}_t$ .
Training objective. We optimize the variational bound on the negative log-likelihood
$$
L := \mathbb {E} _ {q} \left[ - \log \frac {p _ {\theta} \left(\mathbf {x} _ {0 : T}\right)}{q \left(\mathbf {x} _ {1 : T} \mid \mathbf {x} _ {0}\right)} \right] = \mathbb {E} _ {q} \left[ - \log p (\mathbf {x} _ {T}) - \sum_ {t \geq 1} \log \frac {p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right)}{q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t - 1}\right)} \right] \geq \mathbb {E} [ - \log p _ {\theta} (\mathbf {x} _ {0}) ] \tag {5}
$$
With Eq. 4, the objective simplifies to
$$
\mathbb {E} _ {\mathbf {x} _ {0}, \epsilon} \left[ \frac {\beta_ {t} ^ {2}}{2 \sigma_ {t} ^ {2} \alpha_ {t} (1 - \bar {\alpha} _ {t})} \left\| \epsilon - \epsilon_ {\theta} \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \epsilon , t\right) \right\| ^ {2} \right] \tag {6}
$$
w.r.t. learnable network  $\epsilon_{\theta}$ . We refer the readers to [105] for more details.
Output Extraction. The above-described Euclidean space denoising intermediate steps as well as the final output may not strictly lie on the manifold of articulated objects parameterized as in Sec. 3.1 (e.g. the generated joint Plücker coordinates may not be valid because  $l$  and  $m$  may not be orthogonal). We found that utilizing the diffusion as in Euclidean space and projecting the generated  $\mathbf{x}$  back to a valid articulation graph in the final step already leads to practically good results. As mentioned in Sec. 3.1, once we have completed the denoising process, a post-processing step is applied to obtain the final articulated object model from the generated  $\mathbf{x}$ . First, we identify the existing nodes by the generated nodes indicator  $o$  with a threshold and make sure there are at least two foreground nodes. Then we use the predicted edge chirality  $|c|$  as the edge value to find the minimum spanning tree in the generated graph as the output tree topology. As in Sec. 3.1, we model half of the edges in the complete graph with node index  $i < j$  and put the edge direction explicitly to the chirality sign. During the object extraction stage, if  $c < 0$ , we flip the predicted joint direction on this edge and if  $c > 0$  we keep the joint direction, which is equivalent to flipping the parent and child order of this edge when  $c < 0$ . Finally, the predicted joint coordinates are projected from  $\mathbb{R}^6$  to Plücker coordinates by normalizing the predicted  $l$  to unit vector and subtracting the parallel component of  $m$  on  $l$  to make predicted  $m$  orthogonal to  $l$ . We decode the part shape code to an SDF and extract the part mesh via marching cubes. We alternatively can retrieve the nearest part in the training set as most scene generation methods do [23, 22].
Conditioned Generation. A favorable property of diffusion models is that conditions can be directly added to the inference processes using Bayes' rule without any modification to the training process. Here we perform conditioned generation by fixing the known part of variable  $\mathbf{x}$  as in the image inpainting [106] and shape completion works. For a variable  $\mathbf{x} = m\odot \mathbf{x}^{\mathrm{known}} + (1 - m)\odot \mathbf{x}^{\mathrm{unknown}}$  ( $\odot$  means element-wise multiplication) with known entries  $\mathbf{x}^{\mathrm{known}}$  fixed unknown entries  $\mathbf{x}^{\mathrm{unknown}}$  to be completed which are separated by mask  $m$ , we can sample the known entries directly following Eq. 2 by adding Gaussian noise to the known input and generating the unknown entries using reverse diffusion,
$$
q \left(\mathbf {x} _ {t - 1} ^ {\text {k n o w n}} \mid \mathbf {x} _ {0}\right) = \mathcal {N} \left(\sqrt {\bar {\alpha}} \mathbf {x} _ {0}, (1 - \bar {\alpha} _ {t}) \mathbf {I}\right), \quad p _ {\theta} \left(\mathbf {x} _ {t - 1} ^ {\text {u n k n o w n}} \mid \mathbf {x} _ {t}\right) = \mathcal {N} \left(\mu_ {\theta} \left(\mathbf {x} _ {t}, t\right), \Sigma_ {\theta} \left(\mathbf {x} _ {t}, t\right)\right). \tag {7}
$$
Since our method directly diffuses in graph space, we can apply precise control of the parts and joints condition, enabling, thus, a disentanglement. In Sec. 4.4, we will show applications with different  $\mathbf{x}^{\mathrm{known}}$  and  $\mathbf{x}^{\mathrm{unknown}}$ .
Figure 3: Network architecture. Left: input the node and edge list of a noisy articulation graph, a stack of graph layers will fuse and exchange information on the graph and output the noise that has to be removed. Right: details in the graph layer.
3.3 Denoising Network
Network architecture. Since  $\mathbf{x}$  represents both the part geometry and the joint motion constraints, we utilize a graph denoising network as shown in Fig. 3, which exchanges and fuses information between parts and joints. The network inputs a noisy articulate graph  $\mathbf{x}$  and outputs noise in the same shape as  $\mathbf{x}$ . The input attributes for every node are first encoded by input heads shared across nodes to a list of node features  $\{f_i\}$ . Similarly, edge attributes are encoded to a list of edge features  $\{g_{(i,j)}\}$ . Then the node and edge feature lists are updated via the graph layers (Fig. 3 right). Finally, all hidden features, including the input attributes, are concatenated together and decoded to the outputs via shared output heads. Note that the network is shared across all time steps through denoising, and the time step is encoded and concatenated to the hidden features. Similar to recent work on scene graph generation [22], we also append the positional encoding of the part instance id to the node features to provide stronger guidance in the early denoising stages when the part pose information is ambiguous. To learn with positional encoding, we randomly permute the nodes' order during training.