SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Graph layer. The key building block of the denoising network is the graph layer shown in Fig. 3 right. The edge feature is first updated via an edge MLP by fusing the input edge feature  $g_{(i,j)}$  and neighboring nodes  $i, j$ 's features:  $g_{(i,j)}' = \mathrm{MLP}(f_i, f_j, g_{(i,j)})$ . Then, we aggregate the updated edge features to the nodes by attention weights. We compute the query  $Q(f_i)$  and key  $K(f_i)$  from the input node features via two MLPs and use their inner product as the attention weights; the graph attention update of the node  $i$  is:  $f_i' = \sum_{j=1}^{K} \operatorname{softmax}_j (Q(f_i)^T K(f_j)) g_{(i,j)}'$ . We additionally do a PointNet [107]-like global pooling over all the graph node features after the attention aggregation to capture more global information.
We examine 4 important questions with our experiments: (1) How can we evaluate articulated object generation? (Sec. 4.1) (2) How well does NAP capture the distribution of articulated objects? (Sec. 4.2) (3) How effective is each of NAP's components? (Sec. 4.3) (4) What applications can NAP enable? (Sec. 4.4)
4.1 Evaluation Metrics
Since we are the first to study articulated objects in a generative setting, we propose a new distance metric between two articulated objects for adopting widely used shape generation metrics. Since a generated object's shape and motion structure are dependent, we can not evaluate them separately. Such a challenge is poorly addressed in existing works. In articulated object modeling, existing works either consider a fixed kinematic structure [71, 70] or a given geometry [108]. In graph generation works [109-111], structures are examined by themselves without need to measure geometry. Thus, we propose a new distance metric, Instantiation Distance (ID), to measure the distance between two articulated objects considering both the part geometry and the overall motion structure.
We treat an articulated object  $O$  as a template that, given the joint states  $q \in \mathcal{Q}_O$  in object's joint range  $\mathcal{Q}_O$ , it returns the overall articulate mesh  $\mathcal{M}(q)$  and the list of part poses  $\mathcal{T}(q) = \{T_{\mathrm{part}} \in SE(3)\}$ . We compute the distance between two articulated objects in different joint states by
$$
\tilde {d} \left(O _ {1}, q _ {1}, O _ {2}, q _ {2}\right) = \min  _ {T _ {i} \in \mathcal {T} _ {1} \left(q _ {1}\right), T _ {j} \in \mathcal {T} _ {2} \left(q _ {2}\right)} \left\{D \left(T _ {i} ^ {- 1} \mathcal {M} _ {1} \left(q _ {1}\right), T _ {j} ^ {- 1} \mathcal {M} _ {2} \left(q _ {2}\right)\right) \right\}, \tag {8}
$$
Table 1: Articualted object synthesis comparison with Instantiation Distance
Part SDF Shape	Part Retrieval Shape
Generative Paradigm/Method	MMD ↓	COV ↑	1-NNA ↓	MMD ↓	COV ↑	1-NNA ↓
Auto-Decoding (StructNet)	0.0435	0.1871	0.8820	0.0390	0.2316	0.8675
Variational Auto-Encoding (StructNet)	0.0311	0.3497	0.8085	0.0289	0.3363	0.7918
Autoregressive (ATISS-Tree)	0.0397	0.3808	0.6860	0.0333	0.4120	0.6782
Latent Diffusion (StructNet)	0.0314	0.4365	0.6269	0.0288	0.4477	0.6102
Articulation Graph Diffusion (Ours)	0.0268	0.4944	0.5690	0.0215	0.5234	0.5412
where  $T_{i}^{-1}\mathcal{M}_{1}(q_{1})$  means canonicalizing the mesh using its  $i$ th part pose, and  $D$  is a standard distance that measures the distance between two static meshes. Specifically, we sample  $N = 2048$  points from two meshes and compute their Chamfer Distance. Intuitively, the above distance measures the minimum distance between two posed articulated objects by trying all possible canonicalization combinations. Then, we define the instantiation distance between  $O_{1}$  and  $O_{2}$  as:
$$
\begin{array}{l} I D \left(O _ {1}, O _ {2}\right) = \mathbb {E} _ {q _ {1} \in \mathcal {U} \left(\mathcal {Q} _ {O _ {1}}\right)} \left[ \inf  _ {q _ {2} \in \mathcal {Q} _ {O _ {2}}} \left(\tilde {d} \left(O _ {1}, q _ {1}, O _ {2}, q _ {2}\right)\right) \right] \tag {9} \\ + \mathbb {E} _ {q _ {2} \in \mathcal {U} (\mathcal {Q} _ {O _ {2}})} \left[ \inf  _ {q _ {1} \in \mathcal {Q} _ {O _ {1}}} \left(\tilde {d} (O _ {1}, q _ {1}, O _ {2}, q _ {2})\right) \right], \\ \end{array}
$$
where  $q \in \mathcal{U}(\mathcal{Q}_O)$  means uniformly sample joint poses from the joint states range. The instantiation distance measures the two-side expectation of minimum distance to the other object over all possible joint configurations. However, the inf inside the expectation requires expensive registration between two articulated objects so it is non-trackable in practice when computing all distance pairs between the reference and sampled object sets. In practice, we approximate the above distance by uniformly sampling  $M$  joint poses  $Q_{1} = \{q_{k}|q_{k}\in \mathcal{U}(\mathcal{Q}_{O_{1}}),k = 1,\dots ,M\}$  and the approximated distance is:
$$
I D \left(O _ {1}, O _ {2}\right) \approx \frac {1}{M} \sum_ {q _ {1} \in Q _ {1}} \left[ \min  _ {q _ {2} \in Q _ {2}} \left(\tilde {d} \left(O _ {1}, q _ {1}, O _ {2}, q _ {2}\right)\right) \right] + \frac {1}{M} \sum_ {q _ {2} \in Q _ {2}} \left[ \min  _ {q _ {1} \in Q _ {1}} \left(\tilde {d} \left(O _ {1}, q _ {1}, O _ {2}, q _ {2}\right)\right) \right], \tag {10}
$$
and we set  $M = 10$  in our ID for all evaluations. This pairwise distance can be plugged into the standard metrics for shape generation. Specifically, we adopt the following three metrics [112] for our evaluation: minimum matching distance (MMD) that measures the generation quality, coverage (COV) that tests the fraction the reference set is covered, and 1-nearest neighbor accuracy (1-NNA) that measures the distance between the two distributions by 1-nn classification accuracy.
4.2 Articulated Object Synthesis
Baselines. We adapt existing models in related tasks and compare our method with them. Specifically, we adapt architecture designs from semantic-part-based shape generation [11] and scenegraph-based indoor scene synthesis [23], and equip them with different generative paradigms including auto-decoding [113], VAE [11], autoregressive models [23], and latent diffusion [4, 5]. We refer to our supplementary for adaptation and details of these baselines.
**Setup.** We train all the methods on PartNet-Mobility [39] across all categories jointly, with a maximum 8 rigid parts ( $K = 8$ ) and a train-val-test split ratio  $[0.7, 0.1, 0.2]$ . Since the dataset does not include parts orientation and all initial part poses have rotation  $I$ , we ignore the rotation in the parameterization (Sec. 3.1) for all the methods. For a fair comparison, all the methods are controlled to a similar number of learnable parameters. We evaluate the generated articulated object models with both the reconstructed meshes from the generated shape code and the retrieved part meshes from the training set.
Comparison. We report quantitative results in Tab. 1, and qualitative comparisons in Fig. 4. While the Auto-Decoding baseline fits the training set, its generation performance is poor when sampling in the latent space as shown in Fig. 4-AD, which suggests a weak regularization in the latent space. Using VAE to replace auto-decoding brings about better regularization of the latent space, resulting in an increase in all metrics. Sampling from the prior of VAE also leads to more meaningful generations, as shown in Fig. 4-VAE. For the autoregressive ATISS-Tree baseline, we see a relative
Figure 4: Articulated object generation results. Each generated object is visualized with (1) graph topology (top left), where the edge color means blue-prismatic, red-revolute, and orange-hybrid; (2) the predicted part bounding boxes and joints under different joint states (second column), and the overlay of multiple states reflecting the possible motion (bottom left); (3) reconstructed part meshes from the generated shape code (third column); (4) retrieved part meshes (right column).
increase in a majority of metrics compared to VAE. Part shape retrieval leads to further improvement in both motion structure and part shape since using retrieval at each autoregressive step can decrease the deviation from the training distribution. Interestingly, as shown in Fig. 4, we observe that the autoregressive method also has a tendency to append too many nodes to the tree, resulting in overlapping parts. Latent diffusion works the best among our baselines. We hypothesize that it is due to the superiority of the diffusion model as a sampler in the latent space, mapping the prior Gaussian to reliable regions where the trained decoder performs well near training samples. However, as the generation happens in the latent space and the generated latent code have to be decoded, slight error or changes in the latent space may lead to unrealistic or wrong articulations, which is shown in Fig. 4-LD. Different from the latent diffusion, our method directly applies diffusion in the articulation tree space, which can generate diverse and high-quality articulation models and achieves a better performance comparing to baselines.
Table 2: Ablation studies with Instantiation Distance
Part SDF Shape	Part Retrieval Shape
Ablation	MMD ↓	COV ↑	1-NNA ↓	MMD ↓	COV ↑	1-NNA ↓
Full	0.0268	0.4944	0.5690	0.0215	0.5234	0.5412
No PE	0.0282	0.4766	0.5490	0.0227	0.5457	0.5557
No Attn. on Edge	0.0286	0.4766	0.5668	0.0232	0.5234	0.5568
No Graph Conv	0.0331	0.4432	0.6570	0.0281	0.4722	0.6481
We verify our denoising network design by ablating components in our full network, and the comparison is shown in Tab. 2. Specifically, we examine the effectiveness of positional encoding, attention weights on edges, and graph convolutions. First, removing the positional encoding will slightly worsen performance, since its presence can help to distinguish the nodes at the early stages of reverse diffusion [22]. Second, replacing attention weights with mean pooling when aggregating the neighboring nodes' information results in a drop in performance. Finally, we justify the importance of node-edge information exchange in the graph convolution by removing the graph convolution layer and replacing it with a PointNet [107]-like global pooling layer, where the connectivity of the graph is ignored and the information exchanges through a symmetric global pooling. We observe a larger performance decrease, which justifies our graph-based processing.
4.4 Applications with Conditioned Generation
Figure 5: Part2Motion: Known part condition on the left, diverse motion proposals on the right.
Figure 6: PartNet Imagination: Input over-segmented static PartNet shapes (top left) are grouped into rigid parts (bottom left) and be hallucinated with articulations (right). Both training categories (encircled by dashed lines) and out-of-distribution objects are shown.
Once trained, our method can be used directly for conditioned generation. Following Sec. 3.2, we demo applications conditioning on various known attributes in  $\mathbf{x}$ .
Part2Motion. We first show that when knowing the static part attributes, how NAP can suggest motion structures, i.e., with  $\mathbf{x}^{\mathrm{known}} = \{v_i\}$  and  $\mathbf{x}^{\mathrm{unknown}} = \{e_{(i,j)}\}$  in Eq. 7. We use the unseen object part attributes from the test set as conditions and the generated motion structure is in Fig. 5. We observe diverse and plausible motion suggestions that cover the ambiguity of the closed doors.
PartNet Imagination. NAP uses PartNet-Mobility [39] for training, which is only a small subset of the large-scale but static PartNet [75]. We show that NAP can be used to imagine possible motion structures in static PartNet [75]. Starting from the finest semantic part labeled in PartNet [75] (as an over-segmentation), we use a simple contrastive learned grouping encoder (see Suppl.) to group the fine semantic parts into rigid parts and then apply the same node condition as in Part2Motion. Fig. 6 shows our predictions. Interestingly, we find the learned articulation prior can be applied to out-of-distribution object categories and generate intriguing motion structures that adhere to some human recognizable logic (e.g., the mug in Fig. 6 bottom, where the handle rotates around the body). This further suggests that our model captures an underlying articulation prior.
Figure 7: Motion2Part: Known motion structure (left) and suggested parts (right).
Figure 8: GAPart2Object: GAParts [81] (left) completed into full articulated objects (right).
Motion2Part. Similar to Part2Motion, we can flip the conditions and suggest possible parts from the motion structure. Given known edge attributes and graph topology  $\mathbf{x}^{\mathrm{known}} = \{e_{(i,j)}\} \bigcup \{o_i\}$  and  $\mathbf{x}^{\mathrm{unknown}} = \{T_{gi}, b_i, f_i\}$ , NAP can generate possible parts that fit the motion structure in Fig. 7.
GAPart2Object. As Part2Motion and Motion2Part are pure edge or node conditions, we can also combine the node and edge conditions to complete an articulated object from one part. GAPart [81] is human-labeled semantically generalizable parts and one GAPart has a part geometry plus a joint type. We use the GAPart [81] from our testset as the known condition. Specifically, we set  $\mathbf{x}^{\mathrm{known}} = \{[o_0,b_0,f_0],[o_1]\} \bigcup \{[c_{(0,1)},l_{(0,1)},r_{(0,1)}]\}$  otherwise unknown, where we ensure that a GAPart is part-0 and there must be one part-1 connected to it. We constrain the joint direction  $l_{(0,1)}$  but free the momentum  $m_{(0,1)}$  and also let the non-zero joint range be freely denoised. The completion results are shown in Fig. 8.
We introduce the articulated object synthesis problem to our community and propose the first deep neural network that is able to generate articulated object models. Such generation is achieved via a diffusion model over a novel articulation graph parameterization with a graph denoising network. We further introduced a new Instantiation Distance to adopt the widely used 3D shape generation metrics to this task. Our learned neural 3D articulation can be used under different conditions for diverse application settings. We believe this initial step towards articulated object understanding and generation will bring more opportunities to understand and simulate our dynamic and complex real world.
Figure 9: Limitations: Left The number of rigid parts in the current dataset has a long tail distribution, leading to extremely challenging objects, for example, a remote or a keyboard. Middle: Object violating tree or screw joint assumption. Right: The generated objects do not strictly obey physical constraints, for example, self-collision, making them not directly simulatable.
Limitations and future work. Although we have taken initial steps towards this new challenge, there are still many issues that require further exploration.
1. Long tail distribution and unbalanced structures: We noticed that our methods and baselines work better for small graphs. Large graphs with many parts are rare in the training set, making the generative model biased towards small graphs.
2. Overfitting: Like many other generative models, we do observe that NAP slightly overfits the training dataset. One bottleneck is that the current dataset still contains too few objects (around 2k) for training generative models. Studying how to learn an articulation prior from the camera observations or building a larger dataset will help to address this issue.
3. Tree and screw assumption: As shown in Fig. 9-Mid. NAP currently can not model kinematic cycles (pairs) or model several degrees of freedom on a single joint. Introducing dummy nodes and synchronization between joints may solve this limitation.
4. More structured diffusion: Although demonstrated to be effective, the diffusion we applied is straightforward and may be improved by introducing discretization, for example, the MST in the intermediate steps or by constraining the diffusion to the Plücker manifold.
5. Physically plausible generation: As shown in Fig. 9-Right, another exciting direction is how to generate directly simulatable objects whose joints and parts geometry fulfill physical constraints. This may require novel intersection loss and optimization techniques between parts and joints.
Broader Impacts. We do not see any direct negative ethical aspects or societal consequences of our work. Our paper can broadly improve the perception, digitization, and simulation of our physical world.
The authors appreciate the support of the following grants: NSF FRR 2220868, NSF IIS-RI 2212433, NSF TRIPODS 1934960, NSF CPS 2038873 awarded to UPenn; TRI University 2.0 grant, ARL grant W911NF-21-2-0104, and a Vannevar Bush Faculty Fellowship awarded to Stanford University.
[110] Jaehyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. In International Conference on Machine Learning, pages 10362-10383. PMLR, 2022.
[111] Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734, 2022.
[112] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4541-4550, 2019.
[113] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.