Unsupervised Pose-Aware Part Decomposition for 3D Articulated Objects

Articulated objects exist widely in the real world . However , previous 3D generative methods for unsupervised part decomposition are unsuitable for such objects , because they assume a spatially fixed part location , resulting in inconsistent part parsing . In this paper , we propose PPD ( unsupervised Pose-aware Part Decomposition ) to address a novel setting that explicitly targets man-made articulated objects with mechanical joints , considering the part poses . We show that categorycommon prior learning for both part shapes and poses facilitates the unsupervised learning of ( 1 ) part decomposition with non-primitive-based implicit representation , and ( 2 ) part pose as joint parameters under single-frame shape supervision . We evaluate our method on synthetic and real datasets , and we show that it outperforms previous works in consistent part parsing of the articulated objects based on comparable part pose estimation performance to the supervised baseline . 1 INTRODUCTION . Humans are capable of recognizing complex shapes by decomposing them into simpler semantic parts . Researchers have shown that infants learn to group objects into semantic parts using the location , shape , and kinematics as a cue ( Spelke et al. , 1995 ; Slater et al. , 1985 ; Xu & Carey , 1996 ) . Moreover , even very young infants can learn to reason about kinematics using non-sequential single frames ( Shirai & Imura , 2014 ; Kourtzi & Kanwisher , 2000 ) . Although humans can naturally achieve such reasoning , it is challenging for machines , particularly in the absence of a rich supervision . Generative part decomposition and abstraction methods have a long-standing history in computer vision ( Roberts , 1963 ; Binford , 1971 ) . Learning to represent complex target shapes with simpler part components has a wide range of applications , such as structure modeling ( Mo et al. , 2020 ; Roberts et al. , 2021 ) and unsupervised 3D part parsing ( Chen et al. , 2020 ; Paschalidou et al. , 2021 ; Tulsiani et al. , 2017 ) . Previous studies have mainly focused on non-articulated objects . Because they exploit the consistent part location as a cue to group shapes into semantic parts , these approaches are unsuitable for decomposing articulated objects when considering the kinematics of dynamic part locations . In contrast , there exist discriminative approaches targeting man-made articulated objects for part segmentation , in addition to part pose estimation from single-frame input . However , they require explicit supervision , such as segmentation labels and joint parameters ( Yi et al. , 2018 ; Xiang et al. , 2020 ; Li et al. , 2020 ) . Removing the need for such expensive supervision has been an important step toward more human-like representation learning ( Becker & Hinton , 1992 ) . In this study , as a novel problem setting , we investigate the generative part decomposition task for man-made articulated objects with mechanical joints , considering part poses as part kinematics , in an unsupervised fashion . Specifically , we consider the revolute and prismatic parts with a 1 degree-of-freedom joint state as the part kinematics because they cover most of the kinematic types that common man-made articulated objects have ( Xiang et al. , 2020 ; Abbatematteo et al. , 2020 ; Michel et al. , 2015 ) . This task aims to learn consistent part parsing as a generative shape abstraction approach similar to ( Chen et al. , 2019b ) for man-made articulated objects with various part poses from single-frame shape observation . An overview is shown in Figure 1 . This task expands the target of the current generative part decomposition ’ s applications to articulated objects in novel ways , such as part pose consistent part segmentation and part pose estimation . To realize the task , we identify the two challenges ; ( 1 ) for pose-aware part decomposition , the model must consider the kinematics between possibly distant shapes to group them as a single part and ( 2 ) has to disentangle the part poses from shape supervision . A comparison with previous studies is presented in Table 1 . To address these challenges , we propose PPD ( unsupervised Pose-aware Part Decomposition ) that takes an unsegmented , single-frame point cloud with various underlying part poses as an input . PPD reconstructs part-wise shapes transformed using the estimated joint parameters as the part poses . We train PPD as an autoencoder using single-frame shape supervision . PPD employs category-common decoders to capture category-specific rest-posed part shapes and joint parameters . Learning to transform the rest-posed shapes properly disentangles shape and pose , and ( 2 ) restricting the position of the parts by the joint parameters forces shapes in distant space that share the same kinematics to be recovered as the same part . We also propose a series of losses , including an adversarial loss , to regularize the learning process . Furthermore , we employ non-primitive-based part shape representation and utilize deformation by part poses to induce part decomposition , in contrast to previous works that employ primitive shapes and rely on its limited expressive power as an inductive bias . Our contributions are summarized as follows : ( 1 ) We propose a novel unsupervised generative part decomposition method for man-made articulated objects based on part kinematics . ( 2 ) We show that the proposed method learns a non-primitive-based implicit field as the decomposed part shapes and the joint parameters as the part poses , using single-frame shape supervision . ( 3 ) We also demonstrate that the proposed method outperforms previous generative part decomposition methods in terms of semantic capability ( parsimonious shape representation , consitent part parsing and interpretability of recovered parts ) and show comparable part pose estimation performance to the supervised baseline . 2 RELATED WORKS . Existing unsupervised generative part decomposition studies mostly assume non-articulated objects in which the part shapes are in a fixed 3D location ( Tulsiani et al. , 2017 ; Paschalidou et al. , 2020 ; Chen et al. , 2019b ; 2020 ; Deng et al. , 2020a ; Kawana et al. , 2020 ) , or also targeting human body and hand shapes without considering part pose ( Paschalidou et al. , 2021 ) . They induce part decomposition by limiting the expressive power of the shape decoders by employing learnable primitive shapes . Closest work of ours is BAE-Net ( Chen et al. , 2019b ) , whose main focus is consistent part parsing by generative shape abstraction . It also employs a non-primitive-based implicit field as the part shape representation , similar to ours . However , it still limits the expressive power of the shape decoder using MLP with only three layers . In contrast , our approach assumes parts to be dynamic with the consistent kinematics and induces part decomposition through rigid transformation of the reconstructed part shapes with the estimated part poses to make the decomposition pose-aware . A growing number of studies have tackled the reconstruction of category-specific , natural articulated objects with a particular kinematic structure , such as the human body and animals . Representative works rely on the use of category-specific template models as the shape and pose prior ( Loper et al. , 2015 ; Zuffi et al. , 2017 ; Bogo et al. , 2016 ; Zuffi et al. , 2019 ; Kulkarni et al. , 2020 ) . Another body of works reconstruct target shapes without templates , such as by reconstructing a part-wise implicit field given a part pose as an input ( Deng et al. , 2020b ) or focusing on non-rigid tracking of the seen samples ( Božič et al. , 2021 ) . In contrast , our approach focuses on man-made articulated objects with various kinematic structures . Our approach learns the shape and pose prior during training , without any part pose information either as supervision or input , and is applicable to unseen samples . In discriminative approaches , a number of studies have focused on the inference of the part segmentation of the input point cloud and part poses as joint parameters ( Li et al. , 2020 ; Xiang et al. , 2020 ; Abbatematteo et al. , 2020 ) targeting man-made articulated objects . These approaches require expensive annotations , such as part labels and ground-truth joint parameters . Moreover , they require category-specific prior knowledge of the kinematic structure . In contrast , our model is based on generative approach and is category agnostic . Moreover , it only requires shape supervision during training . A very recent work ( Huang et al. , 2021 ) assumes an unsupervised setting where multi-frame , complete shape point clouds are available for both input and supervision signals during training and inference . Whereas our approach assumes a single-frame input and shape supervision , it also works with partial shape input during inference . Note that , in this study , the purpose of part pose estimation is , as an auxiliary task , to facilitate consistent part parsing . It is not our focus to outperform the state-of-the-art supervised approaches in part pose estimation . 3 METHODS . In our approach , the goal is to represent an articulated object as a set of semantically consistent part shapes based on their underlying part kinematics . We represent the target object shape as an implicit field that can be evaluated at an arbitrary point x ∈ R3 in 3D space as O : R3 → [ 0 , 1 ] , where { x ∈ R3 |O ( x ) = 0 } defines the outside of the object , { x ∈ R3 |O ( x ) = 1 } the inside , and { x ∈ R3 |O ( x ) = 0.5 } the surface . Given a point cloud I ∈ RPe×3 as an input , we approximate the object shape using a composite implicit field Ô that is decomposed into a collection of N parts , where Pe is a number of points in the point cloud . The i-th part has an implicit field Ôi : R3 × RPe×3 → [ 0 , 1 ] as part shape and part pose Bi ∈ SE ( 3 ) . We ensure that O is approximated as O ( x ) ≈ Ô ( x | I , { Bi } Ni=1 ) through the losses . An overview of PPD is shown in Figure 2 . PPD employs an autoencoder architecture , and is trained under single category setting . Given a point cloud I , the encoder derives the disentangled shape latent vector zs and the pose latent vectors zp and zp , c . Category-common pose decoder F p , c captures joint parameter biases given zp , c . Instance-dependent pose decoder F p , z models residual joint parameters to the biases given zp . The part-wise category-common shape decoder F s , ci captures category-common shape prior . Given zs and conditioned by F s , ci , instance-dependent shape decoder F s , zi infers residual shape details of the target shape to decode a part-wise implicit field Ôi . We discuss the details about F p , z and F p , c in Section 3.1 , and F s , zi and F s , c i in Section 3.2 . 3.1 PART POSE REPRESENTATION . We characterize part pose Bi by its part kinematic type yi ∈ { fixed , prismatic , revolute } and joint parameters . Each yi is manually set as a hyperparameter . The joint parameters consist of the joint direction ui ∈ R3 with the unit norm and joint state si ∈ R+ . Additionally , the ” revolute ” part has the pivot point qi ∈ R3 . We refer to the joint direction and pivot point as the joint configuration . For the ” fixed ” part , we set Bi as an identity matrix because no transformation is applied . For the ” prismatic ” part , we define Bi = T ( siui ) , where T ( · ) represents a homogeneous translation matrix given the translation in R3 , and si and ui represent the translation amount and direction , respectively . For the ” revolute ” part , we set Bi = T ( qi ) R ( si , ui ) , where R ( · ) denotes a homogeneous rotation matrix given the rotation representation , and si and ui represent the axis-angle rotation around the axis ui by angle si . In human shape reconstruction methods using template shape , its pose is initialized to be close to the real distribution to avoid the local minima ( Kanazawa et al. , 2018 ; Kulkarni et al. , 2020 ) . Inspired by these approaches , we parametrize the joint direction as [ ui ; 1 ] = R ( ri ) [ ei ; 1 ] , where ei is a constant directional vector with the unit norm working as the initial joint direction as a hyperparameter and ri ∈ R3 represents the Euler-angle representation working as a residual from the initial joint direction ei . This allows us to manually initialize the joint direction in a realistic distribution through ei by initializing ri = 0 . Figure 3 illustrates the joint parameters . Based on our observations , we assume that the joint configuration has a category-common bias , while the joint state strongly depends on each instance . This is because the location of each part and the entire shape of an object can constrain the possible trajectory of the parts , which is defined by the joint configuration . To illustrate this idea , we propose to decompose the joint configuration into a category-common bias term and an instance-dependent residual term denoted as ri = rci + r z i and qi = q c i + q z i , respectively . We employ the category-common pose decoder F p , c ( qt ( zp , c ) ) , which outputs { rci | i ∈ Ap } and { qci | i ∈ Ar } , where Ap = { i ∈ [ N ] | yi 6= fixed } , Ar = { i ∈ [ N ] | yi = revolute } , zp , c denotes a pose latent vector , and qt ( · ) is a latent vector quantization operator following VQ-VAE ( Razavi et al. , 2019 ) . The operator qt ( · ) outputs the nearest constant vector cp to the input latent vector zp , c among theNqt candidates . Instead of using a single constant vector , the model selects a constant vector among multiple constant vectors to capture the discrete , multi-modal category-common biases . We also employ an instance-dependent pose decoder F p , z ( zp ) that outputs { si | i ∈ Ap } , { rzi | i ∈ Ap } , and { qzi | i ∈ Ar } . We constrain the possible distribution of the joint configuration around the category-common bias by the loss function explained in Section 3.3 . This constraint incentivizes the model to reconstruct the instance-dependent shape variation by the joint state , which constrains the part location along the joint direction . This kinematic constraint biases the model to represent the shapes having the same kinematics with the same part . Because the previous works ( Kawana et al. , 2020 ; Deng et al. , 2020a ; Paschalidou et al. , 2019 ) do not impose such a constraint on the part localization , learned part decomposition is not necessarily consistent under different poses .

Summary: The proposed paper introduces a pose-aware part-based method for articulated objects. The proposed model takes as input a pointcloud and predicts a set of part poses as implicit functions. The network architecture comprises an encoder that maps the input pointcloud to three latent vectors, a category-common decoder and an instance pose decoder that take the two latent vectors and regress the part pose parameters and a set of shape decoders that are used for generating the part-wise implicit fields. To the best of my knowledge, this architecture is novel and the authors demonstrate that their model yields better segmentation results than various baselines on the Motion dataset. Moreover, the authors evaluate their model wrt the reconstruction accuracy of the final prediction. From the provided analysis it seems to perform worse than existing baselines. Finally, while the authors claim that their model can be used on various generative tasks, they only provide some interpolation results. To summarize, I think this paper is an important step towards pose-aware primitive-based representations, however some parts of the text require further clarifications.