Articulate your NeRF: Unsupervised articulated object modeling via conditional view synthesis

Authors: jianning  deng
kartic  subr
hakan  bilen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 11:28:44

Abstract: We propose a novel unsupervised method to learn pose and part-segmentation of articulated objects with rigid parts.
    Given two observations of an object in different articulation states, our method learns the geometry and appearance of object parts by using an implicit model from the first observation, distills the part segmentation and articulation from the second observation while rendering the latter observation.
    Additionally, to tackle the complexities in the joint optimization of part segmentation and articulation, we propose a voxel grid based initialization strategy and a decoupled optimization procedure.
    Compared to the prior unsupervised work, our model obtains significantly better performance, generalizes to objects with multiple parts while it can be efficiently from few views for the latter observation.

Main paper: 
 introduction: 
 Articulated objects, composed of multiple rigid parts connected by joints allowing rotational or translational motion, such as doors, cupboards and spectacles are ubiquitous in our daily lives. Automatically understanding the shape, structure and motion of these objects is crucial for numerous applications in robotic manipulation [12, 43] and character animation [1, 32] . Many works [10, 30, 34] that focused on this problem use groundtruth 3D shape, articulation information, and/or part segmentation to learn articulated object models but acquiring accurate 3D observations and manual annotations is typically complex and too expensive for building real large-scale datasets.In this paper, we introduce a novel unsupervised technique that learns part segmentation and articulation (i.e., axis of movement, and translation/rotation of each movable part) from two sets of observations without requiring groundtruth shape, part segmentation or articulation. Two sets contain images of the same object from multiple viewpoints in two different articulation states. Our key idea is that articulation changes only the poses of the object parts but not their geometry or texture. Hence, once the geometry and appearance are learned, one can transform to another articulation state given the part locations and target articulation parameters.Building on this idea, we frame the learning problem as a conditional novel articulation (and view) synthesis task (see Fig. 1(a) ). Given a source observation, multiple views of an object in one articulation state, we first learn the object's shape and appearance by using an implicit model [21] and then freeze its weights. Next we pass the target observation, multi-view images of the same object in a different articulation state, to a tight bottleneck that distills part locations and articulations. We constrain our model to assign each 3D coordinate that is occupied by the object to a part and to apply a valid geometric transformation to the 3D coordinates of each part through ray geometry. The predictions of part segmentation and articulation, along with the target camera viewpoint, are passed to the implicit function and its differential renderer to reproduce the target observations (see Fig. 1(b) ). Minimizing the photometric error between the rendered and target view provides supervision for learning part segmentation and articulation. However, joint optimization of these intertwined tasks is challenging and very sensitive to their initialization. To address the optimization challenge, we propose an initialization strategy using an auxiliary voxel grid, which provides an initial estimate for moving parts by computing the errors in the foreground masks when rendering target views for the source articulation. Additionally, we introduce a decoupled optimization procedure that alternates between optimizing the part segmentation on the photometric error and the articulation parameters on the foreground prediction error.The key advantage of our method, compared to other unsupervised articulation prediction methods [11, 17] , is its stable performance across different object and articulation types, its ability to learn from few target views and to model multiple moving parts. Thanks to the stage-wise training, we achieve high-quality object geometry and appearance by using the well-optimized implicit models [2] . Since the part segmentation and articulation parameters form a small portion of the total weights, along with the initialization and decoupled optimization strategies, our method efficiently learns them from few target views, unlike the most relevant work [17] that jointly optimizes multiple part-specific implicit functions from scratch. 
related work: 
 Analysis of articulated objects The analysis of articulated objects typically involves segmentation of movable parts, and estimating their attributes such as position and direction of joint axes, rotation angles, and translation distances. Prior works study articulated objects using 3D input such as meshes [31, 22] , point clouds [45, 44, 37, 14] or RGBD images [18, 6, 40, 13] that are error-prone and labor-intensive to collect in real-world scenarios. Recent works [10, 30, 34] that use RGB images to segment 3D planar object parts and estimate their articulation parameters simultaneously require ground-truth labels for segmentation and 3D articulation.Neural implicit models [20, 26, 21] that are originally designed to model static objects in a 3D consistent way have been extended to articulated objects by multiple recent works [24, 36, 11, 16, 38, 17, 40] . A-SDF [24] learns a disentangled latent space for shape and articulation to synthesize novel shapes and unseen articulated poses for a given object category. CARTO [7] extends A-SDF to multiple object categories and uses a stereo image pairs as input. Similarly, CLA-NeRF [36] learns to perform joint view synthesis, part segmentation, and articulated pose estimation for an object category from multiple views. NARF22 [16] learns a separate neural radiance fields (NeRF) for each part and composes to render the complete object. Unlike our method, both A-SDF and CARTO require 3D shape and articulation pose supervision, CLA-NeRF requires part segmentation labels and is limited to modeling only 1D revolutions for each joint, NARF22 relies on groundtruth part labels and articulation pose.Most related to our work, DITTO [11] , PARIS [17] and Weng et al. [40] aim to estimate part segmentation and articulation pose without labels using from an observation pair of an object in two different articulations. DITTO uses a pair of point cloud as input, can only model shape, whereas both PARIS and our method uses pairs of multi-view images, and to model both geometry and appearance. PARIS adopts the dynamic/static modelling of [42, 46] , learns a separate NeRF for the dynamic and static parts. They are composited using the estimated relative articulation to render the observations with the different articulation. Similarly, [40] first reconstructs the object in two articulation states with two sets of RGBD images. Later, the part-level segmentation is performed based on image registration results. While our method is also based on the same analysis-by-synthesis principle to supervise the training, it differs from previous work in several key ways. Compared to DITTO [11] and Weng et al. [40] , our method relies completely on 2D input data. And compared to PARIS [17] , our method involves only a single NeRF that is learned on multiple views of an object instance in a fixed articulation pose. Once the NeRF is learned, we freeze its parameters, and learn a segmentation head and articulation to selectively deform the rays while rendering views of different articulations. This means our model's size remain nearly constant when the number of parts increases, which, combined with our two step optimization strategy, leads to more stable and data-efficient learning, yielding significantly better performance. Furthermore, we show that our model goes beyond modeling a single moving object part as in PARIS, and successfully learn multiple moving parts.Deformable NeRF There exists several techniques [27, 29, 35, 28, 5 ] that model differences between subsequent video frames of a dynamic scene through a deformation field in 3D. While these techniques are general could be used in modeling articulated objects in principle, the deformation field does not provide explicit predictions for part segmentation and articulation. Prior methods that focus on modeling specific articulated and deformable objects such as human body [39, 33, 8, 9, 25] and four-legged animals [41] leverages specialized 3D priors [19] that are not applicable to arbitrary object categories.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Details about discussions about limitations can be found in Sec. 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
review: nerf: 
 Given a set of images I of an object, a NeRF function [21] maps a tuple (x, d), where x ∈ R 3 is a 3d position and d ∈ R 2 is a direction, to an RGB color c, and a positive scalar volume density σ. The model outputs volume density σ at the location x along with a latent feature vector z which is then concatenated with the viewing direction d and fed into an MLP to estimate the view-dependent RGB color c. With a slight abuse of notation, we also use σ and z as functions σ(x) and z(x) respectively and color as a function with c(x, d). That is, we use boldface to denote vectors.NeRFs are trained by expressing the color at each pixel of images in the training subset. The color at a pixel C is estimated as C N via the volume rendering equation as a sum of contributions from points along the ray through the pixel, say r(t) = o + td where o is the origin of the ray (usually the point of projection of a camera view) and t is a scalar. Points x i , i = 1, . . . , n are sampled along the ray asx i = o + t i d.The estimated color is given byEQUATIONwhere we use the shorthand notation c r i and σ r i to denote c(x i , d) and σ(x i ) respectively, and δ i = t i+1 -t i is the distance between samples. And the opacity value is calculated asO N (r) = n i=1 T r i (1 -exp(-σ r i δ r i ).The parameters of c and σ are obtained by minimizing the photometric loss between predicted color C N (r) and the ground truth color C(r) asL photo = r∈R ||C N (r) -C(r)|| 2 2 .(2)Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: In this paper, we focus more on the practical results, which is shown in Sec. 5.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
method: 
 Let I and I ′ be two observations of an object with k (known a priori) rigid parts in articulation poses P and P ′ respectively. Each observation contains multiple views of the object along with the foreground masks. We call I the source observation and I ′ the target observation. P (resp. P ′ ) is a pose tensor composed of P ℓ ∈ SE(3), ℓ = 1, . . . , k (resp. P ′ ℓ ) transformations as 4 × 4 matrices corresponding to the local pose of each of the k parts. Our primary goal is to estimate a pose-change tensor M composed of M ℓ ∈ SE(3), ℓ = 1, . . . , k so that P ′ ℓ = M ℓ P ℓ . We solve this by starting with the construction of a NeRF from I, as described in Sec. 3 along with the efficient coarse-to-fine volume sampling strategy in [2] , followed by a novel modification to build awareness at the part-level. We use this modified parameterization to optimize an auxiliary voxel grid corresponding to the parts via pose estimation and part segmentation pipelines. Once we have identified the parts and their relative transformations, we are able to render novel views corresponding to articulation pose P ′ by transforming view rays suitably by M -1 ℓ .Once a NeRF function, which we call 'static NeRF', is learned over I, we freeze its parameters and append a segmentation head s to it towards obtaining part-level information. s is instantiated as 2-layer MLP, and maps the latent feature vector z(x) to a probability distribution over the k object parts. We denote the probability of a 3D point x to belong object part ℓ = 1, . . . , k as s ℓ (z(x)).If the segmentation class-probabilities s ℓ and pose-change transformations M ℓ are known, then the object in an unseen articulation pose can be synthesised by suitably transforming the statically trained NeRF without modifying its parameters and then compositing the individual part contributions. To model pose change in each part, we use a different ray, virtual ray associated with each partEQUATIONand the final rendered color is:EQUATION)whereT r i = i-1 j=1 exp - k ℓ=1 s r ℓ ℓ (x j ) σ r ℓ (x j ) δ r ℓ jis the transmittance sum. Compared to the original formulation in Eq. ( 1), it can seen as the density replaced by a part-aware density which is the product of s and σ. Note that we made a similar modification to the efficient volume sampling strategy of [2] and refer to the Appendix A.1 for more details.Since the groundtruth part segmentation and articulation are unknown, one could optimize the parameters of s and M on the target observation such that the photometric error in between the partaware rendered views and I ′ . Note that C P (r) is function of M (through r ℓ in Eq. (3)) and s, they can be jointly optimized through backpropagation. However, jointly solving these two intertwined problems through rendering is challenging and highly sensitive to their initial values, as their solutions depend on each other. Hence, we introduce an alternating optimization strategy next.To learn the part segmentation and pose-change, we first introduce an auxiliary voxel grid that assigns each 3D coordinate either to a background or a part. Once the voxel is initialized, our training iterates multiple times over a three-step procedure that includes optimization of M, s, and refinement of the voxel grid entries.Here our key idea is to find the pixel differences between the source and target observations by rendering a target view v ′ by using the static NeRF, and label the 3D locations of the different pixels as the movable parts to provide a good initialization for estimating the articulation (see Fig. 2 ). To this end, we first build a 3D voxel of the static NeRF by discretizing its 3D space into 128 bins along each dimension. To compute the voxel entries, we query the density value from the static NeRF at each voxel coordinate and binarize it into occupancy values with criteria exp(-σ(x)δ) > 0.1. Next, we use the static NeRF to render opacity and depth images for the viewpoints v ′ . The pixels rendered as foreground in rendered opacity but not in the corresponding foreground mask of I ′ will be collected. For those pixels, we further compute their depth by accumulating their density values along the corresponding ray, and use their estimated depth values to tag the occupied voxel entries either with static or dynamic parts. In the case of multiple moving parts, which is assumed to be known, we perform a clustering step [4] to identify k -1 clusters corresponding to moving parts, as we assume that 1 part of the object is static without loss of generality. Finally, we gather the voxel coordinates that is assigned to part ℓ to form a matrix of 3D coordinates X ℓ . Step 1: Optimization of M ℓ As depicted in the right side of Fig. 3 , we project the 3D coordinates X ℓ for part ℓ onto the image plane for each camera viewpoint v ′ included in I ′ by using 3 × 4 dimensional intrinsic camera matrix K as U ℓ = KM -1 ℓ v ′ X ℓ , and denote the projected 2D point matrix on the image plane as U ℓ . On the left side of Fig. 3 , we collect 2D coordinates from the overlap region between rendered opacity and target image I ′ to obtain U ′ . Then we concatenate U ′ and part-specific matrices U ℓ to obtain U . We want to optimize U to be the same as the target coordinates F , which is obtained by gathering the 2D coordinates from the foreground mask of I ′ . Note that both U and F are matrices where each row corresponds to a 2D image coordinate where F has significantly more rows, as U is obtained from a coarse voxel grid. Then we minimize the 2D Chamfer distance between U and F :EQUATIONIn practice, we only project X ℓ for the moved parts, and stack them with 2D image coordinates of the foreground pixels in both the rendered image and the groundtruth image, as illustrated in Fig. 3 .Step 2: Optimization of s ℓ Once we obtain the solution of the pose-change M * from Step 1, we plug in it to the ray deformation in Eq. ( 3) and render each view in I ′ in Eq. ( 4). Then we compute photometric loss between the rendered views and I ′ , and minimize it with respect to the parameters of s only. In the case of multiple moving parts, we initialize the parameters in s using X ℓ for supervision. Please refer to Appendix A.4 for more details.Step 3: Voxel grid refinement The initial part segmentation estimates in the voxel grid are often sparse and noisy due to inaccurate density estimation in NeRF and misassignment of pixels around the foreground and part boundaries. To obtain a more accurate voxel grid, we follow the same steps in the initialization stage except that the 3D coordinates are assigned to the parts based on the predicted label from segmentation head s. We denote the new 3D coordinates as X * ℓ . For the consistency, we accept the voxels to X * ℓ only when the predicted part labels agree within a 3D local neighborhood of X ℓ . The neighborhood is defined as being within the distance of one grid cell at a resolution of 128. Note that we only perform the refinement step, after approximately 2k iterations to ensure that the segmentation head produces confident results. Then we also increase the voxel resolution to 256 for each dimension.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Yes, the details about our method can be found in Sec. 4. 
dataset: 
 We evaluate our method on the synthetic 3D PartNet-Mobility dataset [43, 23, 3] . While the dataset contains more than 2000 articulated objects from 46 different categories, we use a subset of the dataset with 6 shapes that was used in [17] . For a fair comparison, we downloaded the processed dataset from [17] which contains 2 sets of 100 views along with their foreground masks, each with a different articulation, and also groundtruth part segmentation labels, for each shape. In addition, we select 4 additional shapes, each with two moving parts, and apply the same data generation process to them. Additionally we collected images of a toy car with a handheld device, with camera viewpoint estimated from kiri engine application [15] .We train the static NeRF on 100 views from the first observation, train the part segmentation and articulation on 100 views from the second observation. We report the performance of our method for varying number of views from the second observation in Tab. 4.Following [17] , we report performance in different metrics for pose estimation, novelview/articulation synthesis, and part segmentation. Pose estimation: To report articulation pose performance, we report results in i) direction error e d that measures the angular discrepancy in degrees between the predicted and actual axis directions across all object categories, ii) position error e p and geodesic distance e d for only revolute objects to evaluate the error between the predicted and true axis positions and the error in the predicted rotation angles of parts in degrees, respectively, iii) translation error e t to measure the disparity between the predicted and actual translation movements for prismatic objects. Novel-view and -articulation synthesis: We evaluate the quality of novel view synthesis generated by the models using the Peak Signal-to-Noise Ratio (PSNR) where higher values indicate better reconstruction fidelity. Part segmentation: We use mean Intersection over Union (mIoU) on the rendered semantic images to evaluate the accuracy of part segmentation, which is tightly related to the rendered image quality of objects in different articulation states. The ground truth segmentation is generated within the Sapien framework in [43] . Finally, due to the lack of groundtruth segmentation label and articulation in the real data, we only report PSNR and provide qualitative evaluation.Baseline We compare our approach with the state-of-the-art unsupervised technique, PARIS [17] which constructs separate NeRF models for each part of an articulated object and optimizes motion parameters in an end-to-end manner. However, it is limited to two-part objects with only one movable part. As the authors of PARIS do not report their results over multiple runs in their paper, we use their official public code with the default hyperparameters, train both their and our model over 5 random initializations and report the average performance and the standard deviation. We use 2 sets of 100 views for each object to train the models. More implementation details can be found in the supplementary material. We would like to note that the performance of PARIS in our experiments significantly differ from the original results despite all the care taken to reproduce them in the original way 1 .Part-level pose estimation For part-level pose estimation, we provide quantitative results in average performance over 5 runs and their standard deviations in Tab. 1, and a qualitative analysis in Fig. 6 .The results indicate that our method consistently achieves lower errors across most evaluation metrics compared to PARIS, except the joint position error e p where the differences are negligible, within the 10 -3 range. Notably, the performance of PARIS on the stapler and blade exhibits significantly higher errors. We observe that PARIS fails to converge to a good solution in all 5 runs for these objects. As shown in Fig. 6 , PARIS fails to accurately segment the parts in the stapler and blade, which can also be easily identified in the novel articulation synthesis experiments in Fig. 6 . Poor part reconstruction in PARIS results in inaccurate part-level pose estimations. Additionally, the lower standard deviation across all reconstructed objects for our method indicates its stable learning and ability to work for different object instances. We attribute the stability of our method to the decoupled optimization along with the stage-wise training, which contrasts with the joint optimization approach used by PARIS. As our model builds on a static NeRF that achieves high quality novel view synthesis for one observation, the rendering quality is largely preserved after the second stage. We provide more detailed analysis in Appendix A.3. Additionally, benefiting from accurate pose estimation via a decoupled approach, our method achieves more robust and precise part segmentation, as depicted in Fig. 4 . Here, our method consistently delivers accurate segmentation results for challenging objects such as the blade, stapler, and scissors, where PARIS struggles with accurate part reconstruction. In the other instances including the laptop, storage, and oven, our method achieves visibly better results.Evaluation on objects with multiple movable parts A key advantage of our model is its ability to model objects with multiple moving parts. For such objects, we report pose estimation results in Tab. 3, qualitative for part segmentation in Fig. 5 and novel articulation synthesis in Fig. 6 . As in the single part moving object experiments, our method delivers similar performance for the multi-part objects. Notably, we observed a marginally higher joint direction error for glasses, which we attribute to the thin structures such as temples and failure to segment them accurately which can be possibly improved by using higher resolution images. We assessed the effectiveness of the proposed decoupled pose estimation of M ℓ (DP) and iterative refinement of X ℓ (IR) on the 'fridge' object. When DP is disabled, the segmentation and articulation are simultaneously learned. In contrast, disabling IR maintains the initial X ℓ for pose estimation. The results in Tab. 5, particularly the first row, demonstrate that joint optimization without DP inaccurately predicts the articulated pose, treating the entire object as static. While enabling DP improves the performance, as shown in the second row, the performance is still poor due to the noisy initial values compared to our full model.Additionally, we evaluated the impact of the number of views in the target observation for training. We train our model on the randomly subsampled views for multiple runs and averaged their performance (see Tab. 4). Results show a significant drop in pose estimation and rendering quality with 4 images or less. Notably, even with only 8 images, our approach surpasses the performance of PARIS trained with 100 images (indicated with subscript P in Tab. 4). This result clearly show that our method are more robust against fewer viewpoints from the target viewpoints and allows efficiently learning 'articulate' a pretrained NeRF from few target views only. The qualitative evaluation can be found in Fig. 7 . Novel articulation synthesis in different views are showed in two sides of Fig. 7 . The GT is shown in pose P ′ while the segmentation image is the toy car in pose P. The average PSNR we obtain from the static NeRF is , and the PSNR for 28.19, while the PSNR for pose P ′ is 24.32. More detail results for segmentaion and articulation interpolation are also presented in Fig. 7 .Our method has few limitations too. As we use rendering to supervise our segmentation and pose, our method may fail to segment very thin parts or small movable parts when the rendering error is small. Our method is less stable for multipart objects compared to two-part objects. When the object parts are nearly symmetry, our method may fail to find the correct articulation and choose another articulation that produces similar rendering. We provide examples of such failure cases in Appendix A.3. Additionally, it also inherits the limitations of the implicit models that are built on such as failure to model transparent objects, and inaccurate 3D modeling when the viewpoint information is noisy. Finally, our method is limited to articulated objects with rigid parts and cannot be used to model soft tissue deformations.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Code and data will be released upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: This project conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
conclusion: 
 In this study, we tackled the challenges of self-supervised part segmentation, articulated pose estimation, and novel articulation rendering for objects with multiple movable parts. Our approach is the first known attempt to model multipart articulated objects using only multi-view images acquired from two arbitrary articulation states of the same object. We evaluated our method with both synthetic and real-world datasets. The results suggest that our method competently estimates part segmentation and articulation pose parameters, and effectively renders images of unseen articulations, showing promising improvements over existing state-of-the-art techniques. Further, our real-world data experiments underscore the method's robust generalization capabilities. At last, the code and the data used in this project will be released upon acceptance. However, the reliance on geometric information from moving parts for articulation pose estimation poses challenges in modeling highly symmetrical objects. Future work could improve our method by incorporating both appearance and geometric data into the pose estimation process, potentially enhancing accuracy and applicability. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: We don't discuss the social impact of this project in the paper.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The proposed technique could be potentially used to understand articulated objects for their robotic manipulation in future. Optionally include supplemental material (complete proofs, additional experiments and plots) in appendix. All such materials SHOULD be included in the main submission.To achieve part-aware composite rendering for articulated objects, we also need to modify the proposal network for correct sampling along the ray r ∈ R ′ . In the composite rendering, the proposal network is now required to produce the similar distribution of the weights for samples along the ray. Thus, following the similar design in the part-aware NeRF, we append an extra segmentation field to the proposal network. Now the valid density for each samples becomes segmentation-mask density as s r ℓ ℓ (x j )σ r ℓ . The transmittance are then calculated using Eq. ( 4) without the color term, which will be later used for online distillation. Please refer to the original paper [2] for more details about the online distillation. During the training of part-aware NeRF, the original parameters in the proposal network will be frozen and only updates the segmentation field.Here we also provide the 5-time best comparison between ours and PARIS. Besides, the reported performance for PARIS is also provided in this subsection. From Tab. 6, we can see that the original reported performance from PARIS are exceptionally better compared to the reproduced ones, even though with the same data and the same configurations. If we only focus on the 5-time best comparison between reproduced PARIS and ours, we can see that we achieve comparable performance. Another thing we can notice is that for prismatic objects, while PARIS has much better estimations on joint axis direction. However, our method later show much better estimation for estimating the moved distance for the dynamic parts. While one thing to notice is that PARIS failed to reconstruct the stapler for all 5 runs in our experiment.Details about the limitations and failure cases of our methods.In our decoupled optimization strategy, the articulated pose estimation relies exclusively on the geometric information from the moving parts. As a result, our method faces challenges with highly (c) Artifacts for thin parts, the left one is the groundtruth, the right rendering result. symmetrical components. For example, in the case of a folding chair Appendix A.3, although the segmentation accurately identifies the chair's seat, the pose estimation mistakenly flips the seat by 180 degrees, resulting in the seat being oriented with the bottom upwards as shown in Fig. 8(b) . Additionally, our method may encounter difficulties when the articulation motion is minimal, which can lead to insufficient pixel differentiation in images across different articulation poses for initial model estimation. In such cases, both the pose estimator and the iterative updates of X ℓ may underperform or even fail. We also notice that the proposal network have difficulties handling very thin parts in the novel articulation synthesis. As we can see in Appendix A.3, some artifacts appeared at the edge of the frame and the temples for the glasses as in Fig. 8(c ), which should be ideally masked out without any opacity and color for those pixels.Furthermore, the reliance on a pre-trained static NeRF limits the upper boundary of our method's performance in rendering novel articulations. Addressing these limitations could significantly enhance the robustness and applicability of our strategy in handling complex articulated objects. For further analysis of this, please refer to . As detailed in Sec. 4.2, we perform optimization on M ℓ for 4000 iterations and on s ℓ for 2000 iterations for all evaluated objects. Checkpoints are saved following Step 3. We conduct this process through 5 cycles for objects with a single moving part and 6 cycles for objects with multiple moving parts. Checkpoints with best PSNR during validation will be used for test.During the first step of M ℓ optimization, we begin with a learning rate of 0.01, which linearly decays by 0.5 every 500 iterations. We accumulate gradients from 8 viewpoints to simulate a batch size of 8, initializing M ℓ identically in the first cycle and using the previously estimated M ℓ for subsequent cycles.In the second step of s ℓ optimization using the Adam optimizer, the initial learning rate is set at 0.01 and linearly decays by a factor of 0.01 every 100 iterations. For multiple moving parts, initialization involves training the segmentation head s using pre-assigned labels on X ℓ , and querying predictions for x ∈ X ℓ . Cross-entropy loss optimized over 1000 iterations with a learning rate of 1e -3 shapes the learnable parameters in s.Our experiments, requiring around 16 GB of VRAM, complete in approximately 30 minutes on a single RTX 4090 GPU for a single object. As for the training of static NeRF, it takes about 10 minutes with less than 10 GB of VRAM. The estimated total GPU time for this project would be about 2 GPU months.We also provide an ablation study to investigate the performance drops for conditional novel articulation synthesis compared to the pre-trained static model. As shown in Tab. 7, the performance drop is subject to different category of objects ranging from smallest 13.3% for the fridge and blade to 31.4% laptop. We suspect the significant drops for laptop is caused by the deteriorate performance of the proposal network on the thin laptop screen. Visualizations can be found in Fig. 9 . The results indicate that our method can benefit from the high quality of appearance reconstruction from the pre-trained static NeRF.Here we show more visualization for qualitative evaluations. Besides, additional animated images for the real-world toy car is also provided in the supplementary materials. 8 : Quantitative evaluation for the same door instance. We report average performance over 5 runs. For the description of e d , ep, eg, P SN R and mIoU , please see Section 5.1 in the submission. Our method outperforms PARIS consistently in all the metrics.Input for P ′ Rendered for P ′Seg. GT for P Seg. Pred. for P Seg. GT for P ′ Seg. Pred. for P ′Figure 13 : Here we show qualitative results for a 'door' instance along with its frame. In contrast to the instances in our submission, the static part (frame) is smaller than the moving part (door). Given two sets of views in the articulation P and P ′ , we provide the original input images and their rendering in the respective articulations, ground-truth (GT) and predicted part segmentation results. Our method achieves faithful rendering results in different articulations with minor artifacts and accurate part segmentation.Input for P Rendered for P Input for P ′ Rendered for P ′ Seg. GT for P Seg. Pred. for P Seg. GT for P ′ Seg. Pred. for P ′ Figure 14 : Here we show qualitative results for a toy robotic arm with 2 static joints (grey and blue) and 3 moving joints (red, yellow and green). Unlike the objects in the Sapiens dataset that have independently moving parts, the joints of the robotic arm are linearly connected and forming a chain structure. Note that our method independently estimates the 6 DoF SE(3) transformation for each moving part between articulation pose P and P ′ . Given the kinematic structure, the local transformations could be computed. Our method faithfully renders the robotic arm in different poses with minor artifacts, and segments its parts accurately. Note that the static parts are labeled as one part.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Similar problems have been pointed out and the reproducibility of these results has been acknowledged as challenging by the authors in issue 1 and issue 


Decision: Post rebuttal, reviews were split. Three reviewers were in favor of accepting the paper. They argued the work tackled an important and challenging problem, used a method that would be a good contribution, thought the experiments demonstrated the value of the work, and overall believed that the paper would be a good addition to the overall literature. 

One reviewer was strongly opposed to the work, claiming that the method wasn't a substantial enough contribution to the literature, and that it should be compared with more work. One concern was the paper's baseline results differed from the published results. The AC is convinced by the authors' explanation of that difference is by the instability of the baseline methods (which was noted in the NeurIPS submission, in the github issues for the paper in question, and acknowledged in other papers too). The authors' approach of running methods five times and reporting the mean and standard deviation seems a sensible mitigation. A second point of disagreement was a comparisons with a CVPR 24 paper that appeared on Arxiv before the NeurIPS submission deadline. The AC agrees with the authors and one of the other reviewers that the paper is concurrent, following the NeurIPS author FAQ's guideline of 2 months. Comparisons might be a nice addition for the paper's impact, but are not necessary.

Overall, the AC agrees with the reviewers recommending acceptance. This decision was based on the paper, the reviews, the rebuttal, and the discussion after the rebuttal. The AC asks the authors to incorporate their responses to the reviewers into the final version of the paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a method for modeling articulated objects with neural radiance fields (NeRF). It employs a stage-wise training schema, first building a NeRF of the object in a reference configuration. Then a segmentation in parts and relative pose-changes are learned in an alternating fashion. A modified rendering equation and a corresponding volume sampling strategy are introduced that take into account the rigid deformations of the object parts.
strength_weakness: strength_weakness: The text contains a few typos and other errors, like incomplete sentences, missing verbs etc. Some editing of the text is required. 
Additionally, the clarity of the presentation can be improved. It is true that the stage-wise learning schema is somewhat involved, hence some additional effort should be devoted in explaining the steps as clear as possible, especially in Section 4.2. One suggestion is to provide a more detailed explanation of Figure 3, and provide dimensions of matrices U, F etc. Also, the formula in L.167-168 does not seem to be correct, especially regarding how the viewpoint v' is used. Should v' be an argument of a function?

Another aspect that is missing regards the efficiency of the proposed method. It is important to discuss training and inference times in comparison to other baselines. On a similar note, a discussion on how the number of iterations used at each stage affects the final result could be included in the ablations.

Regarding experimental evaluation, ideally, comparison could also consider state-of-the-art articulated object reconstruct methods which are not based on NeRFs, such as  [R1] and [R2].

Finally, not enough details are shared with respect to the models used, making reproducibility of the approach challenging.

[R1] Kawana & Harada, Detection based part-level articulated object reconstruction from single RGBD image. NeurIPS 2023

[R2] Kawana, Mukuta, & Harada, Unsupervised pose-aware part decomposition for man-made articulated objects, ECCV 2022

### Minor comments
- L.10: incomplete sentence
- L.71: needs rephrasing
- L.105: "And the opacity value ..." incomplete
- L.199: It is fine that the same subset of 3D PartNet-Mobility dataset used in [15] is considered for the comparison, but it would be interesting to report results on a wider selection of objects
- Section 5: the order in which figures are presented is not following the text, this creates some confusion to the reader
- L.270: "performs consistently performs"
- L.292: "presented in second and third row" does not follow the structure of the figure
questions: questions: - How does the number of iterations used at each stage affect the final result? Is the method sensitive to these hyper-parameters?
- Can the method handle joints with more than one degree of freedom?
limitations: limitations: Limitations are discussed in the text. I think it would be important to discuss also challenges and limitations of multi-part object reconstruction.
paper_summary: paper_summary: This paper presents an unsupervised framework that jointly learns articulations and part segmentations of objects with rigid parts from multi-view images.

Specifically, they proposed a two-stage approach. In the first stage, a static NeRF is fitted to one of the object states. In the second stage, the optimization alternates between optimizing part assignment and part pose estimations. The key to their approach is a 3D voxel grid heuristic that helps initialize the part assignment.

The proposed method shows promising quantitative and qualitative results in view synthesis, pose estimation, and part segmentation compared to the previous method PARIS. Notably, the proposed method also shows more consistent results across multiple runs.
strength_weakness: strength_weakness: While the approach looks promising, it does have some room for improvement
- As can be observed in the supplementary video, the segmentation is still noisy, containing floaters that have nothing to do with the moving part.
- Using pixel difference as the heuristic for tagging moving parts is potentially problematic. For example, in a real-world setting, the constantly changing environment light produces color differences across views/states, which can greatly affect the tagging accuracy. 
- The examples presented still have a rather simple articulated structure (i.e., no complicated kinematic chains that connect multiple parts). Note that this is an open problem/common issue not specific to the proposed method.
- The approach is limited to rigid deformation (as discussed in L303-304). Learning a deformation field may alleviate this problem, but this is out of the scope of this work.
questions: questions: Below are some questions that I have:
- A concurrent work [1] tackles the same topic and similarly achieves impressive results. It would be nice if the authors briefly discuss the work (what’s in common, and what set the two apart). Note that [1] was not yet published at the time of NeuRIPS submission, so this is just a nice-to-have discussion, and will not affect the final rating.

Typo:
- L212: geodesic distance should be e_g?

[1] Neural Implicit Representation for Building Digital Twins of Unknown Articulated Objects, CVPR 2024
limitations: limitations: The paper adequately addresses the limitations in Section 5.5 and Section 6. It is also pretty awesome that the paper also presents failure cases and further analysis in Supplementary (A.3)
paper_summary: paper_summary: This paper introduces a novel method for decomposing articulated objects and predicting their articulation. The pipeline is trained without supervision. Initially, a "static NeRF" of the object's initial state is obtained. The method then employs part-aware rendering to optimize the pose-change tensor and object segmentation in a decoupled manner. Finally, it reconstructs the voxel grid and performs refinement to achieve high-quality results.
strength_weakness: strength_weakness: 1. There are still many artifacts that can be seen in the visualization results
2. The method only works in limited cases. Most of the examples have one or two joints and a large static part.
questions: questions: 1. To optimize the pose-changing tensor, it requires the difference between the target and initial views. What if the object does not have static parts? For example, when scissors cut something, both parts are moving.
2. The results presented in PARIS are not as poor as those in this paper, which the authors should address.
limitations: limitations: The paper discusses the limitations and shows some failure examples.
paper_summary: paper_summary: This paper presents an unsupervised method to learn the pose and part-segmentation of articulated objects with rigid parts using conditional view synthesis. Their approach learns the NeRF of both states and extracts 3D voxels for optimizing the pose and segmentation.
strength_weakness: strength_weakness: Firstly, my main concern is about the novelty and contribution of this work. 

Compared to the work PARIS, which this paper frequently references, I did not find any significant innovative points. In fact, the method appears to have regressed. This method separates the optimization of pose and segmentation, first optimizing the pose and then optimizing the segmentation. In my opinion, this is not as effective as jointly optimizing them, since the segmentation results can also affect the pose optimization results. However, the 3D voxel approach used in this paper is not differentiable (Lines 150-151), so it cannot jointly optimize both. 

And the voxel grid refinement operation seems more like a simple engineering post-processing. 

Additionally, I have some issues with the experimental results. The values reported for the existing method PARIS in this paper differ from the results reported in the original paper. Have the authors considered comparing their method with other existing methods?
questions: questions: Please see the Weakness section.
limitations: limitations: I agree with the limitation section, but, I find the method for finding U (Line 170) to be insufficiently robust. If an issue arises, all subsequent operations will fail.

Comments: title: Post-rebuttal comments
Thanks for the reply
Thanks for the reply
Thanks for the responses.
Feedback
comment: The author responses have addressed my main concerns and clarified some important aspects regarding notation, comparison to prior work and implementation details. For these reasons, I increase my rating to 6.
1. **Sufficient innovation and contribution compared to existing works:** We refer the reviewer to the originality point in the reviewer guidelines of the conference and invite the reviewer to provide a more substantiated comment on their understanding of “significant innovation”.
2. **Comparative experimental results:** The reviewer missed the point in our response. We would like to reiterate that our method without the decoupled pose estimation (DP) and iterative refinement (IR), which is reported at the first row of Table 5, is the joint optimization baseline that is requested by the reviewer. This baseline jointly optimizes part segmentation along with part transformation parameters, and it does not use $X_{\ell}$. The results in Table 5 show that the joint optimization strategy performs poorly and the proposed alternating strategy is significantly better.
3. **R2, I do not consider this work to be concurrent, as it was already published before your submission:** R2 was published in the CVPR proceedings on 17 June after the NeurIPS submission deadline. The reviewer might be referring to the day that it became public in the arXiV, which is 1 April, 1 month and 21 days before our submission. Such a short duration makes it a concurrent work. The same opinion was shared in the review of Reviewer DWX2. We already discussed the differences to R2 in our response and indicated that this method uses two sets of RGB-D views as input and relies on the accurate depth information to estimate 3D object structure unlike our method that uses only RGB views. They do not provide results with only RGB views. Hence it is not comparable to ours.
4. **Both R2 and the works they compared can be used as benchmarks for comparison:** We already compared our method to the prior works (Ditto[R3] and PARIS) that use the same input type and supervision. Unlike ours, Ditto [R3] relies on accurate point cloud input which is significantly easier than performing the same task with multi-view input. We already cited and discussed [R3] in our submission.

[R3] Jiang, Zhenyu, Cheng-Chun Hsu, and Yuke Zhu. "Ditto: Building digital twins of articulated objects from interaction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
Thank you again for your review!
Thank you for your suggestion! We will improve the Figure 2 in the final version for better clarity.
Thanks for the authors' response. I do not have further concerns and I would like to increase my rating.
Thanks for addressing and clarifying the concerns I have.

My apology for misunderstanding the pixel difference computation -- certainly using opacity only could avoid the above-mentioned problem. Just one little comment regarding the pixel-difference thing: it would be great if the pixel-difference part in Figure 2 could be improved. Figure 2 depicts that RGB image and foreground mask are used for pixel difference computation. Perhaps it would be better to explicitly mention/depict opacity here, so it would be more accurate/less ambiguous, and align with the text better.

Other than that, all my concerns are addressed or properly discussed.
I still do not believe that this work offers sufficient innovation and contribution compared to existing works, and there are significant issues with the experimental section.

Regarding your statement: "we disagree that joint optimization is guaranteed to obtain better results. It is well known that simultaneously updating intertwined parameters can be challenging and result in poor solutions," I expect to see comparative experimental results or valid references to support this claim rather than an assertion without evidence. Furthermore, if both the segmentation process and the transformation process are differentiable, we can still iteratively update both. I do not believe that replacing a differentiable process with a non-differentiable one, relying on iterative approximation is a good idea.

The voxel grid refinement operation is not a one-time post possessing, while it is still a simple engineering add-on.

As you pointed out in your CVPR2024 review R2, I do not consider this work to be concurrent, as it was already published before your submission. Both R2 and the works they compared can be used as benchmarks for comparison. Since you found that the PARIS work could not be reproduced, it is clearly unacceptable to compare and only compare it against poorly reproduced results.