Articulate your NeRF: Unsupervised articulated object modeling via conditional view synthesis
Authors: Jianning Deng
Kartic Subr
Hakan Bilen
Venue: NeurIPS 2024 poster (2024)
Keywords: articulated object
neural radiance fields
unsupervised learning
analysis by synthesis
Primary area: machine_vision

Abstract: We propose a novel unsupervised method to learn pose and part-segmentation of articulated objects with rigid parts.
    Given two observations of an object in different articulation states, our method learns the geometry and appearance of object parts by using an implicit model from the first observation, distills the part segmentation and articulation from the second observation while rendering the latter observation.
    Additionally, to tackle the complexities in the joint optimization of part segmentation and articulation, we propose a voxel grid based initialization strategy and a decoupled optimization procedure.
    Compared to the prior unsupervised work, our model obtains significantly better performance, generalizes to objects with multiple parts while it can be efficiently from few views for the latter observation.

Articulate your NeRF: Unsupervised articulated object
modeling via conditional view synthesis
Jianning Deng
Kartic Subr
Hakan Bilen
University of Edinburgh
https://githu: wttpm////oooooo//ttttaaaatttoooa
Abstract
We propose novel unsupervised method to learn pose and part-segmentation of articulated objects with rigid parts. Given two observations an object in different articulation states, our method learns the geometry and appearance of object parts by fitting an implicit model on the first observation and renders the latter observation by distilling the part segmentation and articulation. Additionally, challenging joint optimization of part segmentation and articulation, we propose voxel grid based initialization strategy and a decoupled optimization procedure. Compared to the prior unsupervised work, our model obtains significantly better performance, generalizes objects with arbitrary number of parts while can be efficiently learned from few views only for the latter observation.
tackle the
Introduction
Articulated objects, composed of multiple rigid parts connected by joints allowing rotational or translational motion, such as doors, cupboards and spectacles are ubiquitous in our daily lives. Automatically understanding the shape, structure and motion of these objects is crucial for numerous applications in robotic manipulation 112 43 and character animation [1]]. Many works (TO) (30] 34 that focused this problem use groundtruth 3D shape, articulation information, and/or part segmentation to learn articulated object models but acquiring accurate 3D observations and manual annotations is typically complex and too expensive for building real large-scale datasets.
In this paper, we introduce a novel unsupervised technique that learns part segmentation and ar- ticulation (i.e., axis of movement, and translation/rotation of each movable part) from two sets of observations without requiring groundtruth shape, part segmentation or articulation. Two sets contain images of the same object from multiple viewpoints in two different articulation states. Our key idea that articulation changes only the poses of the object parts but not their geometry or texture. Hence, once the geometry and appearance are learned, one can transform to another articulation state given the part locations and target articulation parameters.
Building on this idea, we frame the learning problem as a conditional novel articulation (and view) synthesis task (see Fig. aa). Given source observation, multiple views of an object in one articulation state, we first learn the object's 's shape and appearance by using an implicit model [2I] and then freeze its weights. Next we pass the target observation, multi-view view images of the same object in a different articulation state, to tight bottleneck that distills part locations and articulations. We constrain our model to assign each 3D coordinate that is occupied by the object to a part and to apply a valid geometric transformation to the 3D coordinates of each part through ray geometry. The predictions of part segmentation and articulation, along with the target camera viewpoint, are passed to the implicit function and its differential renderer to reproduce the target observations (see Fig. [b)). Minimizing the photometric error between the rendered and target view provides supervision for learning part segmentation and articulation. However, joint optimization of these intertwined tasks is challenging and very sensitive to their initialization.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Segmentation Field
target image
Part masked NeRF
Tphots
Trained from source images
Articulation
Composite rendering
Articulation
synthetic image
Part-aware my sampling
Composite and accumalate
along the ray
(a) Pipeline overview
(b) Composite rendering
Figure 1: (a) Our method learns the geometry and appearance of an articulated object by first fitting a NeRF from (source) images of an object in fixed articulation. Then, from another set of (target) images of the object in another articulation, we distill the relative articulation and part labels. Green lines show the gradient path during this distillation. (b) Using the part geometry and appearance from NcRF, we render the target images by compositing parts after applying the predicted articulations to segmented parts. The photometric error provides the required supervision for learning the parts and their articulation without groundtruth labels.
To address the optimization challenge, we propose an initialization strategy using an auxiliary voxel grid, which provides an initial estimate for moving parts by computing the errors in the foreground masks when rendering target views for the source articulation. Additionally, we introduce a decoupled optimization procedure that alternates between optimizing the part segmentation on the photometric error and the articulation parameters on the foreground prediction error.
The key advantage of our method, compared to other unsupervised articulation prediction methods [7] its stable performance across different object and articulation types, its ability learn from few target views and to model multiple moving parts. Thanks to the stage-wise training, we achieve high-quality object geometry and appearance by using the well-optimized implicit models [2]. Since the part segmentation and articulation parameters form small portion of the total weights, along with the initialization and decoupled optimization strategies, our method efficiently learns them from few target views, unlike the most relevant work [17 that jointly optimizes multiple part-specific implicit functions from scratch.
Related work
Analysis of articulated objects The analysis of articulated objects typically involves segmentation of movable parts, and estimating their attributes such as position and direction of joint axes, rotation angles, and translation distances. Prior works study articulated objects using 3D input such as
meshes [3 22), point clouds [45] 44
37 [4] or RGBD images [18
6 40 13 that are error-prone
and labor-intensive collect in real-world scenarios. Recent works [10
30 that use RGB images
to segment 3D planar object parts and estimate their articulation parameters simultaneously require ground-truth labels for segmentation and 3D articulation.
Articulated object modeling via novel view synthesis Neural implicit models 220 26 21 that are originally designed to model static objects in 3D consistent way have been extended to articulated objects by multiple recent works [24 36 11 16
40) A-SDF (24) learns disentangled latent space for shape and articulation to synthesize novel shapes and unseen articulated poses for a given object category. CARTO 7 extends A-SDF to multiple object categories and uses stereo image pairs as input. Similarly, CLA-NeRF [36] learns to perform joint view synthesis, part segmentation, and articulated posc estimation for an object category from multiple views. NARF22 [16] Icarns separate neural radiance fields (NeRF) for each part and composes to render the complete object. Unlike our method, both A-SDF and CARTO require 3D shape and articulation pose supervision, CLA-NeRF requires part segmentation labels and is limited to modeling only ID revolutions for each joint, NARF22 relies groundtruth part labels and articulation pose.
38
17
Most related to our work, DITTO [1, PARIS 17 and Weng et al. 40 aim to estimate part segmentation and articulation pose without labels using from observation pair of an object in two different articulations. DITTO uses pair of point cloud as input, can only model shape, whereas both PARIS and our method uses pairs of multi-view images, and to model both geometry and

---

appearance. PARIS adopts the dynamic/static modelling of [42 46 learns separate NeRF for the dynamic and static parts. They are composited using the estimated relative articulation to render the observations with the different articulation. Similarly, 40 first reconstructs the object in two articulation states with two sets of RGBD images. Later, the part-level segmentation is performed based on image registration results. While our method is also based on the same analysis-b--synther principle to supervise the training, it differs from previous work in several key ways. Compared to DITTO III and Weng al. [40]. our method relies completely
2D input data. And compared to PARIS I77, our method involves only a single NeRF that is learned on multiple views of an object instance in fixed articulation pose. Once the NeRF is learned,
freeze its parameters, and learn a segmentation head and articulation to selectively deform the rays while rendering views of different articulations. This means model's size remain nearly constant when the number of parts increases, which, combined with our two step optimization strategy, leads to more stable and data-efficient learning, yielding significantly better performance. Furthermore, we show that our model goes beyond modeling a single moving object part as in PARIS, and successfully learn multiple moving parts.
that model differences between subsequent video frames of a dynamic scene through deformation field in 3D. While these techniques are general could be used in modeling articulated objects in principle, the deformation field does not provide explicit predictions for part segmentation and articulation. Prior methods that focus on modeling specific articulated and deformable objects such as human body [39][3] g 9 25
Deformable NeRF
There exists several techniques 27
35 28
and four-legged animals 41 leverages specialized 3D priors 19 that not applicable arbitrary object categories.
3 Review: NeRF
Given a set of images I of an object, NeRF function 21 maps tuple (ar, d), where R is 3d position and d E IR² is a direction, to an RGB color c, and a positive scalar volume density a. The model outputs volume density a at the location æ along with latent feature vector which is then concatenated with the viewing direction d and fed into an MLP to estimate the view-dependent RGB color c. With a slight abuse of notation, and color as function with c(z, d). That is, we use boldface to denote vectors.
also use a and z as functions (xx) and z(xx) respectively
NeRFs are trained by expressing the color at each pixel of images in the training subset. The color at a pixel is estimated as CN via the volume rendering equation as a sum of contributions from points along the ray through the pixel, say r(t) o fd where o is the origin of the ray (usually the point of projection of a camera view) and is a scalar. Points L;, i = 1,..., n are sampled along the ray x; o + d. The estimated color is given by
Cx(r) = Σ T. - (1-exp(-oof))
where TT = Σ exp(-o700)
(I)
1=1
where we use the shorthand notation cf and of to denote c(æ, d) and oxx; respectively, and d t++ t4 the distance between samples. And the opacity value is calculated (x(r)= Ei=1 T. (1 - exp(-of 8.). The parameters of and a are obtained by minimizing the photometric loss between predicted color (r) and the ground truth color C(r) as
Cphoto Σ [[CN(r) - Crr)
(2)
FER
4 Method
Let I and I'' be two observations of an object with k (known a priori) rigid parts in articulation poses P and P' respectively. Each observation contains multiple views of the object along with the foreground masks. We call I the source observation and I' the target observation. P (resp. P') is pose tensor composed of P/ E SE),, == 1,..., & (resp. Pi) transformations as 4 x matrices corresponding to the local pose of each of the k parts. Our primary goal is to estimate a pose-change tensor M composed of M, E S3), = 1,...,k It so that P/ = MtPr. We solve this by starting with the construction of a NeRF from I. as described in Sec. along with the efficient coarse-to-fin
3

---

volume sampling strategy in 22) followed by novel modification to build awareness at the part-level. We use this modified parameterization optimize an auxiliary voxel grid corresponding the parts via pose estimation and part segmentation pipelines. Once we have identified the parts and their relative transformations, we are able to render novel views corresponding to articulation pose P by transforming view rays suitably by M
4.1 Part-aware rendering
Once a NeRF function, which we call 'static NeRF', is learned over Z. we freeze its parameters and append a segmentation head towards obtaining part-level information. is instantiated as 2-layer MLP, and maps the latent feature vector z z(x) to probability distribution over the lo object parts. We denote the probability of a 3D point to belong object part 1..... k as se(2(x)).
If the segmentation praabbirobabiltties 8y and pose-change transformations Mf are known, then the object in unseen articulation pose can be synthesised by suitably transforming the statically trained NeRF without modifying its parameters and then compositing the individual part contributions. To model pose change in each part, we use a different ray, virtual ray associated with each part
Tt M₁ r
and the final rendered color is:
Cp(r)
(1- exp (- (x.) o" (x:) 671)) c:e
(4)
where 7.- = exp (-Ex (56 (zz) o"(xxx) (())))()))) 87')
is the transmittance sum. Compared to the original formulation in Eq. ( can seen the density replaced by part-aware density which is the product of s and a. Note that we made a similar modification to the efficient volume sampling strategy of 2 and refer to the Appendix A.I for more details.
Since the groundtruth part segmentation and articulation are unknown, one could optimize the parameters of and M on the target observation such that the photometric error in between the part- aware rendered views and I'. Note that Cp(r) is function of M (through re in Eq. (3) and s, they can be jointly optimized through backpropagation. However, jointly solving these two intertwined problems through rendering is challenging and highly sensitive to their initial values, as their solutions depend on each other. Hence, we introduce alternating optimization strategy next.
4.2 Decoupled optimization of and M
To learn the part segmentation and pose-change, we first introduce an auxiliary voxel grid that assigns each 3D coordinate either to background or part. Once the voxel is initialized, our training iterates multiple times over a three-step procedure that includes optimization of M, s, and refinement of the voxel grid entries.
Initialization of voxel grid Here our key idea is to find the pixel differences between the source and target observations by rendering a target view s' by using the static NeRF, and label the 3D) locations of the different pixels as the movable parts to provide a good initialization for estimating the articulation (see Fig. 2). To this end, we first build a 3D voxel of the static NeRF by discretizing its 3D space into 128 bins along each dimension. To compute the voxel entries, we query the density value from the static NeRF at each voxel coordinate and binarize it into occupancy values with criteria exp(-o((x)) ))) > 0.1. Next, we use the static NeRF to render opacity and depth images for the viewpoints v'. The pixels rendered as foreground in rendered opacity but not in the corresponding foreground mask of I' will be collected. For those pixels, we further compute their depth by accumulating their density values along the corresponding ray, and use their estimated depth values to tag the occupied voxel entries either with static or dynamic parts. In the case of multiple moving parts, which is assumed to be known, we perform a clustering step [4] to identify A: clusters corresponding moving parts, as assume that 1 part of the object is static without loss of generality. Finally, we gather the voxel coordinates that is assigned to part E to form a matrix of 3D) coordinates X

---

voxel extraction
depth
pixel difference
Static NoRF
RGB
moving
Foreground musk
Figure Voxel initialization: identify the voxels belonging to moved parts based on pixel opacity difference.
Step 1: Optimization of Mr As depicted in the right side of Fig. we project the 3D coor- dinates X for part & onto the image plane for each camera viewpoint included in I' by us- ing 3 dimensional intrinsic camera matrix K as U2 K KM, ' Xx. and denote the projected point matrix on image plane Ug. On the left side of Fig. 3 we collect 2D coordinates from the overlap region between rendered opac- ity and target image I' to obtain U" Then concatenate U' and part-specific matrices Ue to obtain U. We want to optimize U to be the same as the target coordinates F, which is obtained by gathering the 2D coordinates from the foreground mask of T' Note that both U and F are matrices where each row corresponds to 2D image coordinate where has significantly more rows, as U is obtained from a coarse voxel grid. Then we minimize the 2D Chamfer distance between U and F:
d
Inverse ponc Ad?
overlap
project
Renekree
Chassier
Detange
Target imgges
Tagee condianss
Voade
Figure 3:
Illustration for optimization of M2. The green dotted line shows the gradient flow.
M = arg min M
(CD(U,)) F).
(5)
In practice, only project X₃ for the moved parts, and stack them with 2D image coordinates of the foreground pixels in both the rendered image and the groundtruth image, as illustrated in Fig,
Step 2: Optimization of st Once obtain the solution of the pose-change M from Step 1, plug in it to the ray deformation in Eq. 33 and render each view in I' in Eq. (4). Then we compute photometric loss between the rendered views and I', and minimize it with respect to the parameters of 8 only. In the case of multiple moving parts, we initialize the parameters in using X, for supervision. Please refer Appendix [A.4 for more details.
Step 3: Voxel grid refinement The initial part segmentation estimates in the voxel grid are often sparse and noisy due inaccurate density estimation in NeRF and misassignment of pixels around the foreground and part boundaries. To obtain a more accurate voxel grid, we follow the same steps in the initialization stage except that the 3D coordinates label from segmentation head s. We denote the new 3D coordinates as Xi For the consistency, we accept the voxels X; only when the predieted part labels agree within a 3D local neighborhood of X₁. The neighborhood is defined as being within the distance of one grid cell at resolution of 128. Note that we only perform the refinement step, after approximately 2k iterations ensure that the segmentation head produces confident results. Then we also increase the voxel resolution to 256 for each dimension.
assigned to the parts based the predicted
Experiment
5.1 Dataset
We evaluate our method on the synthetic PartNet-Mobility dataset 43 23 3
While the dataset contains more than 2000 articulated objects from 46 different categories, we use subset of the dataset with 6 shapes that was used in [17]. For a fair comparison, we downloaded the processed dataset from [17 which contains 2 sets of 100 views along with their foreground masks, each with
5

---

Revolut
Prismatic
Metric
Method
laptop
oven
stapler
fridge
blade
storage
PARIS
0.68 = 0.40 1.04 = 0.68
2.42 = 0.91
0.81 = 0.60
48.58 + 25.43
0.34 - 0.09
ea
Ours
0.33 = 0.04 + 0.03
0.33 + 0.04
0.54 0.08
1.54 0.07
1.11 + 0.14
epl
PARIS
0.18 = 0.13 + 0.53
55.54 = 39.88
0.33 - 0.14
(10-2)
Ours
0.48 = 1.29 = 0.04
0.17 + 0.05
0.44 + 0.03
PARIS
0.60 = 0.32
0.68 + 0.39
44.62 = 6.17
0.87 = 0.55
cg.
Ours
0.25 = 0.03 0.351 0.06
0.29011.003
0.60 + 0.05
PARIS
1.13 0.52
0.30 0.01
Ct +
Ours
0.01 + 0.01
0.02 0.03
Table I: Part-level pose estimation results. Our method outperforms PARIS in majority of object categorics while having lower variation over multiple runs in the performance.
different articulation, and also groundtruth part segmentation labels, for each shape. In addition, we select 4 additional shapes, each with two moving parts, and apply the same data generation process to them. Additionally we collected images of a toy car with a handheld device, with camera viewpoint estimated from kiri engine application [5].
We train the static NeRF on 100 views from the first observation, train the part segmentation and articulation 100 views from the second observation. We report the performance of method for varying number of views from the second observation in Tab.
Following 177, we report performance in different metrics for pose estimation, novel- view/articulation synthesis, and part segmentation. Pose estimation: To report articulation pose performance, we report results in i) direction error ea that measures the angular discrepancy in degrees between the predicted and actual axis directions across all object categories, ii) position error Ep and geodesic distance ea for only revolute objects to evaluate the error between the predicted and true axis positions and the error in the predicted rotation angles of parts in degrees, respectively, iii) translation error e, measure the disparity between the predicted and actual translation movements for prismatic objects. Novel-view and -articulation synthesis: We evaluate the quality of novel view synthesis generated by the models using the Peak Signal-to-Noise Ratio (PSNR) where higher values indicate better reconstruction fidelity. Part segmentation: We use mean Intersection over Union (mloU) on the rendered semantic images to evaluate the accuracy of part segmentation, which is tightly related to the rendered image quality of objects in different articulation states. The ground truth segmentation is generated within the Sapien framework in (43. Finally, due to the lack of groundtruth segmentation label and articulation in the real data, we only report PSNR and provide qualitative evaluation.
5.2 Results
Baseline We compare our approach with the state-of-the-ar unsupervised technique, PARIS [7 which constructs separate NeRF models for each part an articulated object and optimizes motion parameters in an end-to-end manner. However, it is limited to two-part objects with only one movable part. As the authors of PARIS do not report their results over multiple runs their paper, we use their official public code with the default hyperparameters, train both their and our model over 5 random initializations and report the average performance and the standard deviation. We use 2 sets of 100 views for each object to train the models. More implementation details can be found in the supplementary material. We would like to note that the performance of PARIS in our experiments significantly differ from the original results despite all the taken reproduce them in the original wayl]
Part-level pose estimation For part-level pose estimation, we provide quantitative results in average performance over 5 runs and their standard deviations in Tab. [ and a qualitative analysis in Fig. 6 The results indicate that our method consistently achieves lower errors across most evaluation metries
Similar problems have been pointed out and the reproducibility of these results has been acknowledged as challenging by the authors in lissue I and fissue 2

---

PARIS
GT
Blade
Laplop
Stapler
Fridge
Storage
Oven
Scissor
Figure 4: Qualitative 2D part segmentation results, Pixels in green denotes the movable parts. Our method demonstrates consistent performance across all tested objects while PARIS failed Blade, Laptop and Scissor.
compared to PARIS, except the joint position error E, where the differences are negligible, within the 10 range. Notably, the performance of PARIS on the stapler and blade exhibits significantly higher errors. We observe that PARIS fails to converge shown Fig. PARIS fails to accurately segment the parts be easily identified in the novel articulation synthesis experiments in Fig. [ Poor part reconstruction in PARIS results inaccurate part-level pose estimations. Additionally, the lower standard deviation across all reconstructed objects for our method indicates its stable learning and ability to work for different object instances. We attribute the stability our method to the decoupled optimization along with the stage-wise training, which contrasts with the joint optimization approach used by PARIS.
good solution in all 5 runs for these objects. As stapler and blade, which can also
Segmentation and composite rendering Part segmentation and novel view synthesis in pose 'p' are reported in Tab. 2 complemented by a qualitative analysis of part segmentation in
Ours
Fig. 4 Our method outperforms PARIS across most evaluated objects in part segmentation and image synthesis quality, with the only exception being minor difference the PSNR for the laptop.
GT
As our model builds on a static NeRF that achieves high quality novel view synthesis for one observation, the rendering quality largely preserved after the second stage. We provide more detailed analysis in Appendix(A.. A.33] Addi- tionally, benefiting from accurate pose estima-
Rox
Glasses
Oven
Storage
Figure 5: Qualitative results for 2D multi-part seg- mentation. pink color denotes the static part, while other colors denote the moving parts.
tion via a decoupled approach, our method achieves more robust and precise part segmentation, depicted in Fig. Here, method consistently delivers accurate segmentation results for challenging objects such as the blade, stapler, and scissors, where PARIS struggles with accurate part reconstruction. In the other instances including the laptop, storage, and oven, our method achieves visibly better results.
Evaluation on objects with multiple movable parts A key advantage of our model is its ability model objects with multiple moving parts. For such objects, we report pose estimation results in Tab. 3 qualitative for part segmentation in Fig. and novel articulation synthesis in Fig. 6 As in the single part moving object experiments, our method delivers similar performance for multi-part objects. Notably, we observed a marginally higher joint direction error for glasses, which we attribute to the thin structures such as temples and failure to segment them accurately which can be possibly improved by using higher resolution images.
7

---

Method
GT in P
Novel articulation synthesis
GT in P
PARIS
Ours
PARIS
Ours
Ours
Figure 6: Qualitative evaluation for novel articulation synthesis. The ground truth axis is denoted in green and the predicted denoted red. Please refer to supplementary more visualizations.
5.3 Ablation studies
We assessed the effectiveness of the proposed decoupled pose estimation of Mt (DP) and iterative refinement of X₃ (IR) on the 'fridge' object. When DP is disabled, the segmentation and articulation simultaneously learned. In contrast, disabling IR maintains the initial X; for pose estimation. The results Tab. 5 particularly the first row, demonstrate that joint optimization without DP inaccurately predicts the articulated pose, treating the entire object static. While enabling DP improves the performance, as shown in the second row, the performance is still poor due to the noisy initial values compared to full model.
Additionally, we evaluated the impact of the number of views in the target observation for training. We train model the randomly subsampled views for multiple runs and averaged their performance (see Tab, 4. Results show significant drop in pose estimation and rendering quality with 4 images or less. Notably, even with only images, our approach surpasses the performance of PARIS trained with 100 images (indicated with subscript P in Tab. 4). This result clearly show that our method robust against fewer viewpoints from the target viewpoints and allows efficiently learning "articulate' a pretrained NeRF from few target views only.
Revolut
Prismatic
Metric
Revolut
Prismatic
oven
glasses
box
storage
Metric
Method
laptop
oven
stapler
fridge
blade
storage
ea +
1.02
2.35
0.56
1.82
ep L
0.16
0.47
0.27
PARIS
0.98
0.99
0.16
0.98
0.76
0.94
mloUt
eg 4
1.03
1.01
0.65
Ours
0.99
0.99
0.98
0.99
0.94
0.96
4 (10)
0.11
PARIS
30.31
31.48
24.36
32.74
31.87
30.63
PSNR
PSNR 32.98
29.22
28.61
28.25
Ours
29.27
32.08
34.31
35.10
36.47
34.51
mloU 1
0.97
0.98
0.99
0.94
Table 2:
Articulation synthesis and part segmentation results.
Table 3: Objects with multiple parts. Er- rors using multiple metries for pose estima- tion (averaged over all joints).
Average performance over runs best results in boldface).

---

Init.
Metric
Num. of images
Metric
2
4
8
16
32
100
100p
DP
IR
ed
6g
PSNR
+
46.05
8.89
0.59
0.58
0.50
0.54
0.81
2.57
26.95
18.09
Eg +
44.74
20.79
0.70
0.60
0.56
0.49
0.87
2.38
14.96
25.16
PSNR
22.65
29.95
34.00
34.28
34.88
35.10
32.74
v
0.54
0.49
35.10
Table 4: Ablation studies with different number of target images.
Table Ablation study initialization strategies.
different
Atticslation interpolation
GT
Sin:
Articalation interpolatioon
GT
Srg.
Figure 7: Results on real world examples, the red line indicates the estimated joint axis direction. Green and purple color denotes the moving car door, while pink denotes the body of the toy car. Please refer Fig. for more qualitative evaluation.
5.4 Real-world example
The qualitative evaluation can be found in Fig. 7 Novel articulation synthesis in different views are showed in two sides Fig. The GT is shown in pose P' while the segmentation image is the toy car in pose P. The average PSNR we obtain from the static NeRF is and the PSNR for 28.19, while PSNR for pose P' 24.32. More detail results segmentaion and articulation interpolation are also presented in Fig.
5.5 Limitations
Our method has few limitations too. As we use rendering to supervise our segmentation and pose, our method may fail to segment very thin parts small movable parts when the rendering error is small. Our method is less stable for multipart objects compared to two-part objects. When the object parts are nearly symmetry, our method may fail find the correct articulation and choose another articulation that produces similar rendering. We provide examples of such failure cases in Appendix AA.3) Additionally, also inherits the limitations of the implicit models that are built on such as failure to model transparent objects, and inaccurate 3D modeling when the viewpoint information is noisy. Finally, our method is limited to articulated objects with rigid parts and cannot be used to model soft tissue deformations.
Conclusion
In this study, we tackled the challenges of supervisdd part segmentation, articulated pose estimation, and novel articulation rendering for objects with multiple movable parts. Our approach is the first known attempt to model multipart articulated objects using only multi-view images acquired from two arbitrary articulation states of the same object. We evaluated our method with both synthetic and real-world datasets. The results suggest that our method competently estimates part segmentation and articulation pose parameters, and effectively renders images of unseen articulations, showing promising improvements over existing state-of-the-an techniques. Further, our real-world data experiments underscore the method's robust generalization capabilities. At last, the code and the data used in this project will be released upon acceptance. However, the reliance on geometric information from moving parts for articulation pose estimation poses challenges in modeling highly symmetrical objects. Future work could improve our method by incorporating both appearance and geometric data into the pose estimation process, potentially enhancing accuracy and applicability.
Broader Impacts The proposed technique could be potentially used to understand articulated objects for their robotic manipulation in future. The authors are aware of any potential harm that may arise when the technology is used

---

Kartic Subr was supported by Royal Society University Research Fellowship. Hakan Bilen was supported by the EPSRC Visual Al grant EP/T028572/1.
Acknowledgements
References
[1] Norman Badler, Brian Barsky, and David Zeltzer. Making Them Move: Mechanics, Control & Animation of Articulated Figures. Routledge, 1990.
[2] Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1470-5479,,2 2022.
[3] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:/512.033112 2015.
[4] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. Density- spatial clustering of applications with noise. In Int. Conf. knowledge discovery and data mining, 1996.
[5] Jiemin Fang. Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIC- GRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
[6] Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. Gapartnet: Cross-category domain-generallzzabee object perception and manipulation via gener- alizable and actionable parts. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7081-7091, 10910, 2023.
[7] Nick Heppert, Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Rares Andrei Ambrus, Jeannette Bohg, Abhinav Valada, and Thomas Kollar. Carto: Category and joint agnostic reconstruction of articulated objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21201 21210,, 2023.
[8] Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung. Arch: Animatable reconstruction of clothed humans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3093-3102, 2020.
[9] Mustafa Işık. Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, and Matthias Nicßner. Humanrff: figh-fidelity neural radiance fields for humans in motion. ACM Transactions on Graphics (TOG), 42(4):1-12, 2023.
[10] Hanxiao Jiang, Yongsen Mao, Manolis Savva, and Angel X Chang. Opd: Single-vier view 3d openable part detection. In European Conference on Computer Vision, pages 410-426. Springer, 2022.
[11] Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of objects from interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5616-5626 2022.
articulated
[12] Dov Katz and Oliver Brock. Manipulating articulated objects with interactive perception. In 2008 IEEE International Conference on Robotics and Automation, pages 272-277. IEEE, 2008
[13] Yuki Kawana and Tatsuya Harada. Detection based part-level articulated object reconstruction from single rgbd image. Advances in Neural Information Processing Systems, 36, 2024.
[14] Yuki Kawana, Yusuke Mukuta, and Tatsuya Harada. Unsupervised pose-aware part decompo- sition for man-made articulated objects. In European Conference on Computer Vision, pages 558-575. Springer, 2022.
[15] KIRI Innovation. Kiri engine.
[16] Stanley Lewis, Jana Pavlasek, and Odest Chadwicke Jenkins. Narf22: Neural articulated radiance fields for configum coniiguattoon-wwaee rendering. In 2022 IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 770-777. IEEE, 2022.
10

---

[17] Jiayi Liu, Ali Mahdavi-Amiri and Manolis Savva. Paris: Part-level reconstruction and motion analysis for articulated objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 352-363, 2023.
[18] Liu Liu, Han Xue, Wenqiang Xu, Haoyuan Fu, and Cewu Lu. Toward real-world category- articulation pose estimation. IEEE Transactions on Image Processing, 31:1072-1088, 2022.
[19] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 3466):448112248:66 October 2015.
[20] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4460-4470, 2019.
[21] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[22] Niloy Mitra, Yong-Liang Yang, Dong-Ming Yan, Wilmot Li, Maneesh Agrawala, et al. Illustrating how mechanical assemblies work. ACM Transactions Graphics-TOG. 29)):588 2010.
[23] Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
[24] Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape representa- tion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13001-13011, 2021.
[25] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neural articulated radiance field. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5762-5772. 2021.
[26] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[27] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla, Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[28] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin- Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXh::2106.1322,
2021.
[29] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[30] Shengyi Qian, Linyi Jin, Chris Rockwell, Siyi Chen, and David F Fouhey. Understanding 3d object articulation in internet videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1599-1609, 2022.
[31] Andrei Sharf, Hui Huang, Cheng Liang, Jiapei Zhang, Baoquan Chen, and Minglun Gong. Mobility-trees for indoor scenes manipulation. In Computer Graphics Forum, pages 2-14. Wiley Online Library, 2014.
11

---

[32] Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13653-13662, 2021.
[33] Shih-Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin. A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. Advances in Neural Information Processing Systems, 34: 2278--1222, 2021.
[34] Xiaohao Sun, Hanxiao Jiang, Manolis Savva, and Angel Xuan Chang. Opdmulti: Openable part detection for multiple objects. arXiv preprint Sadd::2303.10088 2023.
[35] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12959-12970, 2021.
[36] Wei-Cheng Tseng, Hung-Ju Liao, Lin Yen-Chen, and Min Sun. Cla-nerf: Category-level articulated neural radiance field. In 2022 International Conference on Robotics and Automation (ICRA), pages 8454-8460, IEEE, 2022.
[37] Xiaogang Wang. Bin Zhou, Yahao Shi. Xiaowu Chen, Qinping Zhao, and Kai Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8876-8884 2019.
[38] Fangyin Wei, Rohan Chabra, Lingni Ma, Christoph Lassner, Michael Zollhöfer, Szymon Rusinkiewicz, Chris Sweeney, Richard Newcombe, and Mira Slavcheva. Self-supervised neural articulated shape and appearance models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15816-15826, 2022.
[39] Chung- Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher- Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEF/CVF conference on computer vision and pattern Recognition, pages 16210-16220, 2022.
[40] Yijia Weng, Bowen Wen, Jonathan Tremblay, Valts Blukis, Dieter Fox, Leonidas Guibas, and Stan Birchfield. Neural implicit representation for building digital twins of unknown articulated objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3141-3150, 2024.
[41] Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi. Magiepony: Learning articulated 3d animals in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8792-8802, 2023.
[42] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. D' 2nerf: Self-supervised decoupling of dynamic and static objects from monocular video. Advances in neural information processing systems, 35:32653-32666 2022.
[43] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part- based interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[44] Li Yi, Haibin Huang, Difan Liu, Evangelos Kalogerakis, Hao Su, and Leonidas Guibas. Deep part induction from articulated object pairs. arXiv preprint arXiv:1809.07417, 2018.
[45] Qing Yuan, Guiqing Li, Kai Xu, Xudong Chen, and Hui Huang. Space-time co-segmentation of articulated point cloud sequences. In Computer Graphics Forum, pages 419-429, Wiley Online Library, 2016.
[46] Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, and Steven Lovegrove. Star: Self-supervised tracking and reconstruction of rigid objects motion with neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13144-13152, 2021.
12

---

Revolut
Prismatic
Metric
Method
laptop
oven
stapler
fridge
blade
storage
PARIS
0.03
0.03
0.07
0.00
0.00
0.37
ed |
PARIS
0.18
0.48
1.30
0.22
0.27
0.24
Ours
0.26
0.28
0.27
0.42
1.41
0.86
PARIS
0.01
0.03
0.06
0.02
ept
PARIS
0.08
0.04
2.12
0.17
(10-2)
Ours
0.39
1.24
0.13
0.41
PARIS
0.03
0.00
0.00
0.00
cg.
PARIS
0.24
0.25
37.86
0.37
Ours
0.22
0.31
0.24
0.55
PARIS
0.06
0.00
et 4 (10-1)
PARIS
5.84
2.93
Ours
0.12
0.05
Table 6: Comparison with PARIS in 5-time best setting. PARIS denotes performance reported in the original paper. Best results are shown in bold, second best are shown with underline.
A Appendix supplemental material
Optionally include supplemental material (complete proofs, additional experiments and plots) in appendix. All such materials SHOULD be included in the main submission.
A.1 Part-aware proposal network
To achieve part-aware composite rendering for articulated objects, we also need to modify the proposal network for correct sampling along the ray r E R'. In the composite rendering, the proposal network is required to produce the similar distribution of the weights for samples along the ray. Thus, following the similar design in the part-aware NeRF, we append an extra segmentation field to the proposal network. Now the valid density for each samples becomes segmentation-ne mask density as "e (æij)a" The transmittance are then calculated using Eq. 44h without the color term, which will be later used for online distillation. Please refer the original paper (2) for more details about the online distillation. During the training of part-aware NeRF, the original parameters in the proposal network will be frozen and only updates the segmentation field.
A.2 Comparison with PARIS
Here also provide the 5-time best comparison between ours and PARIS. Besides, the reported performance for PARIS also provided in this subsection. From Tab. 6 we can see that the original reported performance from PARIS are exceptionally better compared the reproduced ones, though with the same data and the same configurations. If we only focus on the 5-time best comparison between reproduced PARIS and ours, we can see that achieve comparable performance. Another thing we can notice is that for prismatic objects, while PARIS has much better estimations on joint axis direction. However, our method later show much better estimation for estimating the moved distance for the dynamic parts. While one thing to notice is that PARIS failed to reconstruct the stapler for all 5 runs in our experiment.
A.3 Limitations
Details about the limitations and failure cases of methods.
In our decoupled optimization strategy, the articulated pose estimation relies exclusively on the geometric information from the moving parts. As result, method faces challenges with highly
13

---

(a) Visualization dynamic voxel update. From left to right: initialized Xe, updated Xe, final X
(b) Failure cases for foldchair, from left to right: groundtruth RGB, rendered RGB, part segmentation.
(c) Artifacts for thin parts, the left one is the groundtruth, the right rendering result.
symmetrical components. For example, in the case of folding chair Appendix AA. segmentation accurately identifies the chair's seat, the pose estimation mistakenly flips the seat by 180 degrees, resulting the seat being oriented with the bottom upwards as shown in Fig. 8(b) Additionally, our method may encounter difficulties when the articulation motion is minimal, which can lead to insufficient pixel differentiation in images across different articulation poses for initial model estimation. In such cases, both the pose estimator and the iterative updates of Xe may underperform or even fail. We also notice that the proposal network have difficulties handling very thin parts in the novel articulation synthesis. As can see in Appendix A.3] some artifacts appeared at the edge the frame and the temples for the glasses as in Fig. 8(c) which should be ideally masked out without any opacity and color for those pixels.
although the
Furthermore, the reliance on pre-trained static NeRF limits the upper boundary of our method's performance in rendering novel articulations. Addressing these limitations could significantly enhance the robustness and applicability our strategy in handling complex articulated objects. For further analysis of this, please refer to.
14

---

(a) Ground truth
Rendered
(c) Segmentation in original pose
Figure 9: We can see in the Fig. ( that the corner of the laptop sereen is missing in the novel articulation rendering. While it looks perfect when we check the segmentation in the original pose. Thus, suspect it is the proposal network than failed estimate the density distribution for the screen from certain viewpoints.
A.4 Training setting
As detailed in Sec. 4.2 we perform optimization Mr for 4000 iterations and sy for 2000 iterations for all evaluated objects. Checkpoints are saved following Step 3. We conduct this process through 5 eycles for objects with single moving part and 6 eycles for objects with multiple moving parts. Checkpoints with best PSNR during validation will be used for test.
During the first step of My optimization, by 0.5 every 500 iterations. We accumulate gradients from viewpoints to simulate a batch size of initializing Mj identically in the first cycle and using the previously estimated Mj for subsequent cycles.
begin with learning rate of 0.01, which linearly decays
In the second step of 8ı optimization using the Adam optimizer, the initial learning rate is set at 0.01 and linearly decays by factor of 0.01 every 100 iterations. multiple moving parts, initialization involves training the segmentation head using pre-assigned labels on X,, and querying predictions for æ E X Cross-entropy loss optimized over 1000 iterations with a learning rate of le shapes the learnable parameters in
Our experiments, requiring around 16 GB of VRAM, complete approximately 30 minutes on single RTX 4090 GPU for single object. As for the training of static NeRF, it takes about 10 minutes with less than 10 GB of VRAM. The estimated total GPU time for this project would be about GPU months.
A.5 Ablations
A.5.1
Performance cost for novel articulation synthesis
We also provide an ablation study investigate the performance drops for conditional novel articula- tion synthesis compared to the pre-trained static model. As shown in Tab. 7 the performance drop is subject different category of objects ranging from smallest 13.3% for the fridge and blade 3..4% laptop. We suspect the significant drops for laptop is caused by the deteriorate performance of the proposal network on the thin laptop screen. Visualizations can be found in Fig. g The results indicate that our method can benefit from the high quality of appearance reconstruction from the pre- pre-trained static NeRF.
A.6 Visualization
Here we show more visualization for qualitative evaluations. Besides, additional animated images for the real-world toy car is also provided in the supplementary materials.
15

---

Method
GT in P
Novel articulation synthesis
GT P
PARIS
Ours
PARIS
Ours
PARIS
Ours
PARIS
Ours
Figure 10: Articulation interpolation for single moving part objects.
GT in
Novel articulation synthesis
GT p
Figure 11: Articulation interpolation for multiple moving part objects.
16

---

Object
Method
laptop
oven
stapler
fridge
blade
storage
oven
glasses*
box"
storage"
Static
42.70
39.69
42.39
40.50
42.7
40.33
39.03
37.93
36,56
35.23
Art.
29.27
32.08
34.31
35.1
36.47
34,51
32.98
29.22
28.61
28.25
A
-31.4%
-19.2%
-1911%
-13.30%
-1333%
-14.4%
-15,5%
-22.8%
-21.8%
-19.8%
Table 7: Comparison of rendering quality between objects in their original pose P and articulated pose P Static' refers to the rendering performance of the object in its original pose P. whereas "Art.' indicates the rendering quality of the object in articulated pose P using method with the static NeRF. Objects marked with represent those with multiple movable parts.
Ed +
Cp
Cg 4
PSNR1
mIoUt
PARIS
1.96 = 1.14 0.27 + 0.13 2.04 = 0.62
30.97 + 0.83
0.95 = 0.01
Ours 0.25 + 0.01 0.06 + 0.01 0.80 + 0.19 32.27 + 0.29
0.96 + 0.02
Table 8: Quantitative evaluation for the same door instance. We report average perfor- mance over 5 runs. For the description of Ep, Cy. PSNR and mloU. please see Section 5.1 in the submission. Our method outperforms PARIS consistently in all the metries.
A.7 More results
17

---

Novel articulation synthesis
Segmentation
Figure More Qualitative evaluation on real-worke object.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Each claim is substantiated by methodological discussions in Sec. and rigorous experiment evaluations in Sec. 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
18

---

Input P
Rendered for P
Input for p
Rendered for p
Seg. GT for P
Seg. Pred. for P
Seg. GT for 'p'
Seg. Pred. for '
Figure 13: Here we show qualitative results for a 'door' instance along with its frame. In contrast to the instances submission, the static part (frame) is smaller than the moving part (door). Given sets of views in the articulation P and P', we provide the original input images and their rendering in respective articulations, ground-trun truth (GT) and predicted part segmentation results. Our method achieves faithful rendering results in different articulations with minar artifacts and accurate part segmentation.
Input for P
Rendered for P
Input for P
Rendered for P
Seg. GT for P
Seg. Pred. for P
Seg. GT for P
Seg. Pred. for P
Here show qualitative results for toy robotic arm with static joints (grey and blue) and moving joints (red, yellow and green). Unlike the objects the Sapiens dataset that have independently moving parts, the joints the robotic arm are linearly connected forming chain structure. Note that our method independently estimates the DoF (3) transformation for each moving part between articulation pose P and P'. Given the kinematic structure, the local transformations could be computed. Our method faithfully renders the robotic arm in different poses with minor artifacts, and segments its parts accurately. Note that the static parts labeled part.
Figure 14:
19

---

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Details about discussions about limitations can be found in Sec. S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed
penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: this paper, focus more on the practical results, which is shown in Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
20

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, the details about our method can be found in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code and data will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
hublic/guidess http://guuiies/ooeubbmmiiss SoodeeSbbissiooooll111
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips.co/ http://pww////eeeeeee//eettaaaaa
for more details.
21

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details on the training setting shown in Appendix A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We ran experiments multiple times and report average performance with the standard deviation. In addition, we report results over a set of diverse objects and evaluate them by using several metries.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
22

---

Answer: [Yes]
Justification:
We report the compute resources used in our work in Appendix A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp/////errppp..////ppb///aaia
Answer: [Yes]
Justification:
This project conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We don't discuss the social impact of this project in the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
23

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
There are no such risks in this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cited the origianl paper for the SAPIEN synthetic dataset used in our paper. The asset for SAPIEN follows the MIT license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version URL.
asset used and, possible, include
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Codes and data will be released upon acceptance, together with instructions for use and license.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper focuses on non-living objects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper focuses on non-living objects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "bIxGEuuMRV", "reviewer": "Reviewer_JqAE", "summary": "The paper proposes a method for modeling articulated objects with neural radiance fields (NeRF). It employs a stage-wise training schema, first building a NeRF of the object in a reference configuration. Then a segmentation in parts and relative pose-changes are learned in an alternating fashion. A modified rendering equation and a corresponding volume sampling strategy are introduced that take into account the rigid deformations of the object parts.", "questions": "- How does the number of iterations used at each stage affect the final result? Is the method sensitive to these hyper-parameters?\n- Can the method handle joints with more than one degree of freedom?", "limitations": "Limitations are discussed in the text. I think it would be important to discuss also challenges and limitations of multi-part object reconstruction.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper considers unsupervised segmentation in parts by adding a segmentation head to the \"static NeRF\". The latter is learned by using a number of images capturing the object in a reference configuration, while part segmentation and relative transformations are learned by using a number of images capturing the object in a different configuration. \n\nThe proposed method approaches the problem of articulated object reconstruction with stage-wise training, while segmentation and pose estimation are learned in a decoupled way. The experimental evaluation shows that this approach performs better with respect to competing methods. In addition, the proposed method is able to reconstruct objects with multiple parts.", "weaknesses": "The text contains a few typos and other errors, like incomplete sentences, missing verbs etc. Some editing of the text is required. \nAdditionally, the clarity of the presentation can be improved. It is true that the stage-wise learning schema is somewhat involved, hence some additional effort should be devoted in explaining the steps as clear as possible, especially in Section 4.2. One suggestion is to provide a more detailed explanation of Figure 3, and provide dimensions of matrices U, F etc. Also, the formula in L.167-168 does not seem to be correct, especially regarding how the viewpoint v' is used. Should v' be an argument of a function?\n\nAnother aspect that is missing regards the efficiency of the proposed method. It is important to discuss training and inference times in comparison to other baselines. On a similar note, a discussion on how the number of iterations used at each stage affects the final result could be included in the ablations.\n\nRegarding experimental evaluation, ideally, comparison could also consider state-of-the-art articulated object reconstruct methods which are not based on NeRFs, such as  [R1] and [R2].\n\nFinally, not enough details are shared with respect to the models used, making reproducibility of the approach challenging.\n\n[R1] Kawana & Harada, Detection based part-level articulated object reconstruction from single RGBD image. NeurIPS 2023\n\n[R2] Kawana, Mukuta, & Harada, Unsupervised pose-aware part decomposition for man-made articulated objects, ECCV 2022\n\n### Minor comments\n- L.10: incomplete sentence\n- L.71: needs rephrasing\n- L.105: \"And the opacity value ...\" incomplete\n- L.199: It is fine that the same subset of 3D PartNet-Mobility dataset used in [15] is considered for the comparison, but it would be interesting to report results on a wider selection of objects\n- Section 5: the order in which figures are presented is not following the text, this creates some confusion to the reader\n- L.270: \"performs consistently performs\"\n- L.292: \"presented in second and third row\" does not follow the structure of the figure", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the review!\n\n1. **Typos and other errors**: Thanks ! We will fix them in the final version.\n\n2. **Clarity:** ```explaining the steps in Section 4.2, provide dimensions of matrices U, F, the formula in L.167-168 ... how the viewpoint v' is used```\n\nWe use homogeneous coordinates for representing $X_{\\ell}$ and convert it to a $4 \\times k_\\ell$ matrix by appending an extra 1 as scale to each of its 3D columns before plugging into the projection formula $U_{\\ell} = K M_{\\ell}^{-1} {v'} X_{\\ell}$. Let $k_\\ell$ denote the number of 3D points in $X_\\ell$. In the projection formula, camera intrinsic parameters $K$ is $ 3 \\times 4$, the inverse part-pose transformation matrix $M_\\ell^{-1}$ in shape $4 \\times 4$, camera viewpoint (extrinsic parameter) $\\mathbf{v’}$ in shape $4 \\times 4$, the projection result $U_\\ell$ in shape $3 \\times k_\\ell$.  Finally, we normalize each column of $U_\\ell$ with its last entry to recover the real pixel location after projection. We will clarify these points in the final version.\n\n3. **Efficiency**: ```Another aspect that is missing regards the efficiency of the proposed method```\n\nOur method takes on average about 15 minutes. The inference time for 2-, 3-, 4-part objects are 0.58, 0.87, 1.23 seconds. While the PARIS takes 5 minutes and 6 VRAM for training and the inference time for a 2-part object is 10.20 seconds on average. The above figures are tested with AMD 7950x CPU and Nvidia RTX 4090 GPU. We will include these details in the supplementary.\n\n4. **Hyperparameter sensitivity**: ```How does the number of iterations ... Is the method sensitive to these hyper-parameters?```\n\nWe use the same number of iterations, 6000 for optimizing $M_\\ell$ and 2000 for optimizing $s_\\ell$ in our method over all object instances (see Line 193). This ensures that the training converges. We observe that the variations in the results are negligible after the given number of iterations. We will provide a quantitative sensitivity analysis in the final version.\n\n5. **Implementation details**: ```not enough details with respect to the models```\n\nWe built our part-aware NeRF model based on the proposal estimator in [R3] and the NGP-based radiance field in [R4]. We extended the resulting model for segmentation by adding 2-layer MLPs with ReLU activation and the hidden dimensionality 64 in both estimator and radiance field. More details will be added to the supplementary. We will further release our code and data with the final version.\n\n[R3] Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. CVPR 2022\n\n[R4] Instant Neural Graphics Primitives. ACM Trans. Graph. 2022\n\n6. **Comparison to prior work**: ```compared to state-of-the-art articulated object reconstruct methods ... such as [R1] and [R2]```\n\nThanks for pointing out the related work. The method in [R1] learns to model articulated objects from a single RGBD image by detecting parts and reconstructing parts separately. Unlike our method that is unsupervised, it is supervised and requires ground-truth for joint parameters, joint-type and part-level oriented bounding boxes. In addition, our method estimates 3D from a set of 2D images without requiring measurements from a depth sensor.  The method in [R2] is an unsupervised method that learns to model articulated objects from point cloud data. It assumes a dense and accurate point cloud per object instance as input, while our method focuses on the more challenging task of estimating 3D from a set of 2D images alone. Hence, their results are not comparable to ours due to different input types and levels of supervision. We will include these works in our related work section.\n\n[R1] Kawana & Harada, Detection based part-level articulated object reconstruction from single RGBD image. NeurIPS 2023\n\n[R2] Kawana, Mukuta, & Harada, Unsupervised pose-aware part decomposition for man-made articulated objects, ECCV 2022\n\n7. **More extensive experiments**: ```... report results on a wider selection of objects```\n\nThanks for the suggestion. We reported our results in the benchmark that was used by the prior work, and additionally evaluated our method on objects with multiple parts and a real-object in the submission. We also provided an additional experiment on doors where the static part is smaller and a simple 4-parts chained object in the one-page pdf. In the final version, we will include examples from more categories such as door, fan, monitor that have not been included in the original subset. \n\n8. **Modelling multiple degrees of freedom**: ```Can the method handle joints with more than one degree of freedom?```\n\nOur method is not limited to estimating a single degree of freedom (DoF), as we do not restrict the part transformations to a single DoF. However, the dataset included single DoF joint objects only. Hence, we provide an extra experiment of a chained 4-parts object and estimate the 6 DoF $se(3)$ transformation for each movable part in the one-page PDF.\n\n\n9. **Limitations:** ```... to discuss also challenges and limitations of multi-part object reconstruction.```\n\nOne challenge for multi-part object reconstruction is to reconstruct the occluded parts, as occlusions between parts are more likely to occur more when the number of parts increases. For example, in the two-drawer storage shown in Figure 11 of the supplementary, the drawers will be partially occluded in both articulation status. \nAnother challenge would be about taking physical constraints into consideration. For example, the segmented parts should not have collisions with other parts, or the connected joints should not be detached during articulation pose estimation. Third, recovering complex kinematic structures like chained parts is still an open problem. Finally, a dataset consisting of objects with more complex kinematic structures is currently missing. Such a dataset would enable advancing the progress in articulation object modeling.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RLsqInenXK", "reviewer": "Reviewer_DWX2", "summary": "This paper presents an unsupervised framework that jointly learns articulations and part segmentations of objects with rigid parts from multi-view images.\n\nSpecifically, they proposed a two-stage approach. In the first stage, a static NeRF is fitted to one of the object states. In the second stage, the optimization alternates between optimizing part assignment and part pose estimations. The key to their approach is a 3D voxel grid heuristic that helps initialize the part assignment.\n\nThe proposed method shows promising quantitative and qualitative results in view synthesis, pose estimation, and part segmentation compared to the previous method PARIS. Notably, the proposed method also shows more consistent results across multiple runs.", "questions": "Below are some questions that I have:\n- A concurrent work [1] tackles the same topic and similarly achieves impressive results. It would be nice if the authors briefly discuss the work (what’s in common, and what set the two apart). Note that [1] was not yet published at the time of NeuRIPS submission, so this is just a nice-to-have discussion, and will not affect the final rating.\n\nTypo:\n- L212: geodesic distance should be e_g?\n\n[1] Neural Implicit Representation for Building Digital Twins of Unknown Articulated Objects, CVPR 2024", "limitations": "The paper adequately addresses the limitations in Section 5.5 and Section 6. It is also pretty awesome that the paper also presents failure cases and further analysis in Supplementary (A.3)", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presented approach makes sense and shows remarkable results compared to the baseline. In particular:\n- No labels are required for training.\n- A single NeRF plus a small segmentation head is trained, making it more parameter-efficient compared to the baseline (PARIS).\n- The learned articulated structures show better geometry accuracy/consistency during motion, compared to the baseline approach.\n\nOverall, the proposed method is promising, the writing is easy to follow, and the problem it tackles is of great importance for a wide range of applications in robotics.", "weaknesses": "While the approach looks promising, it does have some room for improvement\n- As can be observed in the supplementary video, the segmentation is still noisy, containing floaters that have nothing to do with the moving part.\n- Using pixel difference as the heuristic for tagging moving parts is potentially problematic. For example, in a real-world setting, the constantly changing environment light produces color differences across views/states, which can greatly affect the tagging accuracy. \n- The examples presented still have a rather simple articulated structure (i.e., no complicated kinematic chains that connect multiple parts). Note that this is an open problem/common issue not specific to the proposed method.\n- The approach is limited to rigid deformation (as discussed in L303-304). Learning a deformation field may alleviate this problem, but this is out of the scope of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper!\n\n1. **Problematic heuristic**: ```Using pixel difference as the heuristic for tagging moving parts is potentially problematic```\n\nWe do not use RGB pixel values but tag the moving parts based on the opacity difference (the alpha channel for RGBA images), which won’t be affected by the RGB values (see line 154). Thus, it works robustly even in adverse and fast-changing lighting conditions. Furthermore, as we shown in the supplementary, Fig 8.a, our pipeline is robust to noisy tagging initialization. \n\n2. **Simple examples**: ```The examples presented still have a rather simple articulated structure. Note that this is an open problem/common issue not specific to the proposed method.```\n\nThanks for pointing out this limitation in our work. In case of multiple connected parts, our method would provide a global transformation of parts. In the presence of known kinematic structure, one can estimate the local transformations for each part. We would like to note that structure estimating is an actively studied problem by itself and out of scope for our paper. \n\n3. **Limitation:** ```The approach is limited to rigid deformation (as discussed in L303-304). Learning a deformation field may alleviate this problem, but this is out of the scope of this work.```\n\nThis is indeed a discussed limitation of our method. Rigid articulated objects are ubiquitous in our daily life and hence modeling such objects is a valuable and challenging problem. We plan to explore different deformation parameterizations including non-rigid ones in our future work.\n\n4. **Discussion for concurrent work [R1]**\n\nThe concurrent work uses two sets of RGBD images for different articulation status as input. Thanks to the depth measurements, they could first reconstruct two 3D mesh models for different articulations and then compute part-level transformation based on 3D point correspondences. Part-segmentation is obtained by grouping points that share similar motions. In contrast, our method does not require measurements from a depth sensor and focuses on a more challenging problem setting. Hence we build a NeRF model based on one articulation status and utilize the 2D information from the second image set for part-segmentation in the NeRF space and part motion estimation.  \n[R1] Neural Implicit Representation for Building Digital Twins of Unknown Articulated Objects, CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cpc6hmNaeb", "reviewer": "Reviewer_1D4A", "summary": "This paper introduces a novel method for decomposing articulated objects and predicting their articulation. The pipeline is trained without supervision. Initially, a \"static NeRF\" of the object's initial state is obtained. The method then employs part-aware rendering to optimize the pose-change tensor and object segmentation in a decoupled manner. Finally, it reconstructs the voxel grid and performs refinement to achieve high-quality results.", "questions": "1. To optimize the pose-changing tensor, it requires the difference between the target and initial views. What if the object does not have static parts? For example, when scissors cut something, both parts are moving.\n2. The results presented in PARIS are not as poor as those in this paper, which the authors should address.", "limitations": "The paper discusses the limitations and shows some failure examples.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The pipeline is end-to-end and the training is completely unsupervised.\n2. The video results are provided and look impressive.", "weaknesses": "1. There are still many artifacts that can be seen in the visualization results\n2. The method only works in limited cases. Most of the examples have one or two joints and a large static part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper.\n\n1. **Artifacts for visualization:** ```There are still many artifacts that can be seen in the visualization results```\n\nThe artifacts during rendering are indeed caused by the imperfect segmentation in the NeRF space, though they do not harm the pose and joint estimation performance. A challenge of segmenting in the implicit 3D NeRF space is that it is non-trivial to apply regularization to smooth the segmentation results as in standard segmentation methods that work in explicit 2D space. In principle, the artifacts can be reduced with more sets of images from different articulated poses. In future work, we plan to move to explicit 3D representations such as Gaussian Splatting to better regulate the segmentation masks.\n\n\n\n2. **More evaluation:** ```“the method only works in limited cases. Most of the examples have one or two joints and a large static part.”```\n\t\nA key advantage of our method over the prior work is its ability to model multiple parts. For the rebuttal, we have further evaluated our model on two more objects, a door with its frame (door-8867) where the frame is static and significantly smaller than the door, from the Sapiens dataset. And the other object is a simple object with 4 chained parts. In the first experiment, our model obtains better performance in all measured metrics compared to PARIS. In the second experiment, we demonstrate our method can model motions of movable parts in chained objects. Yet, how to recover the chained kinematics would be an open question and is out of scope in this work. Detailed results can be found in the attached 1-page PDF. We will include these results along with more such objects in the final version.\n\n\n3. **Limitation:** ```“What if the object does not have static parts? For example, when scissors cut something, both parts are moving.”```\n\t\nIndeed our method assumes one part of the object is static to the canonical frame in different articulated poses, which is a common limitation to ours and as well as the prior work. In such cases, the challenge is to segment and track object parts simultaneously. A potential solution could be ingesting a video input with multiple frames and using optical flow to track and cluster parts. However, modeling from video data is out of scope in this paper. \n\n4. **Baseline performance:** ```The results for PARIS are far worse than the original paper, need to check on it.```\n\nAs we pointed out in the footnote in page 6, similar problems for reproducibility for PARIS have been acknowledged as challenging by other research in issues on the official repo. The author’s response to the reproducibility issue in the official code repo is:`\n\n```“the training for each object is essentially an optimization process, which can be easily affected by the randomness. I was also encountering the unstable training issue you mentioned in my practice”```\n\nFor a fair comparison, we follow the default setting provided from the official repo, and reproduce the result on the officially released data. To account for the randomness, we report mean and std metric for 5 independent runs in our paper. In this setting, we can see that our methods are more robust across different objects and delivered more consistent performance compared to PARIS.\n\nBesides, concurrent work [R1] also reports the reproduced performance of PARIS using the officially released data, which also shows similarly lower performance for PARIS. \n\n[R1] Neural Implicit Representation for Building Digital Twins of Unknown Articulated Objects, CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FD6PwCHC97", "reviewer": "Reviewer_AUv9", "summary": "This paper presents an unsupervised method to learn the pose and part-segmentation of articulated objects with rigid parts using conditional view synthesis. Their approach learns the NeRF of both states and extracts 3D voxels for optimizing the pose and segmentation.", "questions": "Please see the Weakness section.", "limitations": "I agree with the limitation section, but, I find the method for finding U (Line 170) to be insufficiently robust. If an issue arises, all subsequent operations will fail.", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "Using 3D Voxel instead of mesh provides another approach.", "weaknesses": "Firstly, my main concern is about the novelty and contribution of this work. \n\nCompared to the work PARIS, which this paper frequently references, I did not find any significant innovative points. In fact, the method appears to have regressed. This method separates the optimization of pose and segmentation, first optimizing the pose and then optimizing the segmentation. In my opinion, this is not as effective as jointly optimizing them, since the segmentation results can also affect the pose optimization results. However, the 3D voxel approach used in this paper is not differentiable (Lines 150-151), so it cannot jointly optimize both. \n\nAnd the voxel grid refinement operation seems more like a simple engineering post-processing. \n\nAdditionally, I have some issues with the experimental results. The values reported for the existing method PARIS in this paper differ from the results reported in the original paper. Have the authors considered comparing their method with other existing methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper.\n\n1. **Alternating optimization:** ```This method separates the optimization of pose and segmentation, first optimizing the pose and then optimizing the segmentation. In my opinion, this is not as effective as jointly optimizing them, since the segmentation results can also affect the pose optimization results.```\n\nWe agree with the reviewer that the pose and segmentation solutions depend on each other and, however, we disagree that joint optimization is guaranteed to obtain better results. It is well known that simultaneously updating intertwined parameters can be challenging and result in poor solutions. Hence, we use a well-known principled optimization strategy, coordinate block descent [R1] that optimizes first pose and then segmentation iteratively. We also quantitatively supported our argument that our method with the suggested joint optimization (without the decoupled pose estimation (DP) and iterative refinement (IR), the first row in Table 5) performs poorly. Here, the segmentation head and the articulated pose tensor are jointly optimized during the training (Line 276 - 277). \n\n[R1] Wright, Stephen J. \"Coordinate descent algorithms.\" Mathematical programming 151.1 (2015): 3-34.\n\n\n\n2. **Novelty:** ```Compared to the work PARIS, which this paper frequently references, I did not find any significant innovative points.```\n\nWe refer the reviewer to the originality point in the reviewer guidelines of the conference and invite the reviewer to provide a more substantiated comment on their understanding of “significant innovation”. Our method significantly differs from PARIS in three ways. First, it only requires a single neural radiance field (NeRF) for each object, while PARIS requires one for each part. Second, our method can model objects with multiple moving parts by segmenting NeRF space, while PARIS is limited to two part objects (with one as static and the other one as dynamic). Third, our method performs consistently across different objects unlike PARIS thanks to the proposed initialization strategy and iterative 3-step optimization. Clearly, our method is composed of a novel combination of multiple steps which has not been applied to the related problems, and significantly differs from the prior work in terms of methodology and capability.\n\n\n3. **Non-differentiable representation:** ```the 3D voxel approach used in this paper is not differentiable (Lines 150-151), so it cannot jointly optimize both.```\n\nThe 3D voxel strategy is used only for initializing the optimization (see Line 150-151). Therefore, it is not required to be differential. While the initialization can be noisy, we iteratively refine it by acquiring estimates from the segmentation head later and demonstrate its effectiveness in Table 5.\n\n\n4. **Simple post-processing:** ```And the voxel grid refinement operation seems more like a simple engineering post-processing.```\n\nThe voxel grid refinement is not a post-processing step but an update procedure in the optimization that updates the 3D coordinates based on the estimation from the segmentation head to enable more accurate pose estimation in the next iteration. Please refer to line 185-194 for more details. \n\n\n\n\n5. **Reproducing the prior work:** ```The values reported for the existing method PARIS in this paper differ from the results reported in the original paper.”```\n\nThe reviewer missed that we already pointed out the reproducibility issue of PARIS in the footnote of page 6. Similar problems for reproducibility for PARIS have been acknowledged as challenging by other research in issues on the official repo. The author’s response to the reproducibility issue in the official code repo is:\n\n```“the training for each object is essentially an optimization process, which can be easily affected by the randomness. I was also encountering the unstable training issue you mentioned in my practice”```\n\nFor a fair comparison, we follow the default setting provided from the official repo, and reproduce the result on the officially released data. To account for the randomness, we report mean and std metric for 5 independent runs in our paper. In this setting, we can see that our methods are more robust across different objects and delivered more consistent performance compared to PARIS.\n\nBesides, concurrent work [R2] also reports the reproduced performance of PARIS using the officially released data, which also shows similarly lower performance for PARIS. \n\n[R2] Neural Implicit Representation for Building Digital Twins of Unknown Articulated Objects, CVPR 2024\n\n\n6. **Comparison to other works:** ```Have the authors considered comparing their method with other existing methods?```\n\nWe have compared our method to the most recent and related work, PARIS. If the reviewer is aware of other related work which was not discussed in our submission, we are happy to provide further discussion.\n\n7. **Robustness:** ```I find the method for finding U (Line 170) to be insufficiently robust. If an issue arises, all subsequent operations will fail.```\n\nFirst, the U mentioned in line 170 refers to the projected 2D points on the image space, which is obtained using standard camera projection formula from 3D points $X_\\ell$ but not found by our method like the 3D coordinates $X_\\ell$. If the reviewer is referring to the 3D coordinates, we already provided rigorous evaluation of our method on multiple categories of objects and also further study the robustness of our method in case of noisy and incomplete initialization. We refer to Figure 2 and Figure 8 for more details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global comment
We sincerely thank all reviewers for their valuable and insightful comments. We are particularly encouraged by the positive feedback received and appreciate the opportunity to address the concerns raised. We are happy to address these common issues comprehensively in the following sections.
## More examples for evaluation (JqAE, DWX2, 1D4A).
- Reviewer **JqAE**: ```it would be interesting to report results on a wider selection of objects.```
- Reviewer **DWX2**: ```The examples presented still have a rather simple articulated structure.```
- Reviewer **1D4A**: ```the method only works in limited cases. Most of the examples have one or two joints and a large static part.```

To address the concerns for evaluation, we carried out extra experiments for the object door with its frame, which is not included in the PARIS evaluation set (**JqAE**) and it has a larger movable part compared to the static part(**1D4A**). The quantitative results of our methods consistently outperform the PARIS.
Additionally, we use our method to estimate the motions of 4-part chained objects. Since recovering the chained kinematic structure is out of scope in our work, we estimate the 6-DoF se(3) transform for each movable part (**JqAE**, **DWX2**, **1D4A**). Qualitative evaluations show the robustness of motion estimation of our method and details can be found in the one-page PDF.
## Artifacts for visualization (DWX2, 1D4A)
- Reviewer **DWX2**: ```As can be observed in the supplementary video, the segmentation is still noisy, containing floaters that have nothing to do with the moving part.```
- Reviewer **1D4A**: ```There are still many artifacts that can be seen in the visualization results```

The artefacts during rendering are indeed caused by the imperfect segmentation in the NeRF space, though they do not harm the pose and joint estimation performance. A challenge of segmenting in the implicit 3D NeRF space is that it is non-trivial to apply regularization to smooth the segmentation results as in standard segmentation methods that work in explicit 2D space. In principle, the artefacts can be reduced with more images from different articulated poses. In future work, we plan to move to explicit 3D representations such as Gaussian Splatting to regulate the segmentation masks better.

Meta-review: Post rebuttal, reviews were split. Three reviewers were in favor of accepting the paper. They argued the work tackled an important and challenging problem, used a method that would be a good contribution, thought the experiments demonstrated the value of the work, and overall believed that the paper would be a good addition to the overall literature. 

One reviewer was strongly opposed to the work, claiming that the method wasn't a substantial enough contribution to the literature, and that it should be compared with more work. One concern was the paper's baseline results differed from the published results. The AC is convinced by the authors' explanation of that difference is by the instability of the baseline methods (which was noted in the NeurIPS submission, in the github issues for the paper in question, and acknowledged in other papers too). The authors' approach of running methods five times and reporting the mean and standard deviation seems a sensible mitigation. A second point of disagreement was a comparisons with a CVPR 24 paper that appeared on Arxiv before the NeurIPS submission deadline. The AC agrees with the authors and one of the other reviewers that the paper is concurrent, following the NeurIPS author FAQ's guideline of 2 months. Comparisons might be a nice addition for the paper's impact, but are not necessary.

Overall, the AC agrees with the reviewers recommending acceptance. This decision was based on the paper, the reviews, the rebuttal, and the discussion after the rebuttal. The AC asks the authors to incorporate their responses to the reviewers into the final version of the paper.
Decision: Accept (poster)