General Articulated Objects Manipulation in Real Images via Part-Aware Diffusion Process

Authors: zhou  fang
yong-lu  li
lixin  yang
cewu  lu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 08:39:24

Abstract: Articulated object manipulation in real images is a fundamental step in computer and robotic vision tasks. Recently, several image editing methods based on diffusion models have been proposed to manipulate articulated objects according to text prompts. However, these methods often generate weird artifacts or even fail in real images. To this end, we introduce the Part-Aware Diffusion Model to approach the manipulation of articulated objects in real images. First, we develop Abstract 3D Models to represent and manipulate articulated objects efficiently and arbitrarily. Then we propose dynamic feature maps to transfer the appearance of objects from input images to edited ones, meanwhile generating novel views or novel-appearing parts reasonably. Extensive experiments are provided to illustrate the advanced manipulation capabilities of our method concerning state-of-the-art editing works. Additionally, we verify our method on 3D articulated object understanding for
embodied robot scenarios and the promising results prove that our method supports this task strongly. The project page is https://mvig-rhos.com/pa_diffusion.

Main paper: 
 introduction: 
 Image editing is a long-standing popular computer vision task. Specifically, manipulating articulated objects has garnered significant attention owing to its application in various fields, such as image augmentation for downstream tasks [43] , building goal conditions to train reinforcement learning models for robotic manipulation [41, 55] , creating videos with extra supervision information [33] , detecting human-object interactions [21, 22, 25] , reasoning object affordance [24, 23] , etc. Thanks to the large-scale training data and immense computing power, diffusion-based [40] generative models have achieved surprising results in the field of image and video generation.Inspired by these successes, several recent works have adopted diffusion models as the backbone and implemented text-guided object manipulation [19, 15, 7, 51] . We can properly divide these studies into a couple of groups. The first one is to directly edit 2D images by transferring the feature/attention maps from original images to edited ones such as [12, 31, 7] . However, weird artifacts are prone to appear when the objects are rotated and deformed, or novel views appear. Consequently, these methods are restricted to structure-preserving image editing. Another group relies on reconstructing 3D object models. As the most related work to ours, [51] reconstructed 3D object models for manipulation and projected them back to images later. Nevertheless, this approach depends on the quality of reconstructed 3D models heavily. And the reconstruction model has to be fine-tuned when dealing with new categories. Moreover, manipulation has to be done manually which is laborious and impractical to support editing large quantities of images.To address these problems, we propose the Part-Aware Diffusion Model (PA-Diffusion model) for articulated object manipulation in real images, as illustrated in Fig. 1 . Firstly, we introduce the concept of Abstract 3D Model and build a Primitive Prototype Library to represent articulated objects in 3D space, so that our method can not only cover many common objects but also handle novel categories without extra training data or fine-tuning processes. Besides, arbitrary manipulation can be done efficiently. Second, we proposed dynamic feature maps to assist generation models in accurately transferring object appearances to accurate locations in edited images. As a result, weird artifacts are eliminated, and meanwhile, novel views or novel-appearing parts are generated more reasonably. Finally, owing to the simple manipulation and editing process, the procedure is brief and the model can strongly support other tasks by editing a large volume of images.Our main contributions are summarized as follows:(1) We introduce the concept of the Abstract 3D Model which accurately and robustly represents various articulated object categories with primitive prototypes. Meanwhile, novel categories can also be incorporated quickly. In addition, the articulated objects can be efficiently manipulated with text instructions or human interactions in 3D space.(2) We propose dynamic feature maps that let the diffusion model comprehend the object structure. Consequently, the diffusion model can generate novel views or novel-appearing parts of objects reasonably and preserve the appearance of the seen parts simultaneously.(3) We present comprehensive experiments to highlight the advantages of our PA-Diffusion model including comparing with state-of-the-art editing methods both qualitatively and quantitatively, choosing a 3D articulated object understanding experiment to demonstrate how our method supports the tasks in embodied robot scenarios. 
related work 2.1 diffusion model for image generation: 
 In recent years, diffusion models [39, 38, 10] have achieved great success in the fields of image/video generation [6, 16] , segmentation [4, 50] , and many downstream computer vision tasks. To make the generation results controllable, [34] first proposed to extract and incorporate text features into the denoising process. Following this concept, [40, 11, 42, 14, 8] improved the performance of text-guided diffusion models with more effective text embedding methods.However, as an implicit instruction, text guidance is still not strong enough to finish fine-grained image control such as determining the image layouts, objects' shape and texture, and so on. To make up this gap, [46] provided structural guidance by enhancing the similarity between the features of other conditions and the text guidance. [13, 5] proposed to modify the cross-attention maps and then guide the denoising process. To handle more complex scenarios and achieve more precise control, [52] and [32] proposed adding an extra module to the diffusion model. Then extra condition information can be imported to guide the denoising process.Considering the remarkable capability of understanding images, several recent works have also reported editing real and synthetic images with using diffusion models as the backbone. These methods can generally be summarized into two groups: Inversion-Based and Feature-Sharing Based.The first group is primarily based on adding extra control to the inverted noise maps of images, then re-generating the image such as [19] . However, because the deterministic DDIM sampling process cannot be reversed perfectly, these methods struggle to preserve the appearance of original objects and backgrounds precisely. The second group attempts to maintain the appearance of objects by transferring the feature/attention/activation maps between guidance and generation branches or by adding extra loss items during the denoising process, as seen in [7, 31, 12] . Recent approaches like DragGAN [35] and DragDiffusion [31] propose to utilize a point-to-point dragging scheme, which can achieve refined content dragging. Nonetheless, these approaches often perform poorly on articulated object manipulation in real images, resulting in weird and blurry artifacts in edited images.2D-3D-2D is another promising way of image editing, the recent work [51] introduced reconstructing 3D models from 2D images and projecting them back after manipulation. However, this approach highly relies on the quality of 3D reconstructed models, and reconstructing 3D models from a single 2D image is still a challenging task.In contrast to the aforementioned approaches, our method demonstrates advantages when manipulating articulated objects in real images -high fidelity edited images, easy and arbitrary manipulation, covering multiple categories, and incorporating novel categories quickly. 
overview: 
 In this session, we go through the proposed PA-Diffusion model in detail. The overall architecture is demonstrated in Fig. 2 . Initially, we reconstruct abstract 3D models for articulated objects with the Primitive Prototype Library. Then arbitrary manipulation can be done according to text instructions or human interactions. Next, leveraging DDIM Inversion [44, 30] , initial inverted noise maps are created and manipulated following the previous actions. During the generation stage, we introduce dynamic feature maps, including manipulated inverted noise maps and compositional activation maps. These ensure that the appearance of seen parts of objects can be preserved accurately and that novel-appearing parts are generated reasonably. Besides, Texture and Style Consistency Score Loss are introduced to alleviate the blurry and style mismatch problems.Diffusion models aim to convert random Gaussian noise into high-resolution images through a sequential denoising and sampling process [12] . Given the conditioning y, we start from the initial Gaussian noise map z t , and then iteratively estimate the reduced noise εt at each time step t:EQUATIONThe update function could be DDPM [17] , DDIM [44] , or other sampling methods. Nevertheless, conventional sampling from conditional diffusion models often fails to produce high-quality images that align well with the condition y. To enhance the effect of the desired condition, extra class loss guidance is added to the reduced noise during the sampling process such as Classifier or Classifier-free guidance [45, 18] .Classifier guidance is introduced to generate conditional samples from an unconditional model by combining the unconditional score ϵ t with a classifier p(y|z t ), where p(y|z t ) is the probability distribution of condition y based on the noise at time step t:EQUATIONClassifier-free guidance eliminates the need for a separate classifier by incorporating the class information directly into the generative model as follows: Following these concepts, custom energy functions can also be utilized to guide the denoising process, instead of the probability function. In [12] [29] [54] , various energy functions g are incorporated alongside classifier-free guidance to obtain high-fidelity samples as follows:EQUATIONEQUATIONOur proposed PA-Diffusion model is built on the diffusion model with classifier-free guidance. Extra energy functions are employed during the image editing process.As a promising workaround to the methods of dealing with images directly, the 2D-3D-2D pipeline has successfully handled many articulated objects with precise 3D models. Unfortunately, creating 3D models for various categories from a single image remains challenging, particularly for novel categories or instances. In this work, we introduce the concept of Abstract 3D Model that reconstructs accurate 3D models, supports efficient object manipulation, and incorporates novel objects easily.Abstract 3D Model. Unlike previous methods, there is no need for precise 3D models of our method, the conditional information we have to provide to the diffusion model is coarse sketch maps and part-level masks. Therefore, we introduce the use of an abstract 3D model to represent the articulated object. As an abstract 3D model, the object is represented by combining several basic prototypes. As depicted at the bottom of Fig. 2 , the laptop can be represented by two planes, storage furnitures and microwaves can be represented by a plane and a box. Primitive Prototype Library, which includes basic 3D prototypes such as cuboids, cubes, and boxes, supports common articulated object categories involving both rotation and translation joint types.Camera Alignment. Next, we compute the camera pose in 3D space and align the 2D real image view with the 3D space camera view. The pose computation problem is to calculate the intrinsic and extrinsic matrices for the camera that minimize the reprojection error from 3D-2D point correspondences [3] . Thus, in this work, we first employ Large-scale Segmentation Models to obtain the initial part-level segmentation masks of articulated objects M Init and then detect the extreme corner points A, B, C, D (pts 1 ) with simple corner detection functions. These 2D extreme points are aligned with their 3D counterparts A ′ , B ′ , C ′ , D ′ (pts 2 , pre-defined in Primitive Prototype Library) as shown in Fig. 3 . Finally, based on Perspective n-Points 2D-3D method [49] , the camera matrices can be extracted, and 2D-3D views are aligned.Manipulation. By representing objects with primitive prototypes, multiple types of manipulations can be implemented in 3D space efficiently with the assistance of 3D computer graphics software. As shown in Fig. 2 , manipulating objects through text instructions is a concise approach. For example, the instruction opening the laptop 120 • is converted to a script, then the manipulation will be done by running the script in 3D software. Our PA-Diffusion model also supports human interaction, which could be even more efficient. Additional manipulation guidance is provided in the Appendix. Contrary to the tedious manipulation experience of previous SOTA works, our proposed PA-Diffusion model offers a more flexible approach to editing articulated objects.Structure Disentangle. Some object parts could still be seen after manipulation, and some unseen parts would appear. As shown in the top right part in Fig. 3 , the laptop shell can be seen in the input image. After opening the laptop, the shell can still be seen, however, the keyboard and screen are newly revealed. Therefore, to distinguish them and implement part-aware diffusion, we disentangle articulated objects into seen parts and novel-appearing parts. The appearance of seen parts should be consistent between input and edited images, and the style of novel-appearing parts should be consistent with the objects' overall appearance. In this work, we regard all the initial part-level masks M Init as seen. Then after manipulation, we obtain manipulated seen parts mask M Gen s and manipulated novel-appearing parts mask M Gen n , both of them are exported from 3D software automatically. We use M Gen to present the union of M Gen s and M Gen n .To maintain the object's appearance including color and texture, previous editing methods introduced a guidance branch to invert and re-generate the input image, and a generation branch to create the edited image, the attention/feature/activation maps are transferred from the guidance to the generation branch directly [7] . However, when changing the object location or shape during manipulation, directly sharing these maps would transfer the feature from the input image to undesired locations in the edited image. Furthermore, these methods cannot reasonably generate novel views or novelappearing parts.To overcome these problems, we propose dynamic feature maps including manipulated inverted noise maps and compositional activation maps. To keep appearance accurate, manipulated inverted noise maps transfer the feature of seen parts in input images to the manipulated seen parts in edited images. Simultaneously, to make novel-appearing parts reasonably, compositional activation maps let the diffusion model create these parts from random noise. The following content describes how to manipulate the noise maps, how to construct compositional activation maps, and how they work. For clarity, the process is presented in Fig. 3 , and an additional simpler explanation is provided in the Appendix.Manipulated inverted noise map. As shown at the top of Fig. 3 , we firstly reverse the input image to the initial inverted noise map z Init T with DDIM inversion. After 3D manipulation, we calculate the transform function T based on the initial M Init and manipulated seen part-level masks M Gen s and then compute the transformed inverted noise map z T ran T with T . Finally, the manipulated inverted noise map is created as the following equation:EQUATIONWhere M M ake s is the mask generated by XOR(M Init ,M Init ∩ M Gen s). M bg is the background mask created by 1 -M Init ∪ M Gen s . Besides, we also create a random noise map z Ran T . The three noise maps z Init T , z Ran T , and z M an T will be sent to the denoising UNet as a batch in the next step.Compositional activation map. As shown at the bottom of Fig. 3 , the diffusion model runs a pipeline denoising process with the three noise maps as a batch and generates three output activation maps: A Gui t generated from initial inverted noise map within guidance branch, A Ran t generated from random noise map within random branch, and A Gen t generated from manipulated inverted noise map within generation branch. At last, A Ran t and A Gen t are merged according to previously defined masks.EQUATIONOwing to the above steps, we transfer the feature of seen parts in input images to the accurate location in edited images. At the same time, all the other contents and the background in input images can be preserved, as the highlighted yellow part in Fig. 3 . Besides, as the highlighted blue part in Fig. 3 , an extra image is synthesized with random noise map z Ran T , the novel-appearing parts are cropped and pasted to edited images from the extra image, which makes these parts more reasonable and consistent with the original inputs.Texture Consistency Score Loss. However, simply manipulating the inverted noise map will lead to a serious blurry problem. This is due to the denoising process includes several convolution steps. As the initial inverted noise map z Init T is not rotation invariant, manipulating z Init T will disturb the original distribution and make the denoising process fail. To alleviate this limitation, we construct Texture Consistency Score Loss (TCSL) [31] as an extra supervision that lets the specific region in the generation branch match with the one in the guidance branch,EQUATIONwhere φ f g and φ bg are hyper-parameters. We add this loss item as an extra loss in classifier guidance in each denoising iteration step to calibrate the appearance of objects.Style Consistency Score Loss. For novel-appearing parts, the diffusion model is prone to randomly select a style to generate them with text guidance or sketch maps. As a result, the texture and style are usually different from the objects in input images. Therefore, we introduce Style Consistency Score Loss (SCSL) to calibrate the style of seen parts and the novel views and novel-appearing parts.Different from Texture Consistency Score Loss, there is no need to match every pixel in input images and edited images. Thus we calculate L1 loss between the activation maps of the guidance and the generation branch [12] . The loss function is as follows:EQUATIONThis loss item is also added as an extra classifier guidance. The final reduced noise in each denoising iteration is as follows, where γ 1 are γ 2 are the hyper-parameter weights of TCSL and SCSL,EQUATION 
experiment: 
 In this section, we provide two kinds of experiments to prove the advantages of our proposed PA-Diffusion model. First, various image editing tasks are conducted to showcase the model's image editing capabilities. To highlight the superiority of our model compared with state-of-the-art methods, we collect a testbench and evaluate all the methods both qualitatively and quantitatively. Second, we create a synthetic training set to support the challenging 3D articulated object understanding task in the robotic scenarios. In this work, we select Grounded Segment Anything [20, 28] to obtain the initial part-level object segmentation masks. T2I Adapter [32] is chosen as the conditional generation model, and the condition we used is the sketch map. The fundamental diffusion model is Stable Diffusion V1-5.All experiments run on a single NVIDIA A100 GPU. Notably, NO models need to be trained or fine-tuned in the image editing process.Primitive Prototype Library is built within Blender [9] . 3D cuboids, cubes, boxes, and other 3D primitive shapes are created and combined to represent different objects. In this work, 6 primitive shapes are created to represent 6 categories of articulated objects. The ease of creating prototypes allows for the quick incorporation of novel categories or instances. Rotation, view change, and other manipulations are all implemented in Blender.Fig. 4 demonstrates the editing results of some basic manipulations and a sequential manipulation process. As shown in the left part, our PA-Diffusion model naturally moves, scales/shears, rotates, and opens articulated objects with rotation or translation joint types. The edited objects blend seamlessly with other contents and backgrounds in the original images. When we move or rotate the objects, the blank regions in the background are in-painted semantically according to the surroundings. Moreover, in-painting and editing are completed in a single denoising process by the PA-Diffusion model, no extra in-paint model or process is required. Last but not least, novel-appearing parts of objects are generated reasonably. For example, the fruits are in the refrigerator, the storage furniture and the drawer are empty after opening.The right part of Fig. 4 presents a complete operation process of opening articulated objects, from the initial closed state to progressively open states as 30 • , 60 • , 90 • , 120 • . The appearance of objects' seen parts is transferred from the original input to different states accurately. The point that needs to be mentioned is that along with the operation progress, more novel-appearing parts of objects appear, our proposed PA-Diffusion model keeps the style and texture of these novel-appearing parts throughout the process. This capability allows our method to generate a complete manipulation video from a single input image, maintaining consistent object appearance and style even as novel views or parts increase.Generally, articulated objects' parts are rigid and connected with one of the typical joint typesrotation and translation. However, we are surprised to notice that our PA-Diffusion model can also handle non-rigid objects with non-uniform shapes and weird joints and manipulations fabulously. As illustrated in Fig. 5 , we first select toys as examples of non-rigid with non-uniform shapes, the tail of the shark is moved up together with other close parts as deformable objects. Then, we broke the cup in a real image. Third, the kitchen pot and storage furniture are opened to illustrate the case of weird and multiple joint types within one object. No matter whether the shape of object parts has changed after manipulation or joint types are unconventional, the PA-Diffusion model can edit all the objects successfully. Meanwhile, the background is preserved or inpainted very well.As mentioned in Section 3, TCSL is added to the denoising loss function to release the serious blurry problem. As shown in the left of Fig. 6 , we move the storage furniture to the right. It can be seen that without TCSL, the objects are prone to be blurry. More seriously, the edited image could be blurry, as in the storage furniture example. On the other hand, with TCSL, the texture of objects can be transferred to the desired location, meanwhile, the blank region caused by the movement is in-painted well. SCSL is another loss to keep the style consistent between seen parts and novel-appearing parts. Its effectiveness is shown in the left bottom of Fig. 6 (with SCSL). We notice that when opening the storage furniture, the style of the novel-appeared inner body part is more likely to be consistent with the door and outer body part with SCSL, which makes the edited image natural. More quantitative ablation studies about the two score losses are provided in the Appendix.To further evaluate our PA-Diffusion model, we compare it with four state-of-the-art image editing approaches that are based on diffusion models: Imagic, DragDiffusion, MasaCtrl (with T2I Adapter), and Image Sculpting. In this experiment, we require these models to manipulate different categories of articulated objects, including both rotation and translation joint types. The results are shown in the right part of Fig. 6 . It is hard for Imagic to finish the tasks as articulated objects cannot be opened at all or the wrong part is manipulated. This is because the text instruction is too weak and the fundamental generation model cannot understand the structure of objects. Similarly, DragDiffusion cannot finish the tasks even though human interaction is applied.MasaCtrl performs better than Imagic and DragDiffusion. The manipulation can be finished, while the edited images are either unrealistic or unreasonable. Take the laptop as an example (second column in the right part of Fig. 6 ), the object has been moved down and opened, while the region highlighted with the red bounding box in the edited image remains unchanged, which does not make sense. This issue is prevalent across other categories as well. The reason is that MasaCtrl simply shares the whole feature/attention maps between the input and edited image, features of seen parts cannot be transferred to the desired new location when objects move or the shape changes. Finally, Image Sculpting works well on storage furniture. However it is prone to fail to reconstruct precise 3D models of the laptop, trashcan, and drawer, consequently, the edited images are undesirable.In contrast, our PA-Diffusion model consistently produces high-fidelity and reasonable edits. The appearance of seen parts is kept accurate no matter whether objects move or the shapes of parts have changed. The novel parts are reasonable and semantically consistent with the original. To quantitatively evaluate our method, we built an articulated object manipulation testbench. The testbench comprises 6 object categories including storage furniture, laptop, trashcan, microwave, drawer, and refrigerator, which covers both rotation and translation joint types. In total, 660 real images are collected from the website. Considering articulated objects are typically rigid with uniform shapes, this testbench can represent the characteristics of common articulated object categories.The comparison methods we select are Imagic and MasaCtrl (with T2I Adapter). DragDiffusion is excluded as it cannot complete the manipulation tasks. Due to the long processing time and frequent failures in generating 3D models, Image Sculpting is also excluded here. To assess the realism of the edited images, the evaluation metric used is the Frechet Inception Distance (FID) score. The quantitative evaluation results are summarized in Tab. 1.Since Imagic relies solely on text instructions, the edited images often do not align well with the original inputs, resulting in poor scores. Due to previously discussed reasons, the edited images of MasaCtrl are confusing and lack coherence. Sequentially, the FID score is not satisfying. In comparison, the PA-Diffusion model outperforms other methods with an obvious improvement.In this session, we demonstrate how our proposed method supports the task of 3D articulated object understanding. As one of the fundamental steps to understanding 3D articulated objects, estimating the axes and surface normal is still challenging because of the lack of data. Annotating the axes and surface normal in 2D images is expensive and unreliable. On the contrary, leveraging abstract 3D models, our method can easily generate accurate 3D annotations of objects.To release the data limitation, we create a synthetic dataset with the PA-Diffusion model. The dataset includes 660 sequential samples, each sample includes the sequence of opening objects from 0 • to 120 • with step 30  
limitations: 
 Even though our method can handle common articulated objects, there are still some limitations. First, as edited images are generated from inverted noise maps, the quality of the original input images significantly affects the editing outcomes. Blurry or low-resolution inputs will degrade the edited images. Second, when the object undergoes substantial deformation, this editing method is likely to fail. Besides, manipulating deformable objects and fluids remains challenging with this approach. Further explanation is provided in the Appendix. 
conclusion: 
 This work introduces the PA-Diffusion model, a novel articulated object manipulation method that covers common object categories and supports arbitrary manipulation. Both the qualitative and quantitative experiments have proven the feasibility and effectiveness of our method. Besides, the 3D articulated object understanding experiment illustrates that the PA-Diffusion model has positive impacts on helping build robots that interact with the real world smartly. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: It is discussed in the Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for the responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?Answer: [NA] Justification: It is described in the Appendix. The code and data of this work have a quite low risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make the best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited, and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All the models and data used in this work are published, and all the citations have been included in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: No new assets are introduced in the paper.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This work has no crowdsourcing experiments and research.Guidelines:• The answer NA means that the paper does not involve crowdsourcing or research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To facilitate the understanding of our proposed PA-Diffusion model, we present the entire algorithm pipeline in Algorithm 1. eq.1, eq.2, and eq.5 refer to the equations in the main paper. Dynamic Feature Maps: To make it clear, we simplify the 64 × 64 × 4 inverted noise maps and activation maps to pure color block maps, as shown in Fig. 8 . As the following equations show, we calculate the transform function T ransf orm based on pts 1 and pts 3 , where pts 1 and pts 3 are corner points of the input image masks A, B, C, D (pts 1 ) and corner points of manipulated masks A " , B " , C " , D " (pts 3 ) as introduced in the main paper. The corner points are automatically detected with a simple corner detection function. For simple actions such as moving and scaling, affine transform [48] is selected, while for rotation, manipulation, and other complex actions, perspective transform [47] is required. And then we can get the transformed inverted noise map z T ran T by transforming the initial inverted noise map z Init T as following equations. Finally, the manipulated inverted noise map is calculated by adding z T ran T and z Init T . T = T ransf orm(pts 1 , pts 3 ),EQUATIONOn the other hand, the compositional activation maps are generated by adding A Ran t generated with random noise map and A Gen t generated with manipulated inverted noise map, as shown in the right part of Fig. 8 . Articulated Object Manipulation in 3D space: Manipulating the articulated objects in 3D space is straightforward in this work. As noted in the main paper, different manipulations can be done with text instructions or human interaction within Blender.For the text-based method, considering the objects can be manipulated with Python scripts in Blender, we first construct a table mapping the text instructions to actions. Then these actions can be implemented by running Python scripts. As shown in Fig. 9 , we require the laptop to rotate 30 degree, this text instruction is converted to Python script where the object matrix is multiplied with a rotation matrix and then set the new matrix to the object. Finally, running this script can finish the rotation action. For the second type, users can directly manipulate any parts of articulated objects in Blender, which is more flexible and convenient.In Fig. 10 and Fig. 11 , more articulated object manipulation results synthesized by our proposed PA-Diffusion model are demonstrated. Novel categories including door, toilet, and book are experimented with here. For various categories, joint types, and backgrounds, our proposed method can manipulate the objects and preserve other contents in the input images simultaneously.To analyze and explain our proposed PA-Diffusion model in detail, we provide more ablation studies in this session.Additional Loss Items: In the main paper, we qualitatively demonstrate the effect of TCSL and SCSL in the experiment part. Here, we evaluate them quantitatively. Following the main paper, the evaluation metric is the FID score. The results are summarized in Tab. 4. It is obvious that without TCSL, the performance degrades significantly since the feature cannot be transferred to the edited images correctly leading to inconsistent appearance compared to the input images. On the other hand, without SCSL, the style of novel-appearing parts is inconsistent with the original, therefore the outcome is still unsatisfactory. When including the two losses, the edited images are close to the input real images in the aspects of color, texture, and style.Primitive Prototype Library: In the main paper, we create 6 simple primitive prototypes to represent 6 different kinds of articulated objects in the testbench. Since our method does not require precise 3D CAD models of objects, Primitive Prototype Library can cover a wide range of articulated 4 : FID score of edited images with different additional losses: no SCSL and TCSL, with SCSL only, with TCSL only, and with all losses. The performance improves more than 57% with the assistance of the two losses. objects with a small number of primitive prototypes. As shown in Fig. 10 and Fig. 11 , books can be represented as laptops, doors can be represented with simple planes, and toilets can be represented with a plane and a box. No extra prototypes are required when creating abstract 3D models for novel categories. Besides, the edited images are still high-fidelity and high-quality. Furthermore, the Primitive Prototype Library is easy to expand, more primitive prototypes can be created rapidly when dealing with novel articulated objects. 2D-3D Models analysis: Besides the advantage of convenience, we also compare the quality of reconstructed 3D models with state-of-the-art 2D-3D methods and abstract 3D models.Following the method introduced in Image Sculpting [51] , we use ClipDrop to remove the background of input images, and then reconstruct 3D models with Zero123 [27] . The tested images are the same as those in the main paper. The reconstructed 3D object mesh models are shown in Fig. 12 including the front view, side view, and the edited images after manipulation. We notice that the result of storage furniture is acceptable, the shape closely matches the original, and the texture is stored in UV maps correctly. However, for other categories, the reconstructed models are poor which is the main reason for low-quality edited images.In the main paper, we mentioned that manipulating the reconstructed 3D models created by 2D-3D methods is tedious and inaccurate. As shown in Fig. 12 , the reconstructed models are not part-level (only one mesh object), tremendous human effort is required to cut the mesh into parts before manipulation. For example, when trying to split the door from the body of storage furniture, we need to cut the furniture into several parts first (top, bottom, four sides of the body), and then merge others except the door. This process, even with advanced 3D graphics software, is complex and time-consuming. As a result, summarizing reconstruction, manipulation, and generation time, Image Sculpting [51] requires over 10 mins to manipulate one image, making it unsuitable for large-scale image editing tasks. In summary, using abstract 3D models to present articulated objects offers several advantages: (1) State-of-the-art 2D-3D methods are still not robust enough to create precise 3D models such as laptops and trashcans. In comparison, it is easy to achieve abstract 3D models with primitive prototypes, as shown in Fig. 12. (2) Manipulating primitive prototypes in 3D space is easier and more accurate than manipulating a single 3D object mesh. (3) Seen parts and novel-appearing parts can be defined and extracted easily. (4) Novel categories and instances can be handled with our method efficiently. (5) Our method is time-efficient and thus can support various downstream tasks.Overview. In the main paper, we introduce the 3D articulated object understanding experiment and demonstrate how our proposed PA-Diffusion model supports other research fields. Here, we discuss the experiment setup and implementation in more detail. Annotation Generation. In previous works [37, 36, 26] , human labeling is required for annotating the bounding boxes, rotation/translation axes, and surface normal, which is labor-intensive and inaccurate, especially for 3D scenario. On the contrary, by representing objects with abstract 3D models, our method can achieve these annotations automatically.As shown in Fig. 13 , part1 and part2 are the masks of articulated objects' parts that are exported from Grounded SAM or Blender software automatically. The corner points can be calculated with [1, 2] . Then, the bounding box of each object part can be calculated with these corner points. The rotation and translation axis annotations are represented as [x 0 , y 0 , x 1 , y 1 ], where x 0 , y 0 , x 1 , y 1 are the coordinates of corner points in part-level masks, different object category uses different corner points. For the 3D surface normal annotations, as shown in Fig. 14 , we first export the object's part world transform matrix M obj and camera world transform matrix M camera . Then the two matrices are normalized and calibrated to create the aligned transform matrix M aligned . Consequently, we simply select the normal of the outer plane to represent the orientation of the object part, and the surface normal V surf is equal to the multiplication of the aligned matrix and local plane vector V plane :EQUATIONEvaluation Matrix. For quantitative evaluation, we follow [37] to calculate the average precision of the bounding box, axis, and surface normal. The bounding box is the traditional horizontal type, the threshold of IoU is set as 0.5. The predicted axes are measured with EA score as [53] . To demonstrate the results clearly, we calculate the surface normal error and measure the accuracy that the error is smaller than the threshold 30 • .The limitations of our proposed PA-Diffusion models have been discussed in the main paper. Due to the inaccuracy of DDIM inversion, the inverted noise map might be poor if the input image quality is low. Unfortunately, the poor noise map will lead to mismatch error accumulation and propagation during the iterative denoising process. As in the left of Fig. 15 , when the original input image is of low resolution (it is normalized to 512 × 512 before manipulation), the PA-Diffusion model cannot re-generate the original image with the inverted noise map. Simple actions like moving and scaling also cannot be completed.Manipulating the initial inverted noise maps is critical to preserve the appearance of seen parts. However, as discussed in the main paper, this step disturbs the original data distribution. The problem will be too serious to be fixed when the object shape deformation is large. As shown in the right of Fig. 15 , when reshaping the laptop to a slim non-uniform diamond shape, the object's appearance cannot be preserved.Considering this situation, one promising solution is to add stronger and more precise supervision loss in each denoising step. This is beyond the scope of this work, we plan to implement this later.In the future, more categories of articulated objects will be covered and the edited image dataset will be expanded to millions-scale for supporting various computer vision and robotic manipulation tasks.Next, we will extend this method to handle deformable objects and fluids.The articulated object manipulation method presented in this work has profound positive societal implications. This method can serve as a fundamental tool to benefit other computer vision or robot vision tasks. Consequently, the artificial intelligent algorithm can understand and interact with the real world better. Humans will have stronger AI assistants including smart offices, intelligent home or medical robots, and so on.All the models and data used in this work are collected from the Website. No personal information is used. The code and data created in this work have a low risk of misuse. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g. if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no crowdsourcing experiments and research in this work.Guidelines:• The answer NA means that the paper does not involve crowdsourcing or research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 


Decision: This manuscript describes an approach to image generation for articulated objects. The approach proceeds by providing guidance to the image interpretation process in the form of an abstract 3D model which captures the form of the articulation and allows the system to generate results that capture the intended variation. The paper was reviewed by a panel of experts who felt that the method was well presented and they appreciated the fact that the method was able to produce reasonably compelling results with relatively little user input. That being said there was significant discussion about the generality of the method and the need for the user to provide high level guidance. It is not clear from the results whether the system could be generalized to handle objects with larger numbers of degrees of freedom. While robotics is mentioned as a possible application area the experimental results do not show any examples of application to robotic arms with 3 or more joints. Nonetheless, there was a consensus that the method appeared to work reasonably well within the limited domain of abstract 3D models that were presented and readers may appreciate the insights that it offers

Reviews: review: paper_summary: paper_summary: The paper proposes a method for accurately generating edited images for manipulating articulated objects, effectively avoiding hallucinations. The overall idea and approach are very interesting. The implementation of the method is particularly impressive, presenting a novel articulated object manipulation technique that covers common object categories and supports arbitrary manipulation.
strength_weakness: strength_weakness: please refer to question section
questions: questions: 1. The description of section discussing manipulation in L122 is unclear. For instance, if an object has many movable parts, how is it determined which part should be moved? Taking the example in the figure, a laptop has two planes. If the text says "open laptop," how is it determined which plane should be moved?
2. The goal of "Abstract 3D Model" in L105 should be made clearer. For example, I am wondering why single-image reconstruction methods are not used to obtain the 3D model. Although I understand that using prototype reconstruction methods facilitates subsequent manipulation of arbitrary parts, I believe the authors should clearly explain the reasons and purposes behind each method detail.
3. In the abstract and introduction, the authors repeatedly mention that edited images are significant for robotic manipulation learning. Could authors explain in detail why edited images are beneficial for robotic manipulation? If conducting experiments is challenging, analyzing the advantages of image editing for robotics is also acceptable. For example, what limitations do existing image goal-conditioned manipulation policies have, and can this editing method address these limitations?
4. In Figure3, there are masks M in both top and bottom row. But why do the image M^{gen} on the top and image M_{s}^{gen} on the bottom look so different? I suppose mask should be in binary mask, but why M^{gen}  is so colorful?
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces a novel diffusion-based method for manipulating articulated objects in real-world images from input text guidance or human interaction. There are three main contributions in this paper: (i) the authors introduce the concept of an Abstract 3D Model, which eliminates the requirement for a 3D model by using primitives such as cuboids, boxes, etc to represent various types of articulated objects; (ii) the authors present dynamic feature maps, which transfer the seen parts from the input image to the generated image and only generate the novel parts, which is very intuitive; (iii) the test benchmark includes 660 object samples and a total of 3300 images.
strength_weakness: strength_weakness: -  While I like the concept of the Abstract 3D Model, I am not sure about its effectiveness when the input objects have shapes that do not conform to the available primitives, such as a round table, etc. In such cases, the part-level masks and sketch maps can be very different. 

- The authors' observation (L218) and the qualitative results do not well explain why articulated objects like refrigerators or furniture always appear empty when opened. I would expect it is randomly generated, with some cases showing objects, fruits and with other cases empty.

- The generated shadows in the output images also seem under-analyzed. In some instances, the shadows do not change even though the object has been articulated (e.g., some furniture samples in Figure 6).
questions: questions: Overall, I found the paper is well-written, and the contributions are strong and clear. However, I have a few questions to better understanding of the proposed methods and address the weaknesses mentioned above:

- Can the proposed methods effectively handle input objects whose shapes do not match the available primitives?

- Why are the drawers of opened objects consistently empty?

- Could you provide an analysis of shadows in the generated images, as this is a crucial factor in distinguishing whether the images are generated or real?
limitations: limitations: Yes, the authors clearly stated the limitation of proposed method (i) when input images are blurry, in low-resolution, (ii) when input objects are deformable and or fluids.
paper_summary: paper_summary: This paper presents a pipeline to directly manipulate articulated objects in real images and generate corresponding images in different articulated poses. The proposed method adopts a  2D-3D-2D approach with a heuristic model to obtain 3D information from source images and generate new images based on the diffusion model. No extra fine-tuning and training is required for the proposed method. Quantitative evaluation is presented for this method and qualitative comparison with baseline methods.
strength_weakness: strength_weakness: 1. How to obtain 3D information, part-level understanding and structural information is not detailed in Sec. 3.2, which is a key for articulated object manipulation.
2. The text instruction for manipulation generation is based on heuristic models, which may not be flexible.
3. Difficult to follow in Sec. 4.6. Lack of context for the reference work [1][1] (which is [32] in the paper) in this subsection, making it difficult to understand and validate.
   1. As the author mentioned in Sec. 4.1, **NO** training is needed, but Sec. 4.6 talks about dataset spliting and model training, very confusing.
   2. If the "model" is trained in this subsection, what model? with what objectives?
   3. Write the mathematical definition or explain the metrics used in this subsection, Tab. 3.
4. It seems the manipulation is not limited to part-level manipulation, but also applies to the object as a whole, such as move the object to the right. Could this method also extends to object-centric view synthesis? Would be interesting to see experiments on this.
5. It would be better to cite and compare to this similar work [2]

[1] Qian, Shengyi, et al. "Understanding 3d object articulation in internet videos."  *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* . 2022.

[2] Li, Ruining, et al. "Dragapart: Learning a part-level motion prior for articulated objects." *arXiv preprint arXiv:2403.15382* (2024).
questions: questions: The key questions for this paper focus on the Sec. 3.2.

For abstract 3D model generation. How the proposed method analysis the structural 3D information of the object given images? For example, in the Figure. 3, how does the pipeline obtain the following knowledge from the$Z_T^{init}$:

1. The object is composed of 2 parts, especially the base of the laptop is almost invisible (number of parts discovery)
2. These two parts fit with the plane prototypes (part to prototype fitting)
3. These two planes are connected	(part-level relation discovery)
4. One of the plane can be rotated along the connected axis (joint type estimation between revolute and prismatic)
limitations: limitations: See above
paper_summary: paper_summary: This paper focuses on manipulating articulated objects in 2D images by leveraging a 2D-3D-2D approach with a diffusion model. Specifically, the authors propose the abstract 3D model to represent the articulated objects and propose dynamic feature maps to transfer seen regions while generating novel areas. The results on image editing and 3D articulated object understanding show the effectiveness of this method.
strength_weakness: strength_weakness: My major concern about this approach is its limited application. Specifically, the current process of building abstract 3D models from 2D images might significantly hinder the generalization. The authors create 6 primitive prototypes to represent 6 categories in the dataset. However, this ignores the structure variance among instances in the same category (e.g., a cabinet could have 1-4 drawers and even combine with doors). Therefore, this method actually requires a pre-defined abstract 3D model for each structure. Manually designing an abstract 3D model for each image is clearly impractical, whereas automatically matching 3D prototypes to objects is not trivial, and many works have been done on this topic [1,2].

The process of defining an abstract 3D model for each category also injects crucial priors to the model (i.e., the structure, joint configuration, and kinematic property of the object), which gains advantages compared with baselines that do not require information about the category.

[1] Unsupervised learning for cuboid shape abstraction via joint segmentation from point clouds.

[2] Learning to Infer and Execute 3D Shape Programs.
questions: questions: 1. Could the authors please provide a more detailed description of how to build an abstract 3D model regarding my major concern? I would consider raising or lowering the score based on the way of constructing a 3D model from 2D images.

2. The paper mostly describes and showcases the editing of opening or increasing (e.g., opening door and laptop). I am wondering about the process of 'decreasing' since the design of dynamic feature maps needs to be changed accordingly (e.g., ${M_n}^{Gen}$ should be all zeros, or should it refer to novel backgrounds that are previously occluded by the objects).

3. In line 228, the paper shows that this method could be applied to more objects with more manipulation types. Could the authors provide more description about how this can be done? Do they also require abstract 3D models and manipulation in Blender?

4. Minor: In Figure 4, why do the third column edited images share the same backgrounds while they are not the same as the original one?
limitations: limitations: The author described the limitations. Further discussion is better (Please see the Weakness section).

Comments: title: Replying to reviewer pkKj
replying to reviewer YyMw
Tend to accept
comment: Thank you for your reply. I would keep my rating based on the discussion.
Thank you for your reply and additional results.

Based on the reply, I think the current pipeline is able to solve "typical" objects and requires initial abstract model for each category, which limits the application on diverse objects and more categories.

Therefore, I tent to keep my score. I hope the author can improve the 3D abstraction approach in future work. Good luck.
Dear Reviewer,

Thanks for your questions, we answer them as follows.

**Q** : How could the prototype matching solve the ambiguity of 2D information? For example, we know the laptop in the Figure 3 has two parts becasue is a laptop. But why could the prototype matching system identity two planes while only one layer is shown in the image? Does it mean that you give the model the category information? Do the authors use pre-defined abstract model for laptop instead of matching for each image?

**A** : The 2D information is disentangled and interpreted with common structure knowledge for each object category. **Contrary to the single image to 3D model methods, the PA-Diffusion Model integrates the common structure information for each object category, and then the initial Abstract 3D Models are pre-built in alignment with the information.** For example, the initial Abstract 3D Model of the laptop consists of two planes since a laptop typically consists of a screen and a keyboard, similarly, the initial Abstract 3D Model for a typical microwave includes a body and a door. When editing images, these initial Abstract 3D Models are adjusted to align with each input image. Section 3.2 in the main paper discussed this process in detail. Owing to the straightforward process, the PA-Diffusion Model incorporates novel categories efficiently by extending the structure knowledge and initial Abstract 3D Models.

Yes, the PA-Diffusion Model identifies the object category with the text instructions or by leveraging other large-scale models. Please refer to the link provided, Figure 1 illustrates the process of constructing an Abstract 3D model.  https://drive.google.com/file/d/1W2PRIxt65wxK5NkMqKBOVF2IMgh54XTt/view?usp=sharing

(1)	Using Grounded SAM, we have the semantic segmentation masks of the laptop, Figure 1 (b). 

(2)	The initial Abstract 3D Model of the laptop is made up of two 3D planes, one represents the screen and the other represents the body, Figure 1 (c).

(3)	We align the camera view of the Abstract 3D Model and the input image, Figure 1 (d) (e). 

(4)	Manipulation can be performed within the 3D space, Figure 1 (f).

**Q** : I argee with the authors about iterative editing process could probably handle complex objects since the use of diffusion model in this paper is well-designed and reasonable. But I doubt that the 3D abstract model could not be easily obtained for complex objects. Requiring additional URFD files will be a huge drawback of this approach since such data (real image paired with URDF) is rare.

**A** : Thanks for your inspective comments. When constructing a 3D model for an entire complex object is necessary, additional spatial geometrical information is required. Utilizing URDF is one of the possible solutions. With advanced large-scale models such as SAM, SAM2, DINO, and so on, there is potential to extract the necessary spatial geometrical information efficiently and economically. In this work, the PA-Diffusion model is specifically designed to support robotic tasks, therefore we mainly focus on everyday articulated objects. Please refer to the attached PDF in the global rebuttal for an example illustrating how the PA-Diffusion model facilitates the generation of sequential robotic manipulation data. 

Your question highlights an important aspect that warrants further attention. This is a meaningful point, especially for some downstream applications. We are actively working to incorporate this to further enhance our work.

**Q** : Could the authors add an anonymous link to show some results on the prototype matching (image with constrcted 3D abstract model)?

**A** : Please check the link. Thanks.

https://drive.google.com/file/d/1W2PRIxt65wxK5NkMqKBOVF2IMgh54XTt/view?usp=sharing

If there exists any other questions or confusion, please inform us at any time you feel convenient.
Dear Reviewer,

Thank you for your questions and comments. Please check the following answers. 

**Q** : However, I still have some problems given the reply on the 3D modelling from image. If I understand the reply correctly, these 4 steps I asked in the question is done heuristically.

1. The object is composed of 2 parts, especially the base of the laptop is almost invisible (number of parts discovery)

2. These two parts fit with the plane prototypes (part to prototype fitting)

3. These two planes are connected (part-level relation discovery)

4. One of the plane can be rotated along the connected axis (joint type estimation between revolute and prismatic)
And the "modelling" task done in this paper is part-level segmentation and pose fitting.

If that's the case, I would agree with the reviewer pkKj. A 2D to 3D understanding for articulated object is the key for manipulation. I would suggest the author to reframe a bit on the problem they are targeting, perhaps try to avoid the "3D modelling" task and focus more on image synthesis in the pipeline and writing.

**A** : Your understanding is accurate. The structural information for commonly encountered object categories is pre-collected based on established knowledge. Following the four steps you mentioned, Abstract 3D Models integrate this information to facilitate manipulation and editing tasks. Due to the simple process, the PA-Diffusion model can be extended to incorporate additional structural information and novel objects with minimal complexity. 

The transition from 2D to 3D modeling is one of the primary contributions of this work, which benefits both the image editing task and the downstream tasks in robot scenarios. Compared to other state-of-the-art editing methods, the PA-Diffusion model demonstrates superior capability in object editing, largely due to the support provided by Abstract 3D Models.

Additionally, utilizing the Abstract 3D Model offers several advantages including improved accuracy, efficiency, ease of manipulation, the ability to handle multiple categories, and incorporating novel categories quickly. 
Beyond image editing, these advantages suggest that the PA-Diffusion Model holds promise in reducing the reliance on expensive robotic manipulation data. Please check the attached PDF file in the global rebuttal, there is an example of how to generate sequence robot manipulation data with the PA-Diffusion Model. 

We appreciate your suggestion and will take your advice to focus more on image synthesis in the paper.
Thank you for your detailed reply. It address some of my concern.

However, I still have some questions given the reply on the 3D modelling from image. If I understand the reply correctly, these 4 steps I asked in the question is done heuristically. 

```
1. The object is composed of 2 parts, especially the base of the laptop is almost invisible (number of parts discovery)
2. These two parts fit with the plane prototypes (part to prototype fitting)
3. These two planes are connected (part-level relation discovery)
4. One of the plane can be rotated along the connected axis (joint type estimation between revolute and prismatic)
```
And the "modelling" task done in this paper is part-level segmentation and pose fitting. 

If that's the case, I would agree with the reviewer pkKj. A 2D to 3D understanding for articulated object is the key for manipulation. I would suggest the author to reframe a bit on the problem they are targeting, perhaps try to avoid the "3D modelling" task and focus more on image synthesis in the pipeline and writing. 

I will keep my rating based on the above concern.
The author's response addresses my concerns comprehensively. I believe this is a strong paper that merits acceptance at NeurIPS. Good luck!
Thanks for you well-written reply. It addresses some of my concerns.

However, I argue that the construction of the abstract 3D models from 2D images is the most difficult yet important part in your pipeline since 2D-to-3D is definitely not trivial and 3D models contain lots of priors (joint type, joint position, geometry) about the objects for manipulation. From my perspective, having the 3D structure of objects, which is even interactable, usually means the problem is half-solved in robotics. Whereas the approach described in the paper (segmentation module with prototype corner matching) seems too simple to achieve this important and difficult function. Therefore, I still have some concerns about this process.

1. How could the prototype matching solve the ambiguity of 2D information? For example, we know the laptop in the Figure 3 has two parts becasue is a laptop. But why could the prototype matching system identity two planes while only one layer is shown in the image? Does it mean that you give the model the category information? Do the authors  use pre-defined abstract model for laptop instead of matching for each image?

2. I argee with the authors about iterative editing process could probably handle complex objects since the use of diffusion model in this paper is well-designed and reasonable. But I doubt that the 3D abstract model could not be easily obtained for complex objects. Requiring additional URFD files will be a huge drawback of this approach since such data (real image paired with URDF) is rare. 

3. Could the authors add an anonymous link to show some results on the prototype matching (image with constrcted 3D abstract model)? 

Thank you for your rebuttal. Currently I will keep my score due to the concerns above.