General Articulated Objects Manipulation in Real Images via Part-Aware Diffusion Process
Authors: Zhou FANG
Yong-Lu Li
Lixin Yang
Cewu Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: Articulated Objects
Arbitrary Manipulation
Real Images
Primary area: diffusion_based_models

Abstract: Articulated object manipulation in real images is a fundamental step in computer and robotic vision tasks. Recently, several image editing methods based on diffusion models have been proposed to manipulate articulated objects according to text prompts. However, these methods often generate weird artifacts or even fail in real images. To this end, we introduce the Part-Aware Diffusion Model to approach the manipulation of articulated objects in real images. First, we develop Abstract 3D Models to represent and manipulate articulated objects efficiently. Then we propose dynamic feature maps to transfer the appearance of objects from input images to edited ones, meanwhile generating the novel-appearing parts reasonably. Extensive experiments are provided to illustrate the advanced manipulation capabilities of our method concerning state-of-the-art editing works. Additionally, we verify our method on 3D articulated object understanding for
embodied robot scenarios and the promising results prove that our method supports this task strongly. The project page is https://mvig-rhos.com/pa_diffusion.

General Articulated Objects Manipulation in Real Images via Part-Aware Diffusion Process
Zhou Fang
Yong-Lu Li
Lixin Yang
Cewu Lu"
Shanghai Jiao Tong University {joefang, yonglu_li, siriusyang, Jucemu)@sjtu.edus
Abstract
Articulated object manipulation in real images is fundamental step computer and robotic vision tasks. Recently, several image editing methods based on diffu- sion models have been proposed to manipulate articulated objects according to text prompts. However, these methods often generate weird artifacts or even fail in real images. To this end, we introduce the Part-Aware Diffusion Model approach the manipulation of articulated objects in real images. First, we develop Abstract 3D Models to represent and manipulate articulated objects efficiently. Then we propose dynamic feature maps to transfer the appearance of objects from input images edited ones, meanwhile generating the novel-appearing parts reasonably. Extensive experiments are provided illustrate the advanced manipulation capabilities of our method concerning state-of saate-of-the-ar editing works. Additionally, we verify our method on 3D articulated object understanding for embodied robot scenarios and the promising results prove that our method supports this task strongly. The project page is at https://tmm //mig rhos com/pa_di com/pa_diffuse diffusion.
1 Introduction
Image editing is long-standing popular computer vision task. Specifically, manipulating articulated objects has garnered significant attention owing its application in various fields, such as image augmentation for downstream tasks [44], building goal conditions to train reinforcement learning models for robotic manipulation [42, 55], creating videos with extra supervision information [34], detecting human-object interactions [22, 23, 26), reasoning object affordance [25, 24], etc. Thanks to the large-scale training data and immense computing power, diffusion-hasee [41] generative models have achieved surprising results in the field of image and video generation.
Inspired by these successes, several recent works have adopted diffusion models as the backbone and implemented text- text-guided object manipulation [20, 16, 8, 51]. We can properly divide these studies into couple of groups. The first one is to directly edit 2D images by transferring the feature/attention maps from original images to edited ones such as [13, 32, 8]. However, weird artifacts are prone rotated and deformed, novel views appear. Consequently, these
to appear when the objects methods are restricted to structure- pereeuurepreeeriingg. image editing. Another group relies on reconstructing 3D object models. As the most related work to ours, [51] reconstructed 3D object models for manipulation and projected them back to images later. Nevertheless, this approach depends on the quality of reconstructed 3DD models heavily. And the reconstruction model has to be fine-tuned when dealing with new categories. Moreover, manipulation has to be done manually which laborious and impractical to support editing large quantities of images.
To address these problems, we propose the Part-Aware Diffusion Model (PA-Diffusion model) for articulated object manipulation in real images, as illustrated in Fig. 1. Firstly, we introduce the
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Text Instruction
'a photo of microwave with door open, 60"
Noise Maps Masks Skotch Maps
3D
Generation
Part-aware Diffusion
Input Real Image
Normalized 3D Space
Edited Image
Figure I: We propose the Part-Aware Diffusion Model: Abstract 3D model of the articulated object
constructed referring to the input 2D real image. Various manipulations can he done based on the text instruction or human interaction in the 3D space, the generation model then creates the edited image according to the manipulation.
concept Abstract 3D Model and build a Primitive Prototype Library to represent articulated objects in the 3D space, that method can not only cover many common objects but also handle novel categories without extra training data or fine-tur tuning processes. Besides, various manipulation can be done efficiently. Second, proposed dynamic feature maps to assist generation models in accurately transferring object appearances to accurate locations edited images. As a result, weird artifacts eliminated, and meanwhile, the novel-appearing parts generated more reasonably. Finally, owing to the simple manipulation and editing process, the procedure is automatic and the model can strongly support other tasks by editing large volume of images.
Our main contributions are summarized as follows:
(1) We introduce the concept of the Abstract 3D Model which accurately and robustly represents various articulated object categories with primitive prototypes. Meanwhile, novel categories can also be incorporated quickly. In addition, the articulated objects can be efficiently manipulated with text instructions human interactions in the 3D space.
(2) We propose dynamic feature maps that let the diffusion model comprehend the object structure. Consequently, the diffusion model generate novel-appearing parts of objects reasonably and preserve the appearance the seen parts simultaneously.
(3) We present comprehensive experiments to highlight the advantages of our PA-Diffusice model including comparing with state-of-the-art editing methods both qualitatively and quantitatively, choosing 3D articulated object understanding experiment to demonstrate how method supports the tasks embodied robot scenarios.
2 Related Work
2.1 Diffusion Model for Image Generation
recent years, diffusion models [40, 39, 11] have achieved great success in the fields of image/video generation [7, 17], segmentation [5, 50), and many downstream computer vision tasks. To make the generation results controllable, [35] first proposed to extract and incorporate text features into the denoising process. Following this concept, [41, 12, 43, 15, 9] improved the performance of text-guided diffusion models with more effective text embedding methods.
However, as an implicit instruction. text guidance is still not strong enough to finish fine-grained image control such as determining the image layouts, objects' shape and texture, and so on. To make up this gap, [47] provided structural guidance by enhancing the similarity between the features of other conditions and the text guidance. [14, 6] proposed modify the cross-attention maps and then guide the denoising process. To handle more complex scenarios and achieve more precise control, [52] and [33] proposed adding an extra module the diffusion model. Then extra condition information can be imported to guide the denoising process.

---

2.2 Diffusion Based Image Editing
Considering the remarkable capability of understanding images, several recent works have also reported editing real and synthetic images with using diffusion models as the backbone. These methods can generally be summarized into two groups: Inversion- -asee and Feature-Sharing Based.
The first group is primarily based adding extra control to the inverted noise maps of images, then re- generating the image such as [20]. However, because the deterministic DDIM sampling process cannot be reversed perfectly. these methods struggle preserve the appearance of original objects and backgrounds precisely. The second group attempts to maintain the appearance of objects by transferring the feature/attentiooncctiailll
maps between guidance and generation branches or by adding extra loss items during the denoising process, as seen in [8, 32, 13]. Recent approaches like DragGAN [36] and DragDiffusion [32] propose utilize point-to-point dragging scheme, which can achieve refined content dragging. Nonetheless, these approaches often perform poorly on articulated object manipulation in real images, resulting in weird and blurry artifacts in edited images.
2D-3D-2D is another promising way of image editing, the recent work [51] introduced reconstructing 3D models from 2D images and projecting them back after manipulation. However, this approach highly relies on the quality of 3D reconstructed models, and reconstructing 3D models from a single 2D image is still challenging task.
In contrast to the aforementioned approaches, our method demonstrates advantages when manipulat- ing articulated objects in real images high fidelity edited images, easy and various manipulations, covering multiple categories, and incorporating novel categories quickly.
3 Method
3.1
Overview
In this session, we go through the proposed PA-Diffusion model in detail. The overall architecture is demonstrated in Fig. 2. Initially, we reconstruct abstract 3D models for articulated objects with the Primitive Prototype Library. Then various manipulations can be done according to text instructions or human interactions. Next, leveraging DDIM Inversion [45, 31], initial inverted noise maps are created and manipulated following the previous actions. During the generation stage, we introduce dynamic feature maps, including manipulated inverted noise maps and compositional activation maps and images. These ensure that the appearance of seen parts of objects can be preserved accurately and that novel-appearing parts are generated reasonably. Besides, Texture and Style Consistency Score Loss are introduced to alleviate the blurry and style mismatch problems.
3.2 Preliminary
Diffusion models aim to convert random Gaussian noise into high- nighuresolution images through a sequential denoising and sampling process [13]. Given the conditioning y. we start from the initial Gaussian noise map 2t, and then iteratively estimate the reduced noise é, at each time step t:
& = c t, v),
(1)
2--1 update & t,, 1,
The update function could be DDPM [18], DDIM [45], or other sampling methods. Nevertheless, conventional sampling from conditional diffusion models often fails to produce high-quality images that align well with the condition y. To enhance the effect of the desired condition, extra class loss guidance is added to the reduced noise during the sampling process such as Classifier or Classifier-free guidance [46, 19].
Classifier guidance is introduced to generate conditional samples from an unconditional model by combining the unconditional score 6, with classifier p(y 22), where p(yl2;) is the probability distribution of condition y based on the noise at time step t:
é = co((2t))))) + 3 Z p(y))))
(2)
Classifier-free guidance eliminates the need for separate classifier by incorporating the class information directly into the generative model as follows:
& = (1 + a)eo(E(;tt)) (z;(t)))
(3)
3

---

Manipulation
Text Instruction: photo of opened laptop, 60", 120"....
Obj
Manipulation
Value
O
[T(TTL
Denaising U-Net $
Anverdio
Necook
Inverted Noise Map Batch
Input Image
x
Primitive Prototype Library
Cube
Body1
Body2
Body3
Manipulated Part-level Masks
Edited Images
Sketch Map
The overall image editing process. (1) In the Pre-Process stage, articulated objects in 2D images are part-level segmented and reconstructed to abstract 3D models. Meanwhile, inverted noise maps of input images created with DDIM Inversion. (2) In the Manipulation stage, various manipulations can be implemented based on text instructions or human interaction in the 3D space. (3) After manipulation, part-level masks and sketches are rendered and exported. The inverted noise maps are transformed according to these masks. (4) Finally, with the transformer inverted noise maps, sketch maps, and part-level masks, the generation model creates the edited images.
Figure 2:
Following these concepts, custom energy functions can also be utilized to guide the denoising process, instead of the probability function. In [13] [30] [54], various energy functions g are incorporated alongside classifier-free guidance to obtain high-fidelity samples as follows:
é (1 + (2) y) (2;; t) + 3 2. g(2;;t,))
(4)
Our proposed PA-Diffusion model is built on the diffusion model with classifier-free guidance. Extra energy functions are employed during the image editing process.
3.3 Various Manipulations in the 3D Space
As promising workaround to the methods of dealing with images directly, the 2D-3D-2D pipeline has successfully handled many articulated objects with precise 3D models. Unfortunately, creating 3D models for various categories from single image remains challenging, particularly for novel categories or instances. In this work, we introduce the concept of Abstract 3D Model that reconstructs accurate 3D models, supports efficient object manipulation, and incorporates novel objects easily.
Abstract Model. Unlike previous methods, there is no need precise 3D models of our method, the conditional information we have provide to the diffusion model is coarse sketch maps and part-level masks. Therefore, we introduce the use of an abstract 3D model to represent the articulated object. As abstract 3D model, the object is represented by combining several basic prototypes. As depicted at the bottom of Fig. 2, the laptop can be represented by two planes, storage furnitures and microwaves which includes basic 3D prototypes such as planes, cubes, and boxes, can support common articulated object categories involving both rotation and translation joint types.
be represented by plane and box. Primitive Prototype Library,
Camera alignment. Next, we compute the camera pose the 3D space and align image view with the 3D space camera view. The pose computation problem is to calculate the intrinsic and extrinsic matrices for the camera that mínimize the reprojection error from 3D-2D point correspondences [3]. Thus, in this work, we first employ Large-scale Segmentation Models obtain the initial part-level segmentation masks of articulated objects Init and then detect the extreme corner points A, B, C, D.D (pts;) with simple corner detection functions. These 2D extreme points (pts2, pre-defined in Primitive Prototype Library)
real
aligned with their 3D counterparts A' BB, C' B',C', as shown in Fig. 3. Finally, based on Perspective n-Points 2D-3D method [49], the camera matrices can be extracted, and 2D-3D views aligned.
Manipulation. By representing objects with primitive prototypes, multiple types of manipulations be implemented in the 3D space efficiently with the assistance of 3D computer graphics software.

---

Marks fram Peasee
Part1
A
Noñsa Map zirt
Inttal Inverted
Modal
Abstrract
Manipulated
level Masks M
Inttial Masks M
Transform Functions TRA... D). CA D)
Guidance
Random Noise 2f
Novel
Random
Map 4
Mast NO
Generation
Image
Transformed Inverted Noise Map 2p
Manipulated Invorted Map zir
Ciftusion Modet
with noise batch
Act. Map A A-M
&
Keep Seen
Figure 3: Algorithm pipeline of the PA-Diffusion model. Symbols and procedures in the figure the same as those in the content.
As shown in Fig. 2, manipulating objects through text instructions is a straightforward approach. For example, the instruction opening the laptop 120" is converted to script, then the manipulation will be done by running the script in 3D software, Our PA-Diffusion model also supports human interaction, which could be even more efficient. Additional manipulation guidance is provided in the Appendix. Contrary to the tedious manipulation experience of previous SOTA works, our proposed PA-Diffusion model offers more flexible approach editing articulated objects.
Structure disentangle. Some object parts could still be seen after manipulation, and some unseen parts would appear. As shown in the top right part in Fig. 3, the laptop shell be seen in the input image. After opening the laptop, the shell can still be seen, however, the keyboard is newly revealed. Therefore, distinguish them and implement part-aware diffusion, disentangle articulated objects into seen parts and novel-appearing parts. The appearance of seen parts should be consistent between input and edited images, and the style of novel-appearing parts should be consistent with the objects' overall appearance. In this work, we regard the initial part-level masks M /nit as seen. Then after manipulation, export manipulated part-level masks from the 3D software. And then manipulated seen parts mask M and manipulated novel-an wwel-appearing parts mask M can be calculated.
3.4 Dynamic Feature Maps
To maintain the object's appearance including color and texture, previous editing methods introduced a guidance branch invert and re-generate the input image, and generation branch to create edited image, the attentionnfeeatureacciidti
maps are transferred from the guidance the generation branch directly [8]. However, when changing the object location or shape during manipulation, directly sharing these maps would transfer the feature from the input image to undesired locations in the edited image. Furthermore, these methods cannot reasonably generate novel views or novel- appearing parts.
To overcome these problems, we propose dynamic feature maps including manipulated inverted noise maps and compositional activation maps and images. To keep appearance accurate, manipulated inverted noise maps transfer the feature seen parts in input images to the manipulated seen parts in edited images. Simultaneously, to make novel-appearing parts reasonably, compositional images let diffusion model create these parts from random noise. The following content describes how to manipulate the noise maps, how to construct compositional maps, and how they work. For clarity, the process is presented in Fig. 3.
Manipulated inverted noise map. As shown at the top of Fig. 3, we firstly reverse the input image the initial inverted noise map 2 with DDIM inversion. After 3D manipulation, we calculate the transform function based the initial I Iist and manipulated seen part-level masks M Gen and then compute the transformed inverted noise map z with T. The manipulated inverted noise map is created as the following equation:
2/
Man
2T
Tran M.Gen + Tran M Make + afii Mbg.
(5)
5

---

where M, is the makeup mask generated by XOR(M M n M, Mog is the background mask created by 1 - --Meen In addition, we also create a random noise map of The three noise maps z/rat, zzfan, and =/ will be sent to the denoising UNet in the next step.
Compositional activation map and image. As shown the bottom of Fig.3, the diffusion model sequential denoising process with the three noise maps batch. The activation map A and be for seen parts and background are generated separately with the supervision of the guidance branch, and then merged to build A according to the seen part mask M After the decoder, the initial edited images /Gen are created. Owing to the above steps, we transfer the feature of seen parts in input images to the accurate location in edited images. The other contents and the background can also be preserved, as the highlighted yellow part Fig. 3. Besides, as the highlighted blue part in Fig. 3, extra image I Raa is synthesized with random noise map 2ftan, the navel-app parts are cropped and pasted to edited images IGen as follows, which makes these parts more reasonable and consistent with the original inputs.
yGen RRan G + GGen (1 - (1-Me MCen),
3.5 Score Function
Texture Consistency Score Loss. However, simply manipulating the inverted noise map will lead to serious blurry problem. This is due to the denoising process includes several convolution steps. As the initial inverted noise map =/ is not rotation invariant, manipulating zin will disturb the original distribution and make the denoising process fail. To alleviate this limitation, we construct Texture Consistency Score Loss (TCSL) [32] as an supervision that lets the specific region in the generation branch match with the one in the guidance branch,
Loss
cos (oS(F Mmitt] F [MGen)
bg Loss
cos( - Mnntt) GG MGen])
where GGut and FGen are feature maps of the guidance and generation branch. We add these items as extra losses in classifier guidance each denoising iteration step to calibrate the appearance of objects and background.
Style Consistency Score Loss. For novel-appearing parts, the diffusion model is prone to randomly select style to generate them with text guidance or sketch maps. As result, the texture and style could be different from the objects in input images. Therefore, we introduce Style Consistency Score Loss (SCSL) to calibrate the style of seen parts and the novel-appearing parts.
Different from Texture Consistency Score Loss, there is need match every pixel in input images and edited images. Thus we calculate L1 loss between the feature maps R.Ran of the random branch and pGui [13]. The loss function is follows:
Loss, = (M
sum (a.Ran (A (e
(8)
Briefly, the reduced noise é, follows, where y,, 72, and 7/ are the hyper-parametert
_en__g & eGen_bg and ettan in each denoising iteration can be presented as
& Gen & &Gents + 71 =., Loss!9 x MGen
Gen_by = él
Genby + Vz, Loss bg x (1 (1-M M
Ran Ran + 73 V. Loss, x M
&
4 Experiment
In this section, provide two kinds of experiments to prove the advantages of our proposed PA- Diffusion model. First, various image editing tasks are conducted to showcase the model's image editing capabilities. To highlight the superiority of our model compared with state-of-the-art methods, we collect a testbench and evaluate the methods both qualitatively and quantitatively. Second, we create synthetic training set to support the challenging 3D articulated object understanding task in the robotic scenarios.

---

Original
Original
Move
Statel
Scale/Shear
State2
Rotate
State3
Manipulate
State4
Figure 4: Results of basic manipulations: move, scale/shear. rotate, and manipulate. Blank regions caused by the manipulation are in- t--painted automatically. The novel views and novel-appearing parts match with the style of the seen parts (left). Articulated objects are opened from the initial close state with 4 steps. The appearance of the increasing novel-appearing parts keeps being consistent throughout the whole process (right).
4.1 Implementation
In this work, we select Grounded Segment Anything [21, 29] to obtain the initial part-level object segmentation masks. T2I Adapter [33] chosen as the conditional generation model, and the condition used is the sketch map. The fundamental diffusion model is Stable Diffusion V1-5. All experiments run on a single NVIDIA A100 GPU. Notably, NO models need to be trained or fine-tuned in the image editing process.
Primitive Prototype Library is built within Blender [10]. 3D planes, cubes, boxes, and other 3D primitive shapes are created and combined to represent different objects. In this work, 5 primitive shapes are collected and combined represent categories of articulated objects. The ease of creating prototypes allows for the quick incorporation of novel categories instances. Various manipulations can be implemented Blender efficiently.
4.2 Results
Fig. 4 demonstrates the editing results of some basic manipulations and sequential manipulation process. As shown in the left part, our PA- PA-Diffusion model naturally moves, scales/shears, rotates and opens articulated objects with rotation or translation joint types. The edited objects blend seamlessly with other contents and backgrounds in the original images. When we move or rotate the objects, the blank regions in the background are in-painted semantically according to the surroundings. Moreover, in-painting and editing are completed in a single denoising process by the PA-Diffusion model, no extra in mn-paint model or process is required. Last but not least, novel-appearing parts of objects generated reasonably, and the style matches with the object. For example, the storage furniture is empty after opening, and the keyboard of the laptop is designed logically.
The right part of Fig. presents complete operation process of opening articulated objects, from the initial closed state to the open state with 4 steps progressively. The appearance of objects' seen parts transferred from the original input to different states accurately. The point that needs to be mentioned is that along with the operation progress, more novel-appearing parts of objects appear, our proposed PA-Diffusion model keeps the style and texture of these novel-appearing parts being consistent throughout the process. This capability potentially allows method to generate
7

---

Mangukkteed
Manppdatte
Maripuizeed
Figure 5: Manipulate non-rigid objects, non-uniform shapes, and objects with weird or multiple joint types.
complete manipulation video from single input image, maintaining consistent object appearance and style even as novel views or parts increase.
Generally, articulated objects' parts are rigid and connected with one of the typical joint types rotation and translation. However, we are surprised to notice that our PA-Diff PA-Diffusion model can also handle non-rigid objects with non-uniform shapes, weird joints, and manipulations. As illustrated in Fig. we first select toys as examples of non-rigid with non-uniform shapes, the tail of the shark is moved up together with other close parts similar to deformable objects. Then, we broke the cup in real image. Third, the kitchen pot and storage furniture are opened to illustrate the case of weird and multiple joint types within one object. No matter whether the shape of object parts has changed after manipulation or joint types are unconventional, the PA-Diffusion model can edit the objects successfully. Meanwhile, the background is preserved or inpainted well.
4.3 Ablation Study
As mentioned in Section 3. TCSL is added to the denoising loss function to release the serious blurry problem. As shown in the left of Fig. 6, we move the storage furniture, the microwave, and the drawer to different directions. It can be seen that without TCSL, the objects are prone be blurry. More seriously, the edited image could be destroyed, as in the example of the drawer. On the other hand, with TCSL, the texture of objects be transferred to the desired location, meanwhile, the blank region caused by the movement is in-painted well. SCSL is another loss to keep the style consistent between seen parts and novel-appearing parts. Its effectiveness is shown in the left bottom of Fig. (with SCSL). We notice that when opening the storage furniture, the style of the novel-appeared inner door and body part more likely to be consistent with the outer body part with SCSL, which makes the edited image natural. More quantitative ablation studies about the two score losses are provided in the Appendix.
4.4 Comparison
To further evaluate our PA- PA-Diffusion model, we compare it with four state-of-the-art image editing approaches that are based on diffusion models: Imagic, DragDiffusion, MasaCtrl (with T2I Adapter), and Image Sculpting. In this experiment, we require these models to manipulate different categories of articulated objects, including both rotation and translation joint types. The results are shown in the right part of Fig. 6. hard for Imagic to finish the tasks as articulated objects cannot opened at all the wrong part is manipulated. This is because the text instruction is too weak and the fundamental generation model cannot understand the structure of objects. Similarly, DragDiffusion cannot finish the tasks even though human interaction is applied.
MasaCtrl performs better than Imagic and DragDiffussin. The manipulation can be finished, while the edited images are prone to be either unrealistic or unreasonable. Take the laptop an example (second column in the right part of Fig. 6), the object has been opened, while the region highlighted with the red bounding box in the edited image remains unchanged, which does not make sense. This issue is prevalent across other categories as well. The reason is that MasaCtrl simply shares whole feature/attention maps between the input and edited image, features of seen parts cannot be transferred to the desired new location when the objects move or the shape changes. Finally, Image Sculpting works well the storage furniture. However is likely to fail to reconstruct precise 3D models of the laptop, trashcan, and drawer, consequently, the edited images are undesirable.
In contrast, our PA-Diffusion model consistently produces high- fidelityyy and reasonable edits. The appearance of seen parts kept accurate no matter whether objects move or the shapes parts have changed. The novel parts are reasonable and semantically consistent with the original.

---

Original
Original
Move Object
Imagic
No TCSL
DragDiff.
With TCSL
MasaCtrl
manipulate Objectt
Img. Sculpt
With SCSL No SCSL
Ours
Figure 6: TCSL and SCSL are employed to release the blurry and style mismatch problem. The model is required move and open the object (left). Comparison Imagic, DragDiffusion, MasaCul (with T2I adapter), Image Sculpting, and our PA-Diffusion model. The target state is 'a photo of an opened object' (right).
4.5 Quantitative Evaluation
To quantitatively evaluate our method, we built an articulated object manipulation testbench. The testbench comprises 6 object categories including storage furniture, laptop, trashcan, microwave, drawer, and refrigerator, which covers both rotation and translation joint types. total, 660 real images collected from the website. Considering articulated objects are typically rigid with uniform shapes, this testbench can represent the characteristics of common articulated object categories.
The comparison methods we select are Imagic and MasaCtrl (with T21 Adapter). DragDiffusion is excluded here, as it simply reconstructs the original image and cannot complete the manipulation tasks. Due to the long manual processing time and frequent failures in generating 3D models, Image Sculpting also excluded here. To assess the realism of the edited images, the evaluation metric used the Frechet Inception Distance (FID) score. The quantitative evaluation results are summarized in Tab. 1.
Since Imagic relies solely on text instructions, the edited images often do not align well with the original inputs, resulting poor scores. Due previously discussed reasons, the edited images of MasaCtrl are confusing and lack coherence. Sequentially, the FID score is not satisfying. In comparison, the PA-Diffusion model outperforms other methods with over 40. % improvement.
4.6 Articulated Object Understanding
In this session, we demonstrate how our proposed method supports the task articulated object understanding, As one of the fundamental steps to understanding 3D articulated objects, estimating axes and surface normal is still challenging because the lack of data. To release the data limitation, we create synthetic dataset with the PA-Diffusion model. The dataset includes 660 sequential samples, each sample includes the sequence of opening objects from the close state with steps. [38] introduced 3-step training process to develop the object understanding model including

---

Category
Imagic
MasaCtrl
Ours
Category
bbox
bbux+axis 1
normal < 30"
Storage.
5.85
2.27
0.81
Storage.
67.5/65.0
65,0/65.0
59.9/61.5
Laptop
9.48
1.17
1.94
Laptop
87.5/90.0
64.2/75.2
22.8/35.7
Microwave
1.34
3.15
0.87
Microwave
80.0/85.0
80.0/85.0
72.4/70.7
Trashcan
5.62
1.59
0,96
Trashcan
95.0/92.5
74.4/84.9
40,5/49.2
Refrigerator
7.75
1.76
1.38
Refrigerator
70.0/80.0
70.0/80.0
63.2/80.0
Drawer
30.7
0.98
0.60
Drawer
70.0/82.5
70.0/78.8
67.5/82.5
Avg.
10.1
1.82
1.09
Avg.
78.3/82.5
70.6/78.2
54.1/63.2
Table Score edited images with Imagic,
Table 2: Prediction accuracy
model developed with
MasaCtrl (with T21 adapter) and ours.
half (left) and full (right) training set separately.
baxxssxis(rot normal
bbox-axisstraa) normal
Datasct
AUROC
bbox t
bbox+axis(rot)
bbox f
bbox+axis(tran)
Internet Video
74.0
62.1
28.5
16.5
32.0
26.2
14.3
Mixed
74.7
66.1
29.2
18.3
38.1
30.0
19.3
Table 3: Mix the edited images with the training set of the Internet Video dataset, then evaluate the fine-tuned
model on the testing set of the InternetVideo dataset.
Figure 7: Detection results on the testing samples, including rotation and translation joint types.
BBox detection, axis prediction, and plane normal estimation. Following this schedule, evaluate the feasibility of edited images by two kinds of experiments.
First, the generated sequential samples are divided into training/testing sets (612/48). Specifically, we follow the 3-step to develop the model with half and full training samples separately, and then evaluate the model with three matrices, BBox IoU, Axes EA-score, and surface normal error smaller than 30 [38]. Fig. 7 demonstrates the prediction results, the model can understand the structures of articulated objects after training with edited images, including moving plane, joint types, axis and surface normal. Quantitative evaluation results in Tab. 2 indicate that the prediction accuracy improves with more training samples, illustrating that the edited images comparable the real ones.
Second, further evaluate the edited images, merge them with the original training set of Internet Video dataset [38] and fine-tune the pre-trained model. The fine-tuned model is evaluated on testing set (6,231 real images) of the InternetVideo Dataset. Baseline refers to the model trained on the Internet dataset only. Here, we select to use the same evaluation matrices as [38], surface normal accuracy is multiplied with the accuracy of BBox and axis, other evaluation metrics the same as above. The evaluation results are summarized in Tab. Compared with the baseline, the overall performance has been improved by enlarging the training set with edited images. The above two experiments illustrate how our PA-Diffusion model can benefit robotic vision tasks.
5 Limitations
Even though our method can handle common articulated objects, there are still some limitations. First, edited images are generated from inverted noise maps, the quality of the original input images significantly affects the editing outcomes. Blurry or low-resolution inputs will degrade the edited images. Second, when the object undergoes substantial deformation movement, this editing method is likely to fail. Besides, manipulating deformable objects and fluids remains challenging with this approach. Further explanation is provided in the Appendix.
Conclusion
This work introduces the PA-Diffusion model, novel articulated object manipulation method that covers common object categories and supports various manipulations. Both the qualitative and quantitative experiments have proven the feasibility and effectiveness of our method. Besides, the 3D articulated object understanding experiment illustrates that the PA-Diffusion model has positive impacts on helping build robots that interact with the real world smartly.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China under Grants 62306175, the National Key R&D Program of China (No.202IZD01 10704), CCF-Tencent Rhino-Bird Open Research Fund, the National Key Research, Development Project of China (No. 2022ZD0160102),, the National Key Research and Development Project of China (No.202 100212001 10704), Shanghai Artificial Intelligence Laboratory, XPLORER PRIZE grants, STCSM 2023 Pujiang X Program Project (No.23511103144 1031044) and STCSM 2024 Qimingxing- -aaggfffng-aagffann Fund (No.24YF2722000).
References
[1] Opency find contour. contours_begin.Nee html.
[2] Openev find corner wwwann--eeteeeeotttttee
https://ppinngeeaarrdccc///////p ooottt-iin.otttors---tthpppap...
[3] Opencv solve pnp. https://docss http:///......u......//////////t
[4] Pytorch grid sample. http:////gooraaeooooooo////tttat functional. grid_sample html.
[5] Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint (xXiv:2//2.00:300 2021.
[6] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miik Aittala, Timo Aila, Samuli Laine. Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.0132 2022.
[7] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In European conference on computer vision, pages 707-723. Springer, 2022.
[8] Mingdeng Cao. Xintao Wang, Zhongang Qi. Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304. 08465,, 2023.
[9] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.03377. 2023.
[10] Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 11:8780-8794, 2021.
[12] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image image generation via transformers. Advances in Neural Information Processing Systems, 34: 19822-19835.. 2021.
[13] Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski. Diffusion -ulfanui for controllable image generation. arXiv preprint 1XXi:223366009986 2023.
[14] Weixi Feng, Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint urXiv:2212.0500 2022.
[15) Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. European Confer- ence Computer Vision, pages 89-106. Springer, 2022.
[16] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promyy image editing with cross attention control, arXiv preprint arXiv:2208. 01626, 2022.

---

[17] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey Gritsenko, Diederik P Kingma, Ben Poole. Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint 1xXiv:2210.0230:3 2022.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:684006851, 2020.
[19] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 1 12598, 2022.
[20] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017 2023.
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02664 2023.
[22] Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, Zuoyu Qiu, Liang Xu, Yue Xu, Hao-Shu Fang, and Cewu Lu. Hake: knowledge engine foundation for human activity understanding. TPAMI, 2022.
[23] Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Shiyi Wang, Hao-Shu Fang, Ze Ma, Mingyang Chen, and Cewu Lu. Pastanet: Toward human activity knowledge engine. In CVPR, 2020.
[24] Yong-Lu Li, Yue Xu, Xiaohan Mao, and Cewu Lu. Symmetry and group in attribute-object compositions. In CVPR, 2020.
[25] Yong-Lu Li, Yue Xu, Xinyu Xu, Xiaohan Mao, Yuan Yao, Siqi Liu, and Cewu Lu. Beyond object recognition: A new benchmark towards object concept learning. In ICCV, 2023.
[26] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu. Transferable interactiveness knowledge for human-object interaction detection. In CVPR, 2019.
[27] Chen Liu, Kihwan Kim, Jinwei Gu, Yasutaka Furukawa, and Jan Kautz. Planerenn: 3d plane detection and reconstruction from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4450-4459, 2019.
[28] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[29] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303. 05499, 2023.
[30] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.00 01073, 2021.
[31] Ron Mokady. Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038 6047, 2023.
[32] Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.024221 2023.
[33] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint ar 200202230..08433, 2023.
12

---

[34] Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 18444-18435 2023.
[35] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 2021.
[36] Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image manifold. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
[37] Shengyi Qian and David F Fouhey. Understanding 3d object interaction from a single image. arXiv preprint urXiv:2305.0966 2023.
[38] Shengyi Qian, Linyi Jin, Chris Rockwell, Siyi Chen, and David F Fouhey. Understanding 3d object articulation in internet videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 1599-1600 2022.
[39] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 1(2):3, 2022.
[40] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pages 8821-8831, PMLR, 2021.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 10684-10695, 2022.
[42] Javier Romero, Dimitris Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, 36(6), 2017.
[43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494 2022.
[44] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1448, 2019.
[45] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:20100005502, 2020.
[46] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
[47] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided toxt-to-image diffusion models. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
[48] Wikipedia contributors. 3d projection, 2004. [Online; accessed 22-July-2004].
[49] Wikipedia contributors. Perspective-n-poine 2004. [Online: accessed 22-July-2004].
[50] Julia Wolleb, Robin Sandkühler, Florentin Bieder, Philippe Valmaggia, and Philippe C Cattin. Diffusson models for implicit image segmentation ensembles. In International Conference Medical Imaging with Deep Learning, pages 1336-1348. PMLR, 2022.
[51] Jiraphon Yenphraphai, Xichen Pan, Sainan Liu, Daniele Panozzo, and Saining Xie. Image sculpting: Precise object editing with 3d geometry control. arXiv preprint arXiv:2401.0 2024.
13

---

[52] Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to- image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[53] Kai Zhao, Qi Han, Chang- Bin Zhang, Jun Xu, and Ming-Ming Cheng. Deep hough transform for semantic line detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, (000)44444444466 2021.
[54] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. Advances in Neural Information Processing Systems, 35:3609-3623, 2022.
[55] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: 3d lasion-lagguag--an arXiv:2403.09633 00961,, 2024.
acttag action generative world model, arXiv preprint
14

---

Appendix
In this appendix session, we first go through the pipeline the PA-Diffusion model, and then provide more experiment results and detailed explanations, the arrangement is follows:
Sec. A: Additional algorithm pipeline of the PA-Diffusion model.
Sec. B: Additional articulated object manipulation results with the PA-Diffusion model.
Sec. C: Additional ablation study and analysis.
Sec. D: Additional explanation of 3D articulated object understanding experiment.
Sec. E: Limitations and future research.
Sec. F: Societal impacts and potential risks.
A Additional algorithm pipeline of the PA-Diffusion model
To facilitate the understanding of our proposed PA-Diffusion model, present the entire algorithm pipeline in Algorithm 1. eq.5, eq.7, and eq.8 refer to the equations in the main paper.
Algorithm 1 PA-Diffusion Model
Require: Manipulate the articulated objects in RGB images
Input:
RGB image x, Primitive Prototype Library
Output:
Edited RGB image Redit
Pre-Process:
1: Generate initial inverted noise map 2f with DDTM DDIMInverlan Invertion
2: Generate initial part-level masks I with GroundedS AM
3: Create 3D Abstract model and calibrate 2D image 3D camera view Manipulation:
4: Manipulate articulated objects with text instructions or human interaction in the 3D space
5: Calculate manipulated part-level masks MGen of seen part and M of novel-appearing part
Feature Process:
6: Calculated manipulated inverted noise map 2 Man as eq.5.
Generation:
7: Send initial Inét, random Maa and manipulated z inverted noise map to diffusion model
2T 8: for = =T. do
Add extra loss items TCSL Loss; and SCSL Loss, as eq.7 and eq.8
Create activation map AGen
end
9: Output: Edited image Zedit Decoder(zo)
Dynamic Feature Maps: To make it clear, we simplify the x 4 inverted noise maps and compositional images to pure color block maps, as shown in Fig. 8. Simple actions such as moving, scaling, and shearing can be implemented with grid sample[4] directly. However, for complex actions, perspective transform [48] is required. As the following equations show, we calculate the perspective transform function T Trans f form based on plai and pla3, where pls; and pl8z are corner points input image masks A, B, C, D (pls)) and corner points of manipulated masks A", B", C,"", D" (plsg) as introduced in the main paper. The corner points are automatically detected with a simple corner detection function. And then we get the transformed inverted noise map 27 _Tran by transforming the initial inverted noise map //mit as following equations. Finally, the manipulated inverted noise Tran and 2T [mit. map is calculated by adding 27
T = Trans form(pts ptss),
(10)
Tran z
T
On the other hand, the edited images are generated by adding Ron generated with the random noise map and / Geen generated with the manipulated inverted noise map, shown in the right part of Fig, 8.
15

---

x
532 x
512
532
Manipulated Partelevel Mands Myuer Milite
Manipullted Fart-level Macks Mini M, Misiio
Seen Part
hia Masks
Svanduurth
Novel. Part
Novel Part
IGG
BG
BG
BG
BG
intal Inverted
Transformed Inverted Noise Map 24
Manipulated Inverted
Generntion Imaage pro
Rondum Inouge Hun
Edittdd Imagee p
Nome Map ains
Nose Map ap
Figure 8: Manipulating noise maps and constructing compositional images. The manipulation is implemented with grid sample or perspective projection. The final edited image is constructed by merging two activation maps and images.
Text Instruction:
Open the laptop 30
Object
Action
Value
From sathutils, Lepert Voctar
Laptop Move
X
Laptop Open
date sojects (Gube") move the ebject
Vecterll4, 7,
sb.netatiomeeeee rediara()))
Mapping Table
Figure 9: Example of manipulating 3D objects in Blender with text instructions.
Articulated Object Manipulation in 3D space: As noted in the main paper, different manipulations can be done with text instructions or human interaction. For the text-based method, considering the objects can be manipulated with Python scripts in Blender, we first need to construct a table mapping the text instructions to actions. Then these actions can be implemented by running Python scripts. As shown in Fig. 9, require the laptop to open 30 degree, this text instruction is converted to Python script. Finally, running this script can finish the open action. For the second type, users can directly manipulate any parts of articulated objects in Blender, which could be more flexible and convenient.
B Additional Manipulation Results
In Fig. and Fig. 11, more articulated object manipulation results synthesized by our proposed PA- Diffusion model are demonstrated. Novel categories including door, toilet, and book experimented here. For various categories, joint types, and backgrounds, our proposed method can manipulate the objects and preserve other contents in the input images simultancously.
C Additional Ablation Study
To analyze and explain our proposed PA-Diffusion model detail, we provide more ablation studies in this session.
In the main paper, we qualitatively demonstrate the effect of TCSL and SCSL in the experiment part. Here, we evaluate them quantitatively. Following the main paper, the evaluation metric is the FID score. The results are summarized in Tab. It is obvious that without TCSL, the performance degrades significantly since the feature cannot be transferred to the edited images correctly leading to inconsistent appearance compared to the input images. On the other hand, SCSL only aims adjust the style of novel-appearing parts, which does not determine the score. When including the two losses, the edited images are close to the input real images in the aspects of color, texture, and style.
Additional Loss Items:
Primitive Prototype Library: In the main paper, we collect primitive prototypes to represent 6 different kinds of articulated objects in the testbench. Since our method does not require precise 3D CAD models of objects, Primitive Prototype Library can cover a wide range of articulated objects with small number of primitive prototypes. As shown in Fig. 10 and Fig. 11, books can be represented as laptops, doors can be represented with simple planes, and toilets can be represented with a plane and boxes. No extra prototypes are required when creating abstract 3D models for
16

---

Original
Manipulated
Original
Manipulated
Original
Manipulated
Original
Manipulated
Figure 10: Additional demonstration of the images edited with our proposed PA-Diffusion model, including storage furniture, laptop, microwave, trashcan, door, drawer, and refrigerator.
Category
None
only SCSL 4
only TCSL
All Losses
Storage.
1.48
1.54
0.99
0.81
Laptop
2.30
1.92
1.88
1.94
Microwave
1.52
1.71
0.71
0.87
Trashcan
1.39
1.65
0.70
0.95
Refrigerator
1.99
1.81
1.79
1.38
Drawer
1.63
1.25
0.51
0.60
Avg.
1.72
1.65
1.10
1.09
Table 4: FID score of edited images with different additional losses: SCSL and TCSL, with SCSL. only, with TCSL only, and with all losses. The performance improves 36.6% with the assistance of the two losses.
novel categories. Besides, the edited images are still high-fidelity and high-quality. Furthermore, the Primitive Prototype Library easy expand, more primitive prototypes can be created collected rapidly when dealing with novel articulated objects.
2D-3D Models analysis: Besides the advantage of convenience, also compare the quality of reconstructed 3D models with state-of- -fete--rrrrt 2D-3D methods and abstract 3D models.
Following the method introduced in Image Sculpting [51], we use ClipDrop to remove the background of input images, and then reconstruct models with Zero [28]. The tested images are the same as those in the main paper. The reconstructed 3D object mesh models are shown in Fig. 12 including the front view, side view, and the edited images after manipulation. We notice that the result of storage furniture is acceptable, the shape closely matches the original, and the texture is stored in UV
17

---

Figure 11: Additional demonstration of editing images with manipulation process based on our proposed PA-Diffusion model. The last three columns are novel articulated object categories.
maps correctly. However, for other categories, the reconstructed models are poor which is the main reason for low-quality edited images.
In the main paper, mentioned that manipulating the reconstructed 3D models created by 2D-3D methods is tedious and inaccurate. As shown in Fig. 12, the reconstructed models are not part-level (only mesh object), tremendous human effort is required manipulation. For example, when trying to split the door from the body storage furniture, we need cut the furniture into several parts first (top, bottom, four sides of the body), and then merge others except the door. This process, even with advanced 3D graphics software, is complex and labor-intensive. As result, summarizing reconstruction, manipulation, and generation time, Image Sculpting [51] requires over 10 mins to manipulate one image, making it unsuitable for large-scale image editing tasks.
the mesh into parts before
In summary, using abstract 3D models to present articulated objects offers several advantages: (1) State-of- oftte-of-thh-art 2D-3D methods are still not robust enough create precise 3D models such as laptops and trashcans. In comparison, it easy to achieve abstract 3D models with primitive prototypes, as shown in Fig. 12. (2) Manipulating primitive prototypes in the 3D space is easier and than manipulating single 3D object mesh. (3) Novel categories and instances can be handled with method efficiently. (4) Our method is brief and thus support various downstream tasks.
accurate
D 3D Articulated Objects Understanding
Overview. In the main paper, we introduce the 3D articulated object understanding experiment and demonstrate how our proposed PA-Diffusion model supports other research fields. Here, we discuss the experiment setup and implementation in more detail.
Annotation Generation. In previous works [38, 37, 27], human labeling is required for annotating the bounding boxes, rotation/translation axes, and surface normal, which is labor-intensive and inaccurate. On the contrary, by representing objects with abstract 3D models, our method can achieve these annotations automatically.
As shown Fig. 13, part1 and part2 are the masks articulated objects' parts that are exported from Grounded SAM Blender software. The corner points can be calculated with [1, automatically. Then, the bounding box of each object part can be calculated with these corner points. The rotation and translation axis annotations represented [ro, yo, 21, M1], where zo, yo. 21, V1 the coordinates
18

---

Original
Front View
Side View
Manipulated
Front View
Manipulated
Image Sculpting
Ours
Figure 12:
Reconstructed 3D object models with Image Sculpting, and our abstract 3D models created with Primitive Prototype Library.
of corner points part-level masks, different object category uses different corner points. For the 3D surface normal annotations, shown in Fig. 14, we first export the object's part world transform matrix Masj and camera world transform matrix Mzamera- Then the two matrices are normalized and calibrated to create the aligned transform matrix Maligned- Consequently, we simply select the normal of one plane to represent the orientation of the object part, and the surface normal Vsurs is equal the multiplication of the aligned matrix and local plane vector Vplane:
M Maligned = Align( N Form(M) Worm(MMaamne (Maamera), N Norm Motg)) )):
(11)
Vsurf Moligned Vplone-
Evaluation Matrix. For quantitative evaluation, we follow [38] to calculate the average precision of the bounding box, axis, and surface normal. The bounding box is the traditional horizontal type, the threshold of IoU is set as 0.5. The predicted axes are measured with EA score as [53]. To demonstrate the results clearly, calculate the surface normal eиrоr and measure the accuracy that the error is smaller than the threshold 30".
E Limitations and Future Research
The limitations of our proposed PA-Diffusion models have been discussed in the main paper. Due to the inaccuracy of DDIM inversion, the inverted noise map might be poor if the input image quality is low. Unfortunately, the poor noise map will lead to mismatch error accumulation and propagation during the iterative denoising process. As in the left of Fig. 15, when the original input image is of low resolution (it normalized to 512 x 512 before manipulation), the PA-Diffusion model cannot re-generate the original image with the inverted noise map. Simple actions like moving and scaling also cannot be completed.
Manipulating the initial inverted noise maps is critical preserve the appearance of seen parts. However, discussed the main paper, this step disturbs the original data distribution. The problem will be too serious to be fixed when the object shape deformation is large. As shown in the right of
19

---

Fig. 15, when reshaping the laptop to non-uniform diamond shape, the object's 's appearance cannot be preserved.
Considering this situation, one promising solution is to add stronger and more precise supervision loss in each denoising step. This is beyond the scope of this work, plan implement this later.
In the future, more categories of articulated objects will covered and the edited image dataset will be expanded to millions-scale for supporting various computer vision and robotic manipulation tasks. Next, we will extend this method to handle deformable objects and fluids.
F Societal impacts and potential risks
The articulated object manipulation method presented in this work has profound positive societal implications. This method can serve a fundamental tool benefit other computer vision robot vision tasks. Consequently, the artificial intelligent algorithm can understand and interact with the real world better. Humans will have stronger Al assistants including smart offices, intelligent home or medical robots, and so on.
All the models and data used in this work are collected from the Website. No personal information is used. The code and data created in this work have a low risk of misuse.
20

---

Part1
Part2
RGB
Part1
Part2
RGB
Figure 13: Demonstration of extracting bounding box annotations and rotation/translation axis annotations from part-level masks.
(2)
(5)
(6)
Figure 14: The camera poses and orientations of 6 planes in Blender, (1)-(6) refers to abstract 3D models of cabinet, laptop, microwave, trashcan, drawer, and refrigerator separately. Red arrows the surface normal directions.
Original
Re-generate
Move to right
Original
Target Shape
Edit Result
Figure 15: Limitation of the PA-Diffusion model: dealing with low-quality images or large deforma- tion.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contribution: PA-Diffusion model, novel articulated object manipulation method in real images has been claimed the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitation discussed in both the main paper and the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when the image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
The paper introduces a novel method, there is no theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in the appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The completed algorithm pipeline is provided in the Appendix. Each step is also introduced in both the main paper and the Appendix. The context explanation and figures are aligned to explain the proposed method. Besides, make the algorithm easy to understand, we provide a simpler explanation in the Appendix as well. All the experimental results can be reproduced.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
23

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code will be published.
Guidelines:
The answer NA means that the paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
oolyyyyooollyoooloooolomommmyycp for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cc/ ctc/public/// wwtiiciggg//////eeeeeieiiicisset CCodsS/mmdmmmmcic Sissuubmissssss aassonPollll onnolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details are described in the session of the experiment in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, the appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: The paper reports the experiment results on certain datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar and then state that they have 96% CI if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of computing workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The required computing resources are claimed the session the experiment in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute worker CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required computing than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpp/://////pp..../// Ceabllc//thicsSuideee
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g. if there a special consider- ation due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
It is discussed in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for the responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: It is described in the Appendix. The code and data of this work have quite low risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but encourage authors take this into account and make the best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited, and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the models and data used in this work are published, and all the citations have been included in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datastts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: There is no crowdsourcing experiments and research in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing or research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work has no crowdsoureing experiments and research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "6JcpJX1xGk", "reviewer": "Reviewer_1MgD", "summary": "The paper proposes a method for accurately generating edited images for manipulating articulated objects, effectively avoiding hallucinations. The overall idea and approach are very interesting. The implementation of the method is particularly impressive, presenting a novel articulated object manipulation technique that covers common object categories and supports arbitrary manipulation.", "questions": "1. The description of section discussing manipulation in L122 is unclear. For instance, if an object has many movable parts, how is it determined which part should be moved? Taking the example in the figure, a laptop has two planes. If the text says \"open laptop,\" how is it determined which plane should be moved?\n2. The goal of \"Abstract 3D Model\" in L105 should be made clearer. For example, I am wondering why single-image reconstruction methods are not used to obtain the 3D model. Although I understand that using prototype reconstruction methods facilitates subsequent manipulation of arbitrary parts, I believe the authors should clearly explain the reasons and purposes behind each method detail.\n3. In the abstract and introduction, the authors repeatedly mention that edited images are significant for robotic manipulation learning. Could authors explain in detail why edited images are beneficial for robotic manipulation? If conducting experiments is challenging, analyzing the advantages of image editing for robotics is also acceptable. For example, what limitations do existing image goal-conditioned manipulation policies have, and can this editing method address these limitations?\n4. In Figure3, there are masks M in both top and bottom row. But why do the image M^{gen} on the top and image M_{s}^{gen} on the bottom look so different? I suppose mask should be in binary mask, but why M^{gen}  is so colorful?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is interesting", "weaknesses": "please refer to question section", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThanks very much for your valuable suggestions. In the following, we answer all of your questions in detail and carefully.\n\n**Q** : The description of section discussing manipulation in L122 is unclear. For instance, if an object has many movable parts, how is it determined which part should be moved? Taking the example in the figure, a laptop has two planes. If the text says \"open laptop,\" how is it determined which plane should be moved?\n\n**A** : Thank you for your insightful question, we will incorporate more explanation into the paper for clarity. In our work, we pre-define the movable parts in **the mapping table**, aligning with **common manipulations encountered in daily life**. As illustrated in Figure 9 in the appendix session, the mapping table specifies which parts should be moved based on the provided text instructions. For instance, the instruction ‘open laptop’ pertains to manipulating the screen plane of the laptop. \n\nFurthermore, the PA-Diffusion model is capable of handling multiple movable parts as well. As demonstrated in Figure 5, the text instruction ‘open drawer and open cabinet’ allows for the concurrent opening of both the drawer and the cabinet. More generally, the mapping table can be extended to manage multiple movable parts within a single object, thereby enhancing the model’s flexibility and applicability. Human interaction is also supported.\n\n**Q** : The goal of \"Abstract 3D Model\" in L105 should be made clearer. For example, I am wondering why single-image reconstruction methods are not used to obtain the 3D model. Although I understand that using prototype reconstruction methods facilitates subsequent manipulation of arbitrary parts, I believe the authors should clearly explain the reasons and purposes behind each method in detail.\n\n**A** : Thanks for the instructive suggestion. In this work, we highlight several advantages of using Abstract 3D Models compared to single-image reconstruction methods: \n\n1) Abstract 3D Models demonstrate **better accuracy** in generating 3D models across various object categories.\n\n2) Abstract 3D Models are **easier** to manipulate, particularly for articulated objects.\n\n3) Abstract 3D Models provide a clear definition of object parts.\n\n4) Abstract 3D Models allow efficient coverage of **novel** instances and categories, whereas single-image reconstruction methods require fine-tuning with additional samples.\n\n5) Using Abstract 3D Models makes the image editing process **time-efficient** enough to support other downstream tasks, which is previously challenging.\n\nPlease refer to Columns 3 and 4 in Figure 12, the reconstructed 3D models produced by ZERO123 [6] are inadequate for supporting image editing. Besides, manual effort is required to split and manipulate the 3D mesh model, which is tedious and costly. For further comparison and analysis, please refer to subsection A4 in the appendix section.\n\nNonetheless, we recognize the value of single-image reconstruction methods as a complement to Abstract 3D Models in certain situations. As shown in Figure 5, we selected ZERO123 [6] to generate the 3D model for the toy shark with a non-uniform shape. We observe that the PA-Diffusion can create high-fidelity edited images with this 3D model as well.\n\n**Q** : In the abstract and introduction, the authors repeatedly mention that edited images are significant for robotic manipulation learning. Could authors explain in detail why edited images are beneficial for robotic manipulation? If conducting experiments is challenging, analyzing the advantages of image editing for robotics is also acceptable. For example, what limitations do existing image goal-conditioned manipulation policies have, and can this editing method address these limitations?\n\n**A** : Thanks. We respectfully explain how the PA-Diffusion model can benefit robotic manipulation tasks in two key areas: **sub-goal generation** and **data augmentation**. **Please check the attached file in the global rebuttal, there is a robot manipulation demo**.\n\nRegarding sub-goal generation, recent works [1], [2], [3] have introduced the concept of generating sub-goal conditions to assist with long-horizon tasks. This approach improves the overall manipulation success rate. \n\nFor data augmentation, [4] proposed augmenting datasets by changing the appearance of objects or backgrounds. In comparison, the PA-Diffusion model supports arbitrary articulated object manipulation in real images, **significantly enriching** the dataset and supporting more complex robotic tasks. \n\nMoreover, directly predicting the manipulation process and extracting manipulation policies have emerged as promising methods to improve the success rate [5]. The PA-Diffusion model shows great potential in enhancing these methods by generating high-quality manipulation processes for various objects. We are currently conducting further research on robotic manipulation alignment with this approach. \n\n**Q** : In Figure3, there are masks M in both top and bottom row. But why do the image Mgen on the top and image Mgen on the bottom look so different? I suppose mask should be in binary mask, but why Mgen 118 is so colorful?\n\n**A** : In Figure 3, Bottom Seen mask refers to top Part1 mask, bottom Novel mask refers to top (Part2 - Part1) mask. We have made the masks colorful to distinguish the seen and novel-appearing parts, aiming to simplify and clarify the editing process. \n\nWe appreciate all of these questions and will incorporate these analyses into our paper.\n\nRefs are shrunk.\n\n[1] Subgoal Diffuser: Coarse-to-fine Subgoal Generation to Guide Model Predictive Control for Robot Manipulation\n\n[2] Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models\n\n[3] OpenVLA: An Open-Source Vision-Language-Action Model\n\n[4] GenAug: Retargeting behaviors to unseen situations via Generative Augmentation\n\n[5] Generative Image as Action Models\n\n[6] Zero-1-to-3: Zero-shot One Image to 3D Object", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0KwR9TBhmv", "reviewer": "Reviewer_N2Nu", "summary": "This paper introduces a novel diffusion-based method for manipulating articulated objects in real-world images from input text guidance or human interaction. There are three main contributions in this paper: (i) the authors introduce the concept of an Abstract 3D Model, which eliminates the requirement for a 3D model by using primitives such as cuboids, boxes, etc to represent various types of articulated objects; (ii) the authors present dynamic feature maps, which transfer the seen parts from the input image to the generated image and only generate the novel parts, which is very intuitive; (iii) the test benchmark includes 660 object samples and a total of 3300 images.", "questions": "Overall, I found the paper is well-written, and the contributions are strong and clear. However, I have a few questions to better understanding of the proposed methods and address the weaknesses mentioned above:\n\n- Can the proposed methods effectively handle input objects whose shapes do not match the available primitives?\n\n- Why are the drawers of opened objects consistently empty?\n\n- Could you provide an analysis of shadows in the generated images, as this is a crucial factor in distinguishing whether the images are generated or real?", "limitations": "Yes, the authors clearly stated the limitation of proposed method (i) when input images are blurry, in low-resolution, (ii) when input objects are deformable and or fluids.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The most significant strength is the training-free method, i.e, given an input image at test time, the proposed method uses texture and style guidance loss to optimize the output image using only different pre-trained models. This training-free approach allows easy generalization to novel categories and different types of objects.\n\n- I like the concept of the Abstract 3D Model, as acquiring 3D models of in-the-wild objects can be challenging and costly (although I have some doubts, as noted in the Weaknesses section below). \n\n- The proposed dynamic feature maps are also very intuitive, ensuring that the generated image remains consistent with the seen parts of the input image while generating only the novel parts. \n\n- The proposed methods outperform previous works on several metrics and benchmarks.", "weaknesses": "-  While I like the concept of the Abstract 3D Model, I am not sure about its effectiveness when the input objects have shapes that do not conform to the available primitives, such as a round table, etc. In such cases, the part-level masks and sketch maps can be very different. \n\n- The authors' observation (L218) and the qualitative results do not well explain why articulated objects like refrigerators or furniture always appear empty when opened. I would expect it is randomly generated, with some cases showing objects, fruits and with other cases empty.\n\n- The generated shadows in the output images also seem under-analyzed. In some instances, the shadows do not change even though the object has been articulated (e.g., some furniture samples in Figure 6).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for all of your kind suggestions and questions. In the following, we answer all of your questions in detail.\n\n**Q** : Can the proposed methods effectively handle input objects whose shapes do not match the available primitives?\n\n**A** : Thank you for your valuable question. Generalization is indeed a critical factor, and the proposed PA-Diffusion model is designed to handle various articulated objects from the following three aspects:\n\nFirst, **Primitive Prototype Library** encompasses many categories of common articulated objects by utilizing prototypes. As illustrated in Figure 4, Figure 10 and Figure 11, six primitive prototypes can adequately represent objects such as laptops, doors, and other categories. By extending the library, we can accommodate additional categories and instances, thereby significantly enhancing support for robotic tasks.\n\nSecond, when the shapes of objects **mismatch significantly** from existing primitive prototypes, we can create novel prototypes as needed, which is both efficient and straightforward. \n\nLastly, leveraging current **image-to-3D methods** gives us another viable option. As demonstrated in Figure 5, we utilize ZERO123 [1] to generate 3D models for non-uniform objects before proceeding with image editing, and the edited image is still reasonable and high-fidelity.\n\n**Q** : Why are the drawers of opened objects consistently empty?\n\n**A** : Thank you. We consider this consistency an **advantage** of the proposed PA-Diffusion model. As illustrated in Figure 4, when objects are manipulated continuously, the style and appearance of the novel-appearing parts remain consistent. This consistency makes the method suitable for video generation.\n\nThis consistency is achieved through **the text prompts** and **sketch conditions** used during the image editing process. For instance, when manipulating drawers, the text prompt \"a photo of an opened drawer\" is applied uniformly across all test cases, while the sketch map consistently depicts an empty drawer. These two factors contribute to the uniform appearance of opened drawers.\n\n**Q** : Could you provide an analysis of shadows in the generated images, as this is a crucial factor in distinguishing whether the images are generated or real? \n\n**A** : This is an inspective question. Our primary focus is on manipulating articulated objects in images, meanwhile preserving the appearance of the background. The shadow region is considered part of the background and is therefore maintained during the image editing process. This challenge could be addressed by incorporating additional prompts or other shadow manipulation techniques [2], [3].\n\nWe appreciate your suggestion in guiding our research, and in the next steps, we will consider the impact of shadows to enhance the quality of the edited images, which is indeed an important aspect.\n\nReference\n\n[1] Ruoshi Liu, et. al, Zero-1-to-3: Zero-shot One Image to 3D Object, ICCV, 2023\n\n[2] Qingyang Liu, et. al, Shadow Generation for Composite Image Using Diffusion Model, https://arxiv.org/pdf/2403.15234\n\n[3] Jinting Luo, et. al, Diff-Shadow: Global-guided Diffusion Model for Shadow Removal, https://www.arxiv.org/pdf/2407.16214", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RtYlWMqtDG", "reviewer": "Reviewer_YyMw", "summary": "This paper presents a pipeline to directly manipulate articulated objects in real images and generate corresponding images in different articulated poses. The proposed method adopts a  2D-3D-2D approach with a heuristic model to obtain 3D information from source images and generate new images based on the diffusion model. No extra fine-tuning and training is required for the proposed method. Quantitative evaluation is presented for this method and qualitative comparison with baseline methods.", "questions": "The key questions for this paper focus on the Sec. 3.2.\n\nFor abstract 3D model generation. How the proposed method analysis the structural 3D information of the object given images? For example, in the Figure. 3, how does the pipeline obtain the following knowledge from the$Z_T^{init}$:\n\n1. The object is composed of 2 parts, especially the base of the laptop is almost invisible (number of parts discovery)\n2. These two parts fit with the plane prototypes (part to prototype fitting)\n3. These two planes are connected\t(part-level relation discovery)\n4. One of the plane can be rotated along the connected axis (joint type estimation between revolute and prismatic)", "limitations": "See above", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Training-free pipeline for articulated object manipulation is a plus\n2. Works out-of-box for real-world objects is a plus\n3. The qualitative evaluation demonstrates impressive results", "weaknesses": "1. How to obtain 3D information, part-level understanding and structural information is not detailed in Sec. 3.2, which is a key for articulated object manipulation.\n2. The text instruction for manipulation generation is based on heuristic models, which may not be flexible.\n3. Difficult to follow in Sec. 4.6. Lack of context for the reference work [1][1] (which is [32] in the paper) in this subsection, making it difficult to understand and validate.\n   1. As the author mentioned in Sec. 4.1, **NO** training is needed, but Sec. 4.6 talks about dataset spliting and model training, very confusing.\n   2. If the \"model\" is trained in this subsection, what model? with what objectives?\n   3. Write the mathematical definition or explain the metrics used in this subsection, Tab. 3.\n4. It seems the manipulation is not limited to part-level manipulation, but also applies to the object as a whole, such as move the object to the right. Could this method also extends to object-centric view synthesis? Would be interesting to see experiments on this.\n5. It would be better to cite and compare to this similar work [2]\n\n[1] Qian, Shengyi, et al. \"Understanding 3d object articulation in internet videos.\"  *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* . 2022.\n\n[2] Li, Ruining, et al. \"Dragapart: Learning a part-level motion prior for articulated objects.\" *arXiv preprint arXiv:2403.15382* (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nIn the following, we answer all your questions in detail. Thanks very much!\n\n**Q** : Difficult to follow in Sec. 4.6. Lack of context for the reference work [1] (which is [32] in the 187 paper) in this subsection, making it difficult to understand and validate. As the author mentioned in Sec. 4.1, NO training is needed, but Sec. 4.6 talks about dataset splitting and model training, very confusing.\n\n**A** : We sincerely apologize for any confusion in our paper and thank you very much for informing us. We will refer to your comments to revise Section 4.6 and clarify this in the final version.\n\nFirst and foremost, NO training is required for the proposed PA-Diffusion model. Subsection 4.6 serves as an **additional robotic application to illustrate the value of the PA-Diffusion model**. The selected application is the Articulated Object Understanding task, which involves developing a model to estimate the bounding boxes, joint axes, and surface normals of articulated objects [1] in real image/video. We call it the Articulated Object Understanding Model (AOU Model) in the following content.\n\nWe generated 3,300 edited images using the proposed PA-Diffusion model, NO training is required in this step. Subsequently, we conducted two separate experiments to validate the quality of the edited images (these edited images are used as training/testing data in the experiments):\n\nWe regard the first experiment as a self-test. These 3,300 images were split into training/testing sets to develop/evaluate the AOU Model. The results, presented in Table 2 indicate that more training data can improve the model’s performance, which illustrates that edited images play the same role as real ones\n\nIn the second experiment, we select InternetVideo Dataset (a real image dataset) [1] as the baseline. The 3,300 images were mixed with the training set of InternetVideo [1] to train the AOU Model, and evaluate the AOU Model on the testing set (real image as well). The results, shown in Table 3 prove that edited images can enhance the model’s performance, even when evaluated on real images\n\n**Q** : If the \"model\" is trained in this subsection, what model? with what objectives?\n\n**A** : Thank you for this question. The Articulated Object Understanding Model is trained in subsection 4.6 (not the PA-Diffusion model). The objectives are: (1) Bounding box loss, (2) Axes loss, and (3) Surface Normal loss. Due to the number limitation of character, we will include detailed explanations and definitions in the appendix section. \n\n**Q** : Write the mathematical definition or explain the metrics used in this subsection, Tab. 3.\n\n**A** : The evaluation matrix used in Tab. 3 are: (1) Bounding box evaluation matrix, (2) Axes evaluation matrix - Euclidean distance and Angular distance Score, and (3) Surface normal evaluation matrix. Thresholds are set for these metrics, and we utilize the accuracy to evaluate the model. Due to the number limitation of characters, we will include detailed explanations and definitions in the appendix section. \n\n**Q** : It seems the manipulation is not limited to part-level manipulation, but also applies to the object as a whole, such as move the object to the right. Could this method also extends to object-centric view synthesis? Would be interesting to see experiments on this.\n\n**A** : Thank you for this insightful suggestion. **Given that objects can be manipulated in various ways, the proposed method is intended to support object-centric view synthesis**. We are actively working on implementing this feature, as it represents a compelling aspect of our research.\n\n**Q** : It would be better to cite and compare to this similar work [2] \n\n**A** : Thank you. We will cite [2] and include an analysis of their work in our paper. Reference [2] proposed an innovative method for manipulating articulated objects by modifying feature/attention maps to perform various editing tasks. In comparison, our approach maps the objects to 3D space for manipulation and then modifies the initial inverted noise maps to (1) preserve the objects’ appearance, and (2) generate novel-appearing parts.\n\n**Q** : The key questions for this paper focus on the Sec. 3.2. For abstract 3D model generation. How the proposed method analysis the structural 3D information of the object given images? For example, in the Figure. 3, how does the pipeline obtain the following knowledge from the : The object is composed of 2 parts, especially the base of the laptop is almost invisible (number of parts discovery) \n\nThese two parts fit with the plane prototypes (part to prototype fitting)\n\nThese two planes are connected (part-level relation discovery)\n\nOne of the plane can be rotated along the connected axis (joint type estimation between revolute and prismatic)\n\n**A** : Thank you for your insightful question. The structural information is pre-defined for each articulated object category according to the common knowledge. For instance, a laptop consists of two parts and a rotation axis. Based on this knowledge, objects in real images are segmented using Grounded SAM, and then abstract 3D models are created referring to the part-level segmentation masks. This procedure ensures that the structural knowledge is incorporated into the abstract 3D models. The flexibility of the proposed PA-Diffusion model further allows for the efficient incorporation of novel structural information and objects.\n\nIf there exist any other questions or confusion, please inform us at any time you feel convenient. Your questions and suggestions are so beneficial to our research. Thank you very much.\n\nReference\n\n[1] Qian, Shengyi, et al. \"Understanding 3d object articulation in internet videos.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022\n\n[2] Li, Ruining, et al. \"Dragapart: Learning a part-level motion prior for articulated objects.\" arXiv preprint arXiv:2403.15382, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "icB79LgHaf", "reviewer": "Reviewer_pkKj", "summary": "This paper focuses on manipulating articulated objects in 2D images by leveraging a 2D-3D-2D approach with a diffusion model. Specifically, the authors propose the abstract 3D model to represent the articulated objects and propose dynamic feature maps to transfer seen regions while generating novel areas. The results on image editing and 3D articulated object understanding show the effectiveness of this method.", "questions": "1. Could the authors please provide a more detailed description of how to build an abstract 3D model regarding my major concern? I would consider raising or lowering the score based on the way of constructing a 3D model from 2D images.\n\n2. The paper mostly describes and showcases the editing of opening or increasing (e.g., opening door and laptop). I am wondering about the process of 'decreasing' since the design of dynamic feature maps needs to be changed accordingly (e.g., ${M_n}^{Gen}$ should be all zeros, or should it refer to novel backgrounds that are previously occluded by the objects).\n\n3. In line 228, the paper shows that this method could be applied to more objects with more manipulation types. Could the authors provide more description about how this can be done? Do they also require abstract 3D models and manipulation in Blender?\n\n4. Minor: In Figure 4, why do the third column edited images share the same backgrounds while they are not the same as the original one?", "limitations": "The author described the limitations. Further discussion is better (Please see the Weakness section).", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This method is well-motivated: editing images with 3D prior and transferring seen parts is intuitively suitable for articulated object manipulation. The specific techniques applied are also very reasonable and helpful for the task. For example, the design of an abstract 3D model is a good workaround for representing articulated objects instead of using fine-grained 3D meshes. Additional loss functions also make sense.\n\nThe experiments are thorough, including two tasks, both qualitative and quantitive, showing the effectiveness of the method in both image editing and down-stream understanding.\n\nThe writing is easy to follow, and the appendix provides abundant details.", "weaknesses": "My major concern about this approach is its limited application. Specifically, the current process of building abstract 3D models from 2D images might significantly hinder the generalization. The authors create 6 primitive prototypes to represent 6 categories in the dataset. However, this ignores the structure variance among instances in the same category (e.g., a cabinet could have 1-4 drawers and even combine with doors). Therefore, this method actually requires a pre-defined abstract 3D model for each structure. Manually designing an abstract 3D model for each image is clearly impractical, whereas automatically matching 3D prototypes to objects is not trivial, and many works have been done on this topic [1,2].\n\nThe process of defining an abstract 3D model for each category also injects crucial priors to the model (i.e., the structure, joint configuration, and kinematic property of the object), which gains advantages compared with baselines that do not require information about the category.\n\n[1] Unsupervised learning for cuboid shape abstraction via joint segmentation from point clouds.\n\n[2] Learning to Infer and Execute 3D Shape Programs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThanks for your comments and questions. In the following, we answer all of your questions carefully.\n\n**Q** : Could the authors please provide a more detailed description of how to build an abstract 3D model regarding my major concern? I would consider raising or lowering the score based on the way of constructing a 3D model from 2D images.\n\n**A** : Thank you for your thoughtful question. Generalization is indeed a critical challenge for real-world applications. We believe that the proposed PA-Diffusion model is capable of building Abstract 3D Models for **both simple and complex** articulated objects.\n\nIn the paper, we demonstrate that owning to primitive prototypes and Abstract 3D Models, the PA-Diffusion model can handle **a wide range of simple** articulated objects and benefit downstream tasks in robotic applications. The model efficiently incorporates novel shapes, instances, or categories, even when the creation of new prototypes is necessary. Compared to the costly manual collection of robot data, the PA-Diffusion model presents a promising solution for augmenting robot datasets.\n\nMoreover, the PA-Diffusion model can still facilitate the manipulation of **complex objects with extra information**. This research focuses on articulated objects with a single joint, but the PA-Diffusion model can also effectively manage more complex items, such as a cabinet with multiple drawers and doors. This is achieved through an iterative editing process, where each component is modeled and manipulated individually in one step. Sequentially, the results from these manipulations can be concatenated with extra information such as URFD files or descriptions of articulated object structures. Lastly, the PA-Diffusion model edits the images following the same pipeline as simple objects. As depicted in Figure 5, we map both the drawer and cabinet door to a 3D space and manipulate them accordingly. The PA-Diffusion model subsequently edits the images based on these manipulation results.\n\nWe appreciate your interest in this aspect of our work, and we hope this clarifies the capabilities of the proposed PA-Diffusion model.\n\n**Q** : The process of defining an abstract 3D model for each category also injects crucial priors to the model (i.e., the structure, joint configuration, and kinematic property of the object), which gains advantages compared with baselines that do not require information about the category. \n\n**A** : While Abstract 3D Models incorporate prior knowledge about the object’s structure, joint configuration, and kinematic properties, we have taken measures to **ensure the fairness of the experimental comparison**. \n\nWe selected Imagic, DragDiffusion, MasaCtrl, and Image Sculpting as baselines. Specifically, (1) DragDiffusion requires human interaction, which introduces a more critical prior; (2) MasaCtrl and Image Sculpting utilize the same prior knowledge as our proposed method; (3) Imagic relies solely on text instruction, however, its performance is notably lower compared to the others. Therefore, we believe that this constitutes a fair comparison. We appreciate your question and will include this explanation in the paper.\n\n**Q** : The paper mostly describes and showcases the editing of opening or increasing (e.g., opening door and laptop). I am wondering about the process of ’decreasing’ since the design of dynamic feature maps needs to be changed accordingly (e.g., should be all zeros, or should it refer to novel backgrounds that are previously occluded by the objects).\n\n**A** : For the ’decreasing’ process, our PA-Diffusion model would in-paint backgrounds previously occluded by objects. The situation is similar with moving/rotating objects, as illustrated in Figure 4. When objects are moved or rotated, the resulting blank regions are semantically in-painted based on the surrounding background. Notably, the editing and in-painting are completed in a single step, eliminating the need for an additional in-painting phase. \n\n**Q** : In line 228, the paper shows that this method could be applied to more objects with more manipulation types. Could the authors provide more description about how this can be done? Do they also require abstract 3D models and manipulation in Blender?\n\n**A** : Indeed, for other articulated objects and manipulation tasks, the proposed PA-Diffusion model **adheres to the established pipeline**: constructing abstract 3D models, manipulating them in Blender, and generating edited images. Figure 5, Figure 10, and Figure 11 demonstrate more objects and more manipulation briefly. Please refer to Figure 5, we demonstrate the model’s capability to manipulate non-articulated and non-uniform articulated objects through novel operations, such as breaking cups and opening kitchen pots. Additionally, in Figure 10 and Figure 11, we tested this pipeline on more categories such as doors, toilets, and books.\n\n**Q** : Minor: In Figure 4, why do the third column edited images share the same backgrounds while they are not the same as the original one? \n\n**A** : Thank you. In Figure 4, the variation in backgrounds results from several factors. First, changes in lighting contribute to the variation. Lighting is one of the elements influencing the generation of images with modern Diffusion Models. As we manipulate objects, Diffusion Models account for the altered objects' shapes and adjust the lighting effects accordingly. Second, in-painting the blank regions introduces variations, since in-painting and editing are implemented in the same denoising process. Lastly, the quality of original input images could lead to this as well. To mitigate this, we might enhance the supervision to better preserve the backgrounds during the editing process.\n\nIf there exist any other questions please inform us at any time you feel convenient. Thanks a lot.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We sincerely appreciate all of your insightful and valuable questions and suggestions. We have made a concerted effort to address each query thoroughly and have revised the paper comprehensively following your recommendations. Your perceptive insights have undeniably enriched our work.

It is widely acknowledged that modern deep learning-based robotic manipulation models can perform a variety of tasks in real-world environments [1-5]. However, the available datasets are still insufficient for developing a truly generalizable model since collecting large-scale robotic datasets is expensive. In light of this limitation, the proposed PA-Diffusion model is introduced to manipulate various articulated objects in real images, which approaches this challenge from **a novel perspective by utilizing low-cost image editing techniques to mitigate the reliance on expensive robotic data**.

In this work, we demonstrate how the PA-Diffusion model manipulates various articulated objects in real images, and the edited images can serve as valuable resources for other computer vision and robotics-related tasks. Furthermore, the flexibility of the PA-Diffusion model allows for easy adaptation to new objects, positioning it as a promising approach to bridge the gap between general robotic manipulation and the challenges posed by limited data availability. 

To illustrate this point, we present an intuitive example in the field of robot data generation. As demonstrated in Figure 1 in the attached file, the PA-Diffusion model effectively manipulates the microwave to key states as represented in (a), and then the 3D end-effector pose can be extracted in each state (b). Consequently, within robot simulators such as RLBench, the robot arm, and end-effector are moved to specific poses (c). During this process, the necessary information, including joint positions, point clouds, and so on, can be loaded and recorded by reading the current state from the simulator. In the end, a complete manipulation sample is generated from a single RGB image.

When compared with relying on simulators [6],[7],[8],[9] to synthesize manipulation samples, the PA-Diffusion model offers **more versatile samples featuring different objects and backgrounds**. This ongoing work holds significant potential and merits further investigation.

We are truly grateful for your invaluable guidance. Thank you again.

Reference

[1] M. Shridhar, et. al, Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785–799. PMLR, 2023

[2] C. Chi, et. al, Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.

[3] Z. Xian, et. al, Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference on Robot Learning, 2023.

[4] X. Ma, et. al, Hierarchical diffusion policy for kinematics aware multi-task robotic manipulation. arXiv preprint arXiv:2403.03890, 2024.

[5] V. Vosylius, et. al, Render and diffuse: Aligning image and action spaces for diffusion-based behaviour cloning, 2024.

[6] E. Rohmer, et. al, Coppeliasim (formerly v-rep): a versatile and scalable robot simulation framework. In Proc. of The International Conference on Intelligent Robots and Systems (IROS), 2013. www.coppeliarobotics.com

[7] S. James, et. al, Pyrep: Bringing v-rep to deep robot learning. arXiv preprint arXiv:1906.11176, 2019

[8] S. James, et. al, Rlbench: The robot learning benchmark learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020

[9] Soroush Nasiriany, et. al, RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots, Robotics: Science and Systems, 2024

Meta-review: This manuscript describes an approach to image generation for articulated objects. The approach proceeds by providing guidance to the image interpretation process in the form of an abstract 3D model which captures the form of the articulation and allows the system to generate results that capture the intended variation. The paper was reviewed by a panel of experts who felt that the method was well presented and they appreciated the fact that the method was able to produce reasonably compelling results with relatively little user input. That being said there was significant discussion about the generality of the method and the need for the user to provide high level guidance. It is not clear from the results whether the system could be generalized to handle objects with larger numbers of degrees of freedom. While robotics is mentioned as a possible application area the experimental results do not show any examples of application to robotic arms with 3 or more joints. Nonetheless, there was a consensus that the method appeared to work reasonably well within the limited domain of abstract 3D models that were presented and readers may appreciate the insights that it offers
Decision: Accept (poster)