AdaManip: Adaptive Articulated Object Manipulation Environments and Policy Learning
Authors: Yuanfei Wang
Xiaojie Zhang
Ruihai Wu
Yu Li
Yan Shen
Mingdong Wu
Zhaofeng He
Yizhou Wang
Hao Dong
Venue: ICLR 2025 Poster (2025)
Keywords: Articulated Object Manipulation
Adaptive Mechanism Environments
Imitation Learning
Primary area: applications to robotics, autonomy, planning

Abstract: Articulated object manipulation is a critical capability for robots to perform various tasks in real-world scenarios.
Composed of multiple parts connected by joints, articulated objects are endowed with diverse functional mechanisms through complex relative motions. For example, a safe consists of a door, a handle, and a lock, where the door can only be opened when the latch is unlocked. The internal structure, such as the state of a lock or joint angle constraints, cannot be directly observed from visual observation. Consequently, successful manipulation of these objects requires adaptive adjustment based on trial and error rather than a one-time visual inference. However, previous datasets and simulation environments for articulated objects have primarily focused on simple manipulation mechanisms where the complete manipulation process can be inferred from the object's appearance. To enhance the diversity and complexity of adaptive manipulation mechanisms, we build a novel articulated object manipulation environment and equip it with 9 categories of objects. Based on the environment and objects, we further propose an adaptive demonstration collection and 3D visual diffusion-based imitation learning pipeline that learns the adaptive manipulation policy. The effectiveness of our designs and proposed method is validated through both simulation and real-world experiments.

Published as a conference paper at ICLR 2025
ADAMANIP: ADAPTIVE ARTICULATED OBJECT MANIP- ULATION ENVIRONMENTS AND POLICY LEARNING
Yuanfei Wang" Xiaojie Zhang"? 2, Ruihai Wu'! Yu Li, Yan Shen', Mingdong Wu',
Zhaofeng He?, Yizhou Wang' 34.5 Hao Dong'
1 Center on Frontiers of Computing Studies, School of Computer Science, Peking University
2 Beijing University of Posts and Telecommunications
3 Inst. for Artificial Intelligence, Peking University
4 Nat'l Eng. Research Center of Visual Technology, Peking University
5 State Key Laboratory of General Artificial Intelligence, Peking University
ABSTRACT
Articulated object manipulation is critical capability for robots perform various tasks in real-world scenarios, Composed of multiple parts connected by joints, ar- ticulated objects are endowed with diverse functional mechanisms through complex relative motions. For example, a safe consists of a door, handle, and a lock, where the door can only be opened when the latch unlocked. The internal structure, such as the state of lock or joint angle constraints, cannot be directly observed from visual observation. Consequently, successful manipulation of these objects requires adaptive adjustment based on trial and eтror rather than a one-time visual inference. However, previous datasets and simulation environments for articulated objects have primarily focused simple manipulation mechanisms where the complete manipulation process can be inferred from the object's 's appearance. To enhance the diversity and complexity of adaptive manipulation mechanisms, we build novel articulated object manipulation environment and equip it with 9 categories objects. Based on the environment and objects, we further propose an adaptive demonstration collection and 3D visual diffusion- that learns the adaptive manipulation policy. The effectiveness of our designs and proposed method is validated through both simulation and weal-world experiments. Our project page available at: htttps: /tapaman/ wtlpp.ggtttt github. 1o
imitation learning pipeline
INTRODUCTION
Among the various categories of objects in our daily life, articulated objects are highly significant as they common in surroundings (such cabinets, doors, and laptops) and their components
complex, featuring rich and diverse geometries, semantics, articulations, and functions. Therefore,
learning articulated object representation (Du al. 2023 Wei al. 2022 Heppert al. 2023 Lei
et al. 2024) and manipulation (Xu et al. 2022 Wu et al. 2022 are essential while challenging for
future robots in home-assistant tasks.
Among articulated object manipulation tasks, door manipulation (Urakami et al. 2019) is first and most thoroughly studied, as doors are most common and useful in our daily lives. Afterwards, with the release of diverse articulated object manipulation datasets and environments (Mo et al.) 2019 Xiang al. 2020; [iu et al. 2022, Geng et al., (202b)). various manipulation tasks (like opening, sliding, rotating. and further language-guidee manipulation) many categories of articulated objects (such as pots, lamps, and cabinets) have been studied.
While previous covered various aspects of articulated object manipulation, one of the most essential features of articulated objects, the mechanisms of different parts and articulations for accomplishing the final manipulation goal, has yet to be explored. For example, a safe can be directly opened by pulling the door in previous environments, while in the real world, the robot may have to first turn the key unlock the latch, and then pull open the door. While MniDooraanip (Li et al., 2024b) proposes an environmen with the corresponding dataset that can simulate the mechanisms of doors,
* indicates equal contribution

---

Published as a conference paper at ICLR 2025
Open Safe
Pull Door
O
Turn Key
Open Door
Multi-modal Prior
Unable to update Posterior
Multi-modal Prior
Updute Posterior
Unseen Failure
Optimal Trajectories without Adaptation in Demo
Fallure Recovery present in Demo
Invisible Lock State
Static Policy
Adaptive Policy
Figure Example comparison between Static and Adaptive Policies. The safe can be directly opened if unlocked; otherwise, the key must be turned to unlock the latch before opening the door. However, it is impossible to figure out the lock state from pure visual observations. Static Policy: The demonstrations for training the static policy are optimal trajectories under full observation, including both locked and unlocked states. Consequently, the learned policy is a bimodal distribution based on visual observation alone. If the robot samples the "unlocked trajectory" and fails to open the locked door, it will be out of distribution. daptive Policy: The demonstrations for training the adaptive policy include recovery from the failed door opening. Therefore, the policy learns first pull the door to check the lock state and updates the policy distribution accordingly based on the feedback.
the mechanisms of various types of articulated objects could be much more diverse and complicated. Therefore, we build an environment that can simulate the -becr-ddee complex mechanisms of articulated object manipulation, equipping this environment with categories of different objects covering 5 types of adaptive mechanisms (details described in Section 33).
The different mechanisms of articulated objects call for two core capabilities of the policy: (1) multi- modal action proposal and (2) adaptive manipulation from history actions. For an observed object, the manipulation policy may contain multiple modes, including different manipulation trajectories. For example, when observing the safe with its door closed (Bottom-Left in Figure [ to achieve the goal of opening the door, the policy could be either directly pulling the door (when the door is unlocked) or turning the key and then pulling the door (when the door locked), and the method should be able to model these modalities from the same visual observation.
Furthermore, to identify and execute the accurate action from the modal manipulation action candidates, it is necessary adapt the manipulation policy based on previous actions and their corresponding outcomes. For example, when pulling the door results in no movement, the Adaptive policy should adapt from proposing multi-modal actions (either pulling the door or turning the key) to single-modal action (turning the key) (Bottom-Ripe Right in Figure
To support multi-modal action proposals, take advantage of the designs of diffusion policy CChi et al., (2033) and its following studies (Ze et al., 2024| Yan et al. 2024 Tie et al. 2025), which have demonstrated modeling
distributions from only a few successful demonstrations. To empower the policy with adaptive manipulation abilities, while previous studies only collect optimal success trajectories (without any failures during the manipulation) for training, we introduce trajectories including failure actions and the recovery and adaptation actions from failures for training, as the failure actions help in revealing the accurate mechanisms and manipulation policy. Figure I showcases the superiority of our proposed adaptive policy learning method. The pure visual observation could not tell whether the door was locked or not. The static policy that only takes the passive one- me-ffamm visual input will randomly propose one of the multi-moda trajectory candidates. On the contrary, the adaptive policy will first try pulling the door, and then adapt the policy distribution from multi-modal to single-model accordingly, as its training data include the
2

---

Published as a conference paper at ICLR 2025
failure of direct pulling trial on the locked safe door, and then adaptively turning the key to open the safe after the failure successfully.
Based our proposed novel environment, have conducted extensive experiments on 9 categories of 277 different objects, covering 5 types mechanisms, showcasing the necessity of the proposed environment and dataset, and the effectiveness of the proposed policy learning framework in efficiently and intelligently adapting the manipulation.
In summary, contributions include:
We study the novel problem of adaptively manipulating articulated objects with diverse mechanisms and build an environment with various categories of objects and mechanisms,
We propose novel framework that learns the adaptive manipulation policy for various mechanisms from diverse demonstrations.
Extensive experiments have demonstrated the significance of our proposed environment, and the effectiveness of the proposed adaptive policy learning framework.
2 RELATED WORK
2.1 ARTICULATED OBJECT ENVIRONMENTS AND DATASETS
To facilitate the study of representation and manipulation of diverse and complex articulated ob- jects, DoorGym (Urakami et al. 2019). door manipulation environment first introduced with diverse doors. UniDoorManip (Li et al. 2024b) further empowers door environments with different mechanisms. PartNet-Mobility dataset first introduces multiple categories of articulated objects from PartNet (Mo et al. 2019; Chang et al. 2015), integrated with the sapien environment (Xiang et al., 2020; Mu et al. 2021| Gu et al. 2023) to support various articulated object manipulation tasks. Further, GAPartNet (Geng al. (023b) provides fine-grained part annotations, AKB-4 (Liu et al.)
2022) provides roal-world articulated object models, and Arnold (Gong et al. 2023) provides the environment for language- guided manipulation.
2.2
ARTICULATED OBJECT MANIPULATION
There have been a series of studies studying articulated object manipulation. Where2Act (Mo et 2021) first studies the point-level affordance for short-term manipulation, with affordance- hased (Wu al. 2022), flow-hased (Eisner et al. 2022 Zhang et al.. 2023), part-based (Geng al.) 2023a and
rl based (Geng et al.) 2023c) methods study the long- hong-horizon manipulation. Environment-AWe Aware Affordance (Wu et al. 2023a Li et al. 2024a) further studies the manipulation with environment
constraints. 2here2Explow
(Nin al. 2023) and AdaAfford (Wang et al. 2022) study converting
passive visual priors to manipulation posteriors using the few-shot interactions, which tackle the problem of exploring novel articulated object categories with novel geometries and parts, and manipulation on ambiguous kinematics and dynamics. Besides, coarse-to-fine method (Ling et al., 2024) studies the sim2real framework for real-world manipulation, and language-guided methods (Xu et al. 2024 Gong et al., 2023) explore the manipulation with language guidance. While these works mainly investigated the manipulation with simple mechanisms (such as directly opening door or safe), in our work, we further study the policy for manipulating articulated objects with diverse and complex mechanisms, with a novel proposed environment supporting such objects.
3 ADAPTIVE MANIPULATION ENVIRONMENT
Previous datasets and simulation environments for articulated objects often lack diversity and realistic manipulation mechanisms (Urakami et al., 2019 et al., 2024b Geng et al.) 2023b) Xiang et al.) 2020 Geng et al. 2023a). To address this issue, we developed new environment to better explore complex mechanisms in articulated object manipulation and learn adaptive manipulation policies. Based on IsaacGym (Makoviychuk et al.) 2021), this environment simulates these mechanisms and includes 9 categories objects (Section 3.1 with 5 types of adaptive mechanisms (Section 3.2).
3.1
ARTICULATED OBJECT DATASET
In recent years, several works have proposed datasets for articulated object manipulation. PartNet- Mobility (Xiang et al. 2020) and AKB-48 (Liu al. 2022) offer diverse datasets for articulated

---

Published as a conference paper at ICLR 2025
Table Statisties of our adaptive articulated object dataset, including categories of 277 different instances. CM. PC. respectively denote Coffee Maker and Pressure Cooker.
Category
Bottle
Pen
C.M
Window
Door
Lamp
Microwave
Safe
PC.
Instance
32
36
18
30
25
37
36
Table 2: Adaptive manipulation mechanism comparison between our environment and others.
Environment
Lock
+ #Clockwise
Rotate&Slide
Push/Rotate
Switch Contact
GAPartNet
x
X
X
PartManip
/
x
EXX
X
X
EXX
DoorGym
UniDoorManip
/
X
X
Ours
/
objects but focus on cross-category geometry diversity, neglecting the mechanisms of different parts and articulations needed to achieve the final manipulation goal. For instance, PartNettMMobility includes the safe category, but the door can be directly opened without rotating the knob to unlock the latch. GAPartNNet (Geng al. 2023b) provides fine-gras line-grained part annotations but still fails to model complex manipulation mechanisms. DoorGym (Urakami et al., 2019) claims a large- scale, scalable dataset specifically for door manipulation, considering the latch mechanism of doors. UniDoorManip (Li et al., 2024b) enriches the diversity of door geometry by composing instances. However, both DoorGym and UniDoorManip are limited door manipulation and do cover more diverse and long-term mechanisms.
To investigate real-world articulated object manipulation, we introduce new dataset that encom- passes more realistic adaptive manipulation mechanisms. Our dataset includes categories of 277 objects: Bottle, Pen, Coffee Maker, Window, Pressure Cooker, Lamp. Door, Safe, and Mi- crowave. Table provides detailed statistics of our dataset, and Figure v instances of each category. The object assets our dataset are handcrafted from materials primarily obtained from Warehouse (Trimble). More details can be found in Appendix B
visualizes
3.2 ADAPTIVE MANIPULATION MECHANISM
Most existing articulated object environments focus primarily geometric diversity across different categories of objects. While these objects may contain multiple parts, manipulating one part typically does impact other parts' state or joint limits, resulting in simplified manipulation mechanisms. Common actions in these environments include pushing or pulling part, such as opening a drawer or pressing button, which can be deduced purely from visual observation. However, real-world manipulation often depends on internal joint states that are not visible externally, necessitating adaptive manipulation policies based on feedback.
To better simulate real-world articulated object manipulation as well as corresponding mechanisms, we have identified five adaptive mechanisms that enhance the fidelity our environment:
Common in everyday objects like doors or safes, the lock mechanism requires an initial action such as rotating a key or knob pressing button unlock the object before it can be opened. This mechanism tracks the key part's joint state during manipulation and updates the lock state accordingly.
Lock Mechanism:
lock state transitions to "unlock". the door joint limit lifted to allow opening; otherwise, the door remains locked. Since the lock state cannot be inferred visually, the robot must interact with the object to determine the lock state and adapt its policy accordingly.
When rotating knob, cap, handle, the direction (i.e., clockwise or counterclockwise) is determined by the internal revolute joint limit, Our environment randomly assigns the rotation direction upon initialization, preventing visual inference of the direction. The robot must attempt one direction and switch if unsuccessful.
Random Rotation Direction:
This mechanism requires part to be rotated to specific angle before it can be lifted or pulled out, such as lifting the lid a pressure cooker. The required rotation angle is visually discernible, necessitating the robot rotate the part incrementally and attempt to slide it
Rotate & Slide Mechanism:
4

---

Published as a conference paper at ICLR 2025
Bottle
Coffee Maker
Window
Pressure Cooker
Lamp
Door
Safe
Microwave
Articulated Object Datasets with Diverse Adaptive Mechanisms Geometries
Open Bottle
Open Pressure Cooker
Open Window
Selected Mechanisms
Open Microwave
Adaptive manipulation dataset and environments. Bottle and Pressure Cooker feature the Rotate & Slide mechanism, requiring continued rotation after failed lift. The window includes the Lock and Random Rotation Direction mechanisms, necessitating exploration of the correct rotation direction to unlock the latch. Microwave incorporates the Lock and Switch Contact mechanisms, where the robot must first pull the handle to check the lock state and press the button if locked.
Figure 2:
determine if the correct angle reached. We randomize the revolute joint limit and initially set the prismatic joint limit to zero, lifting it once the correct angle is achieved.
Push/Rotate Mechanism: Due to the similar appearance of buttons and knobs, is unclear whether the part should be pushed or rotated. For example, lamp might be turned by pushing button in some instances and rotating knob others. Our environment includes both revolute and prismatic joints for the same part in the URDF file and randomly determines whether the part should be pushed or rotated, adjusting the joint limit accordingly.
Switch Contact Mechanism: Based on the lock mechanism, this requires the robot to manipulate different key parts, such as a handle and knob, if they are separate. For instance, in a safe, the robot must switch contact points during manipulation due to the lock state ambiguity, preventing it from determining the sequence contact points at the outset.
Table 2 compares the richness and diversity of mechanisms between environment and others. We visualize the adaptive manipulation mechanisms in Figure 2 and Figure More details can be found in Appendix C.2 and D
METHOD
As illustrated in Figure 3 for various mechanisms from collected adaptive demonstrations. To achieve this, leverage the annotated part poses in our dataset to generate expert manipulation trajectories in the environments, considering invisible internal states ensure the trajectories adaptive. Next, model the expert trajectory distribution with high multi-modality, we employ 3D visual diffusion-basel imitation learning (Ze al. 2024 Chi et 2023), which learns the gradient of the action score function generate actions.
we propose a novel framework that learns an adaptive manipulation policy
4.1 ADAPTIVE DEMONSTRATION COLLECTION
Our goal is to generate adaptive demonstrations that are optimal under partial observation. For instance, when opening microwave, the expert adaptive policy initially pulls the handle check the lock state. the latch is locked, the policy will push the button before opening the door. the latch not locked, the policy continues pulling the handle open the door. In contrast, static policy
5

---

Published as a conference paper at ICLR 2025
Adaptive Demonstration Collection
Locked
Initial State
Pull Door
Push Button
Open Door
Unlocked
3D Adaptive Manipulation Diffusion Policy
Pointnet++
3D Features
Proprioceptions
Initial
Multi-modal Prior learned from Adaptive Demo
Observation
History Condition
Previous Actions
Condition
N (O,
Denoising
M
N(O, Denoising
Pusterior Update learned Adaptive Demo
Figure 3: Adaptive demonstration collection: Given the uncertain lock state of microwave, we instruct the robot to first pull the door to check if locked, and then follow two different trajectories based on the result. Diffusson- based 3D adaptive manipulation policy: Conditioning of 3D visual features, proprioceptions, and actions, the policy denoises Gaussian noise into the trajectory distribution. Initially, the policy captures the bimodal distribution in the demonstration based on the initial observation. As the observed lock state is determined, the policy distribution adaptively shifts an unimodal distribution.
the history
with full observation would know the lock state in advance and could directly open the door or push the button without first trying pull the handle.
design rule-based expert adaptive policies to gather adaptive demonstrations in our simulation environments. We start by computing the bounding box of the part mesh and annotating the part pose through an interactive script. Using these annotations, we label the sequences of parts and manipulation actions for each category, ensuring optimal trajectories under partial observation. For example, the manipulation sequence for a locked safe grasping the handle, pulling the door, grasping the knob, rotating the knob, grasping the handle, and opening the door. Note that the recorded trajectories are end effector poses instead of the high-level action labels. More details are provided in Appendix C With these policies, we collect adaptive demonstrations of robot motion trajectories for all 9 categories of objects.
4.2 3D INFFUSIONBASER ADAPTIVE POLICY LEARNING
Given the collected demonstration dataset D ((or, az)), we aim to learn a policy that models the conditional distribution Ot, Ad). Here A, refers to the predicted action sequence A, (oc,...,4+42,, where T, is the action horizon. O₂ refers to the observation history, including 3D
point clouds and proprioception states, O = (o.-To..... ot), where T, is the history horizon. Å refers to action history, A, (a. (ic--T-110 ag az-1).
However, conducting imitation learning on D challenging due to its multi-modal nature: The ambiguity of the internal states of articulated objects results in multiple successful manipulation trajectories under the same visual observation. Thanks to recent progress in diffusion- based meth- ods (Chi al. 2023 Ze et al. 2024; Ke al. 2024). we can better fit the multi-modal distribution by learning the action score function.
Following the Diffusion Policy (Chi et al. 2023) utilize DDPM (Ho et al. 2020) to estimate the conditional distribution P(A, At). The DDPM scheduler performs K iterations of denoising

---

Published as a conference paper at ICLR 2025
steps to transform Gaussian noise A into noise-free action AQ. This process adheres to Eq. where Cr "N. and 7 are parameters determined by the scheduler. This reverse process conditions the action prediction on both observations and previous actions, differing from the vanilla implementation of the diffusion policy.
- Ork -A Or, k)) +0((0,
The noise prediction network Eg is trained by minimizing the loss function in Eq 2 which effectively minimizes the variational lower bound of the KL-divergence between the data distribution and the sample distribution drawn from DDPM:
||e* co co OO, Ag, k)
(()) = E No-et0.100),
(2)
In practice, the observation O₂ consists of an observed third-view partial point cloud and the robot's proprioception state (e.g., end-effector pose, robot joint angles, and velocities). The point cloud is first cropped and downsampled by Farthest Point Sampling (FPS) and then encoded by PointNet++ (Qi et al. 2017). The action de is the next robot end-effector goal pose. Notably, we find that employing 6D rotation representation (Ke al. (024) for the end-effector pose action stabilizes the training process. During execution, the policy only executes a sub-sequence of the predicted actions, as the adaptive manipulation process requires high-frequency adjustments.
Lamp
Pen & Coffee Machine
Door
Figure 4: Adaptive Environments and Qualitative Manipulation Results. This figure shows the manipulation results of object categories apart from Figure
5
EXPERIMENTS
5.1
SETTINGS AND METRIC
We conduct experiments in the category level covering all the 9 object categories, and collect 20 adaptive manipulation demonstrations for each object as the training data. For evaluation metric, success rate of manipulations. To evaluate what kind of adaptive demonstrations is the most beneficial to adaptive policy learning, we train adaptive policies on adaptive demonstrations with different numbers of adaptive trials.
7

---

Published as a conference paper at ICLR 2025
VAT-MART
AdaAfford
Ousswo
Ours
Figure 5: Manipulation Trajectories Proposed by Our Method and Others. Our method can se- quentially propose stable and accurate adaptive actions, while others have their respective drawbacks.
Table Success Rates of Different Methods. Our method outperforms baseline methods and the ablated version in all categories.
Task
Adaptive Manipulation
Method
VAT-
41.41=17744
45.00:112.04
18.33.1 18.33
71.43 14.29
1.25.1088
6.0015.100
13.75.16.73
12.20 34 19.38
AdaAfford
42.86=14.29 10.00=11..3
15.00
61.43=22.18
16.35
77.00= 17.85
11.25::11.25
52.86 18.13
Sampling
17.37:49.72 24.00=#11.14
26.67=
38.57=16.96
15.71=11877 25.00=10:25 18.57:12286
11.25=8.75
28 13.75
ACT
75.79:18.55
DP3
83.16::12.19
83.00:111.77
86.67.1.088
10.10
35.71
12.00.13.22
70.95.1..37
13.20 11
58.75:1 14.84
Ours w/o adaptive 14.19 30.00:110.00
85.00
91.42= 42 38.75-22050
38.00
74.774:2811
56.25 13.98
24.292200000
Ours
95.07:19.70 99.05-10.81 98.33-4.99
97.14:5.71
61.25:133.28
100.00:00.0
94.33:22.38
88.75:
10.38 82.53-214.72
5.2 BASELINES AND ABLATION
To demonstrate the superiority our proposed method and components in manipulating articulated objects with different mechanisms, compare it with state-of-the- art affordance- based methods, offline imitation learning methods, a sampling- based method, and an ablation of our method.
VAT-Mart (Wu et al. 2022). affordance-based (Zhao al. 2022 Wu al. 2023b) method that predicts the open-loop trajectories from the one-frame passive observation.
AdaAfford (Wang et al. 2022), adaptive method that adjusts the manipulation policy based on history interactions using CVAE (Sohn et al.) 2015).
Sampling, planning- based method that samples macro action from a discrete set at each high-level time step then plans to the sub-goal pose associated with the selected maero action computed based on the object part pose annotation.
ACT (Zhao al. 2023), an imitation learning method that uses Action Chunking with Transformers and CVAE.
DP3 (Ze et al.
2024 Diffusion Policy extended with 3D visual representations.
Ours w/o adaptive, replace the adaptive demonstration data with static demonstration data that is optimal under full observation, maintaining the diffusion baffus imitation learning.
8

---

Published as a conference paper at ICLR 2025
Table 4: Effects of Repeated Adaptive Trials in Adaptive Demonstration for Bottle. While adaptive trials are necessary for learning the adaptive policy, repeated adaptive trials make it more difficult model the adaptive manipulation distribution.
Trials
0
l(ours)
3
4
5
Success Rate 0.8714
0.9364
0.7571
0.6571
0.5714
0.4143
0.4428
5.3 SIMULATION RESULTS
Table and Figure respectively show the success rates and manipulation trajectories of different methods. Figure further shows more manipulation trajectories proposed by our method. Our method outperforms all other methods and demonstrates stable and accurate action trajectories for adaptive manipulation. From the visualizations, we can observe that, for VAT-Mart, an open-loop method, it is difficult to fit the whole trajectory space and directly predict manipulation trajectory time, and the open-loop method does not support adapting the policy from previously executed actions. Using the diffusion-basee -baee imitation method, our method can more accurately model the poses of actions using the limited number of data (such advantage is also demonstrated other diffusion-based imitation studies (Chi et al., 2023 Ze al., 2024)). In contrast, AdaAfford suffers from the inferior mudti-modal distribution modeling capability CVAE and the increased training data requirements associated with point-level affordance learning. The Sampling- based method performs worse than AdaManip and other baselines because its sampling process because its sampling process cannot efficiently leverage priors posteriors learned from demonstrations. While ACT and DP3 outperform sarrdance--bssed methods, they fall short of AdaManip due to the absence of an adaptive demonstration collection pipeline. Additionally, ACT is further limited by its lack of the robust multi-modality modeling capability provided by diffusion models.
Ours w/o adaptive is not trained from demonstrations with recovery actions from failure trials, so the learned policy could not adapt from failure actions to finally achieve the goal. As illustrated in Figure 5 when the safe is locked and Ours w/o adaptive attempts to open the door, it fails to switch to rotating the knob and instead continues on the opening trajectory.
Table shows the effects of repeated adaptive trials adaptive demonstrations for training. When only using optimal successful actions (i.e., O adaptive trials) under full observation, the model is the same with Ours w/o adaptive and could not have the adaptation capability. When using more than one adaptive trial at the same object state, these adaptive trials are redundant and increase the complexity of distributions to model, while not increasing the scenarios the policy can handle. Therefore, the performance decreases when the number of repeated adaptive trials increases from 1. These results validate our demonstration collection design, which limits the manipulation sequence only adaptive trial.
Table 5: Real-world Evaluation Results.
Tasks
Open Bottle
Open Microwave
Open Safe
Open Pressure Cooker
Success
8/10
7/10
5/10
5/10
5.4 REAL-WORLD EXPERIMENTS
To validate the generalization of our adaptive diffusion policy to real-worke scenarios, we conduct experiments on various real-world objects like Pressure Cooker, Microwave, Bottle, and Safe. For the real-world settings, we employ Franka Emika Panda Robot Arm as our agent. To capture visual observation, position an Azure Kinect DK camera adjacent to the robot arm. Our policy takes the real-time point clouds from the depth camera, the robot state from the robot arm, and previous actions as the input, and generates the corresponding end-effector action in a close-loop fashion. We collected 35 adaptive expert demonstrations for each object by human teleoperation to train the policies, and conducted 10 evaluation trials per object.
Table 5 presents the number of successful executions across four real-world tasks. The results in Figure demonstrate that adaptive policy can be effectively applied to real-world scenarios. To

---

Published as a conference paper at ICLR 2025
better illustrate this adaptive behavior, visualize different trajectories under different object states in Figure 7 The red trajectory shows that the robot finds out that the microwave is locked after failed to pull the door and turned to push the button to unlock the door. Conversely. if the microwave initially unlocked, the robot will continue to open the door after it grasps the handle, as depicted by the blue trajectory in sub-figure 2 of Figure? 7 For more visualizations, please Appendix D
Open Pressure Cooker
Open Microavee
OpenSafe Figure 6: Manipulation Trajectories of Real-World Scenarios.
Open Bottle
Grasp Handle
Locked: Pull Door Failure Unlocked: Open Door
Push Button
Open Door
Pull Door
Grasp Handle
Figure 7: Visualization of adaptive policy of Open Microwave in the real world. The trajectories vary depending the state of the microwave. The red trajectory represents the executed trajectory when the microwave locked. subfigure 2, the blue trajectory illustrates the robot's ' action when the mierowave is unlocked, prompting continue opening the door. The red trajectory shows that the robot failed to pull the locked door and turned to push the button.
CONCLUSION
We study the problem of adaptive manipulation policy for manipulating articulated objects with diverse and complex mechanisms, build environments with different categories of such objects that support the various manipulation mechanisms, and propose novel framework that learns the adaptive manipulation policy for various mechanisms from diverse adaptive demonstrations based on diffusion policy. The significance of our proposed environment and the effectiveness of the proposed adaptive policy learning framework have been demonstrated by our experiments.
Our paper represents the initial study into the environment suitable for adaptive manipulation policy learning. Currently, our dataset encompasses categories with a total of 277 objects, which we plan to expand by introducing more categories and instances to cover increasingly complex and realistic mechanisms. Additionally, incorporating deformable object manipulation into adaptive tasks represents significant direction for future research. Moving forward, the AdaManip environment will progressively include a broader spectrum of real adaptive manipulation tasks, making it comprehensive platform for both training and testing adaptive manipulation policies.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This project is supported by the National Science and Technology Major Project (2022ZD01 14904). the National Natural Science Foundation China (No. 62376006), the National Youth Talent Support Program (820080008) 1), the National Natural Science Foundation of China (No. 62136001) and NSFC-6247070125.
REFERENCES
Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese. Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, L. Yi, and Fisher
Yu. Shapenet: An information-rich 3d model repository. ArXiv, abs/1512.03012, 2015. URL
https://aag sew.nt httesss/aaa..cmaatttccch........ www.aarrrrroooo.rooraaae
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric A. Cousineau, Benjamin Burchfiel, and Shuran
Song. Diffusion policy: Visuomotor policy learning via action diffusion. ArXiv, 2a//223333011111 2023. /tLt::tss://et
http:/ap....aaaaaammm semant www.aaaatnnncccoco..ooooooooooee wtt.aaacaa.a...oo/ooorraaaaaaaaa
Yushi Du, Ruihai Wu, Yan Shen, and Hao Dong. Learning part motion of articulated objects using spatially continuous neural implicit representations. In British Machine Vision Conference (BMVC), November 2023.
Ben Eisner, Harry Zhang, and David Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. arXiv preprint arXiv:2205.cm 04432,, 2022.
Haoran Geng, Ziming Li, Yiran Geng, Jiayi Chen, Hao Dong, and He Wang. Partmanip: Learning cross-category generalizable part manipulation policy from point cloud observations. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2978-2988. 2023a.
Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. Gapartnet: Cross-category domain-generalin generalizable object perception and manipulation via generalizable and actionable parts. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7081-7091 2023b.
Yiran Geng, Boshi An, Haoran Geng, Yuanpei Chen, Yaodong Yang, and Hao Dong. End-to-end affordance learning for robotic manipulation. ICRA, 2023c.
Ran Gong, Jiangyong Huang, Yizhou Zhao, Haoran Geng, Xiaofeng Gao, Qingyang Wu, Wensi Ai, Ziheng Zhou, Demetri Terzopoulos, Song-Chun Zhu, al. Arnold: A benchmark for language- grounded task learning with continuous states in realistic 3d scenes. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), 2023.
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskil12: A unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302.000 240.00469, 2023.
Nick Heppert, Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Rares Andrei Ambrus, Jeannette Bohg, Abhinav Valada, and Thomas Kollar. Carto: Category and joint agnostic recon- struction of articulated objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 21201-21210, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Tsung-Wee Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint 240. 2024.
Jiahui Lei, Congyue Deng, William B Shen, Leonidas Guibas, and Kostas Daniilidis. Nap: Neural 3d articulated object prior. Advances in Neural Information Processing Systems, 36, 2024.

---

Published as a conference paper at ICLR 2025
Yu Li, Kai Cheng, Ruihai Wu, Yan Shen, Kaichen Zhou, and Hao Dong. Mobileafford: Mobile robotic manipulation through differentiable affordance learning. In 2nd Workshop on Mobile Manipulation and Embodied Intelligence at ICRA 2024, 2024a. URL URU/attps///o/eeeeiiee net met/forunt i d= (6sc69sfr)
Yu Li, Xiaojie Zhang, Ruihai Wu, Zilong Zhang, Yiran Geng, Hao Dong, and Zhaofeng He. Unidoor- manip: Learning universal door manipulation policy over large-scale and diverse door manipulation environments. arXiv preprint Ttf::24003022000 2024b.
Suhan Ling, Yian Wang, Shiguang Wu, Yuzheng Zhuang, Tianyi Xu, Yu Li, Chang Liu, and Hao Dong. Articulated object manipulation with coarse-to-fine affordance for mitigating the effect of point cloud noise. ICRA, 2024.
Liu Liu, Wenqiang Xu, Haoyuan Fu, Sucheng Qian, Qiaojun Yu, Yang Han, and Cewu Lu. Akb-48: A real-world articulated object knowledge base. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14809-14818, 2022.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv: 108.. 10470, 2021.
Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In The IEEE Conference on Computer Vision and Pattern Recognition (C (CVPR), June 2019.
Kaichun Mo, Leonidas J. Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2act: From pixels actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2813-6823, October 2021.
Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Cathera Yang. Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track [Round 2), 2021.
Chuanruo Ning, Ruihai Wu, Haoran Lu, Kaichun Mo, and Hao Dong. Where2explore: Few-shot affordance learning for unseen novel categories of articulated objects. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Chenrui Tie, Yue Chen, Ruihai Wu, Boxuan Dong, Zeyi Li, Chongkai Gao, and Hao Dong. Et-seed: Efficient trajectory-level se(3) equivariant diffusion policy. ICLR, 2025.
Trimble. 3dwarchouse. http:choous..tttsss////a/a/aeaee https///ccwweehooseeseetettteeee wwe.
Yusuke Urakami, Alec Hodgkinson, Casey Carlin, Randall Leu, Luca Rigazio, and Pieter Abbeel. Doorgym: A scalable door opening environment and baseline agent. arXiv preprint 01188,, 2019.
Yian Wang, Ruihai Wu, Kaichun Mo, Jiaqi Ke, Qingnan Fan, Leonidas Guibas, and Hao Dong. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. European conference on computer vision (ECCV 2022), 2022.
Fangyin Wei, Rohan Chabra, Lingni Ma, Christoph Lassner, Michael Zollhöfer, Szymon Rusinkiewicz, Chris Sweeney, Richard Newcombe, and Mira Slaveheva. Self-supervised neural ar- ticulated shape and appearance models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15581-15826, 2022.
12

---

Published as a conference paper at ICLR 2025
Ruihai Wu, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, and Hao Dong. mAT-mart: Learning visual action trajectory proposals for manipulating 3d ARTiculated objects. In International Conference Learning Representations, 2022.
www.wweww.. net www.eee//orrrrdddd1a.......
coly
Ruihai Wu, Kai Cheng, Yan Zhao, Chuanruo Ning, Guanqi Zhan, and Hao Dong. Learning
environment-awwan aware affordance for 3d articulated object manipulation under occlusions. In sevethh Conference on Neural Information Processing Systems, 2023a. URL https
Ruihai Wu, Chuanruo Ning, and Hao Dong. Learning foresightful dense visual affordance for
deformable object manipulation. In IEEE International Conference Computer Vision (ICCV), 2023b.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang. Yifu Yuan, He Wang, al. Sapien: simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11097-11107, 2020.
Ran Xu, Yan Shen, Xiaoqi Li, Ruihai Wu, and Hao Dong. NaturalvIm: Leveraging fine-grained natural language for affordance-guided visual manipulation. arXiv preprint urXiv:2403.08333 2024.
Zhenjia Xu, He Zhanpeng, and Shuran Song. Umpnet: Universal manipulation policy network for articulated objects. IEEE Robotics and Automation Letters, 2022.
Ge Yan, Yuch-Hua Wu, and Xiaolong Wang. Dnact: Diffusion guided multi-task 3d policy learning. arXiv preprint arXiv:2403.04/ 15, 2024.
Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffu- sion policy, ArXiv, abs/2403.03954, 2024. URL httphhtp///// //a:://apppie www.seeeccccccoaa.ooooomm org/
CorpusID:26825530
Harry Zhang, Ben Eisner, and David Held. Flowbot++: Learning generalized articulated objects manipulation via articulation projection. arXiv preprint arXiv:2306. /2893, 2023.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint 13X1v:2304.13705, 2023.
Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang, Qingnan Fan, Kaichun Mo, and Hao Dong. Dualafford: Learning collaborative visual affordance for dual-gripper object manipulation. arXiv preprint arXiv:2207.01977. 2022.
13

---

Published as a conference paper at ICLR 2025
Table Parameters for training and diffusion model
Training
Values
Hardware Configuration
NVIDIA GeForce GTX 4090
Weight Decay
le-6
Batch Size
64
Optimizer
Adam
Learning Rate
le-4
Epochs
500
Time Expense
3h
Model
Values
Pointcloud Size
4096
Backbone
Unet
Observation History Horizon
4
Prediction Horizon
4
Action Horizon
2
EMAModel
True
Diffusion Timestep
100
Noise Scheduler
squaredcos
Action Space
absolute end-effector pose
A HYPER-PARAMETERS FOR EXPERIMENTS
Table summmriizss the hyperparameters for our diffusion policy model and training details.
B DATASET CONSTRUCTION
Apart from object assets collected from existing datasets (Xiang et al. 2020 Li et al. 2024b), most of the object assets in dataset obtained from 3Dwarehouse (Trimble). We dedicate significant time and effort to carefully selecting the available object meshes, segmenting them into distinct parts, re-aligning the object mesh coordinate systems, and subsequently developing Python scripts to facilitate the efficient synthesis of operational dataset instances. Similar to GAPartNet (Geng et al. 2023b), we annotate the object assets with rich and comprehensive labels.
C DETAILED EXPERT POLICY DESIGN FOR ADAPTIVE DEMONSTRATION COLLECTION
C.1 POSE ANNOTATION
To achieve precise part operation, we employ a pose labeling approach for the parts. Initially, we roughly estimate the part pose through the bounding box of the part mesh calculated using the Python library trimesh. Subsequently, we observe the pose annotation results in real-time through an online interactive script to complete the precise annotation of the pose.
C.2 ADAPTIVE MANIPULATION SEQUENCE
Bottle: Grasp the Cap. Rotate Cap after a failed Lift Up. Randomly sample Rotate/Lift if the previous action is Rotate.
Pen: Same Bottle.
Pressure Cooker: Grasp the Handle. Rotate Handle after a failed Lift Up. Randomly sample Rotate/Lift if the previous action is Rotate.
Coffee Maker: Grasp Portafilter. Rotate Portafilter after a failed Pull Down. Randomly sample Rotate/Pull if the previous action is Rotate.
14

---

Published as a conference paper at ICLR 2025
Window:
Grasp the Handle. Randomly choose direction Rotate Handle. If failed, choose the other direction. Rotate Handle after a failed open trial. Randomly sample Rotate/Open the previous action is Rotate.
Door: Same as Window.
Lamp: Randomly choose to Push/Clockwise Rotate/Counter Clockwise Rotate. Never choose failed action.
Safe: Pull Door. If succeed, then continue opening the door. If failed, Rotate Knob. Randomly choose a direction to Rotate and choose the other one if failed. Then Pull Door again to open it.
Microwave: Pull Door. If succeed, then continue opening the door. If failed, Push Button. Then Pull Door again to open it.
C.3 TRAJECTORY SPARSIFICATION
If the history consists of dense trajectory of the robot's end effector poses, the policy would require a long history context length to capture previous failures. However, training policy with a long history context is challenging, this issue, we chose to sparsify the trajectory. Only key frames of the demonstration trajectories are saved for imitation learning, but the recorded actions are still 6D end effector poses instead of high-level macro actions.
requires more computing resources and is less robust. To mitigate
For example, in the open-safe task, the recorded history includes grasping poses and several ma- nipulation poses while omitting most intermediate steps. The history condition for the policy is as follows: Igrasp the handle, pull the door and fail to open]. Other intermediate poses during execution excluded from the history. Based on this context, the robot predicts the next goal pose [unlock the key]. Once the goal pose predicted, we apply inverse kinematics (IK) to plan the path for execution.
Open Safe
Open Bottle
Open Pressure Cooker
Figure 8: Visualization of Open Safe/Bottle/Pressure Cooker Experiment in the real world. The red trajectory represents the executed trajectory. The blue trajectory indicates the actions under other object states.
D MECHANISM VISUALIZATION AND EXPERIMENTAL RESULTS
Figure 4 illlstrates the operating mechanism of the remaining categories and the experimental results. The mechanism the lamp in the upper left comer is "Push/Rotate". which can be operated by pressing/rotating the button/knob, but only way is correct. The pen and coffee machine in the
15

---

Published as a conference paper at ICLR 2025
upper right corner fall under the "Rotate/Slide". To operate them, you need rotate specific part cap or the coffee machine handle. The door at the bottom belongs to
to a certain angle to open the cap the "Random Rotation Direction" and "Lock", where the handle must be rotated either clockwise or counterclockwise to open the door.
Figure&showk the visualizations of the other experiments in real world apart from Open Microwave shown in Figure?? 7 In these visualizations, the red trajectories represent the paths actually executed by the robot, while the blue trajectories indicate potential paths under different object states, For instance, in Sub-figure of the Open Bottle experiment, the blue trajectory shows that if the cap is rotated sufficiently, the robot can then lift the cap. This visualization helps illustrate the adaptability of the robot's s actions based on the observed state of the objects involved.
16


Reviews:
[{"review_id": "r4beAVJlZr", "reviewer": "Reviewer_eEiz", "summary": "This paper introduces AdaManip, a new environment and method for training robots to manipulate articulated objects with adaptive mechanisms.  Existing datasets often simplify these interactions, allowing objects to be manipulated without accounting for hidden states like locked latches or required rotation angles.  AdaManip addresses this by incorporating five key adaptive mechanisms: lock mechanisms, random rotation direction, rotate & slide, push/rotate, and switch contact.  The environment includes nine object categories (bottles, pens, coffee makers, windows, pressure cookers, lamps, doors, safes, and microwaves) with diverse instances, totaling 277 objects.\n\nThe authors also propose an adaptive demonstration collection pipeline that uses rule-based expert policies to generate trajectories reflecting realistic partial observation scenarios. These demonstrations are then used to train a 3D visual diffusion-based imitation learning model. \n\nExperiments in simulation and real-world settings with a Franka Panda robot demonstrate the effectiveness of AdaManip. The proposed method outperforms baseline approaches like VAT-Mart and AdaAfford, achieving higher success rates across all object categories.  Ablation studies confirm the importance of adaptive demonstrations and the strategy of limiting each demonstration to a single adaptive trial. The real-world experiments further validate the policy's ability to generalize and adapt to real-world objects and conditions.", "questions": "- How is the \"unlock mechanism\" and other similar mechanisms simulated? Is this based on a specific collision shape or on some high-level discrete conditions?\n- How is the real-world demonstration collected?\n- Could you provide the detailed hyperparameter values for the diffusion policy, such as the action horizon and prediction horizon?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper points out an important but under-explored problem: the adaptive manipulation of articulated objects with complex mechanisms. Previous work primarily focused on simpler manipulations, neglecting the realistic challenges of hidden object states.\n- The AdaManip environment and associated dataset are significant contributions. They provide a rich and diverse set of objects, as well as the rule-based demonstrations, enabling research on more realistic manipulation tasks.\n- The categorization of five key adaptive mechanisms (lock mechanisms, random rotation direction, rotate & slide, push/rotate, and switch contact) provides valuable insights and serves as an inspiring framework for creating new tasks.\n- The application of 3D visual diffusion-based imitation learning is a good baseline for this problem.  Diffusion models are well-suited to capturing the multi-modal nature of the task, where multiple valid manipulation trajectories can exist for a given observation.\n- The real-world experiments are valuable, demonstrating the practicality of the approach.\n- The paper is generally well-written and easy to follow.", "weaknesses": "- Relying on rule-based expert policies for demonstration collection can limit the complexity of the learned behaviors. Exploring methods for learning these demonstrations or incorporating human demonstrations could lead to more robust and generalizable policies.\n\n- The source code and dataset have not been released.\n\n- Environmental specifications, including frame rate and API details, are not provided.\n\n- The paper primarily compares against affordance-based methods.  A comparison to other offline imitation learning methods (e.g., [1]) and online learning-from-demo methods (e.g., [2][3][4]) would provide a more complete picture of the method's performance.\n\n[1] Zhao, Tony Z., et al. \"Learning fine-grained bimanual manipulation with low-cost hardware.\" arXiv preprint arXiv:2304.13705 (2023).\n\n[2] Ball, Philip J., et al. \"Efficient online reinforcement learning with offline data.\" International Conference on Machine Learning. PMLR, 2023.\n\n[3] Rajeswaran, Aravind, et al. \"Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.\" arXiv preprint arXiv:1709.10087 (2017).\n\n[4] Ho, Jonathan, and Stefano Ermon. \"Generative adversarial imitation learning.\" Advances in neural information processing systems 29 (2016).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HGYGdxZwji", "reviewer": "Reviewer_14u1", "summary": "The authors present an imitation learning study focusing on tasks involving articulated objects. The study necessitates (1) a multi-modal action proposal and (2) adaptive manipulation based on historical actions.\n\nThe author conducted experiments both in simulation and real-world.", "questions": "Why did you choose point cloud input rather than RGB?\n\nFurther details about the real-world experiments are needed, specifically:\n\n1. What type of low-level controller was used?\n2. To what extent does the training generalize to the test scenarios?\n3. What is the frequency of the policy?\n4. How was the expert demonstration collected? \n-- Understanding the data collection process is crucial. Are 35 demonstrations sufficient to capture the task's multi-modal nature?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 1, "contribution": 3, "strengths": "The motivation of the paper is sound. It is an important problem in imitation learning: (1) multi-modal action proposal and (2) adaptive manipulation from history actions.", "weaknesses": "Could the authors clarify any generalizations observed in the simulation results? Further details on the differences between the training and testing datasets would be helpful.\n\nThe manuscript requires significant revisions, particularly Figure 1 and 3, which is quite confusing. If I understand correctly, you employed a rule-based policy described as \"adaptive\" to collect data. However, it appears that this policy does not incorporate any prior knowledge about the object’s mechanism for the learning phase. Could you clarify this?\n\nThe real-world results presented are not convincing. There is a lack of discussion on generalization, and the evaluation seems overly simplistic with limited data. The methods used for collecting demonstrations in the real world are also unclear. Additionally, it is not specified whether the data collection method used in simulations is scalable to real-world applications.\n\nI would like to see more discussion on history information part. How did you incorporate history? would simply adding history cause overfitting problem?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vc4V6EnEjG", "reviewer": "Reviewer_ooAE", "summary": "In this work, the authors investigate the task of learning manipulation policies for articulated objects. Specifically, they recognize a gap in the literature relating to the intersection between articulation and locking/sequential mechanisms that are often required in real-world articulated object manipulation scenarios (e.g. unlock the door before opening). Their contributions are roughly twofold: first, they design a set of novel articulated object simulation environments that model the locking/sequential mechanisms during articulation tasks; second, they propose a demonstration dataset and policy architecture to learn novel policies to tackle these environments. They demonstrate the effectiveness of their policies on their simulated environments and real-world embodiments of the same.", "questions": "* Why wasn’t 3D Diffusion policy considered as a policy class? https://3d-diffusion-policy.github.io/\n* Why was the decision made to use a logical lock/mechanism for task progression instead of a physical mechanism?\n* How is success rate defined?\n* How many history steps are needed for success across the tasks? Would be good to understand this mechanism a bit more.\n* How is sim2real accomplished? Is a new dataset collected in real setting? Or is it transferred zero-shot.\n\n# Update post-rebuttal\n\nThe authors have done a good job responding to my questions. In particular, it is good to see that the DP3 baseline and their policy baseline are roughly equivalent in performance, matching my expectations. It's also good to see that additional high-quality recovery demonstrations improve success rates. This makes it a stronger paper - although because the policy-level changes are primarily a data contribution rather than a policy contribution, I maintain that my current score of 6 (marginal accept) is warranted.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "# Originality\n\nWhile locking mechanisms and articulated objects have been studied (and modeled) independently, the interface of the two has not yet been particularly well-studied, despite the fact that many articulated objects require navigating locking/latching mechanisms to accomplish opening. Proposing tasks that model this aspect of the task is a good original contribution.\n\nOn the policy side, I can’t fully evaluate whether the design of the policy contains much novelty/originality, as it appears to be a straightforward modification of diffusion policies, but without enough architectural detail provided to assess novelty.\n\n# Quality\n\nThe environment-modeling side of the paper seems high quality, with good diversity over tasks and attention to relevant details for releasing a new benchmark. The proposed policy and its analysis is of OK quality - not enough detail is provided in the experiments section for me to evaluate.\n\n# Clarity\n\nThe paper is clear and well-written (except figures, which I will discuss later).\n\n\n# Significance\n\nThe paper is of moderate significance, as this set of tasks (and an environment which accurately models it) is indeed important to study and is currently underexplored. Difficult to assess the significance of the policy proposal, however.", "weaknesses": "There are 3 primary categories of weakness\n* The policy itself was not fully characterized. Not many details were provided as to the architecture, training procedure, etc. and I was not able to really assess in comparison to existing literature. In particular, the treatment of point clouds and history were glossed over, when in other works it has been shown that these details matter for different kinds of generalization. Additionally, the ablations don’t provide much insight into why the design decisions of the policy lead to improved performance over the baselines - “Ours” without history is much stronger than the baselines.\n* Experimental details: It was unclear whether the baselines were fairly conducted - were the VAT-MART and AdaAfford methods trained on the same data as “Ours”? Or were pretrained weights used?\n* Generalization characterization: when releasing benchmarks with metrics, ideally we would also receive a set of metrics with a clear experimental protocol, as well as what conditions/generalization criteria the experiment is trying to measure. Overall “success rate” for a single kind of evaluation is somewhat coarse. Instead, it would be desirable to have a clear set of different benchmarks, e.g. to measure cross-instance generalization, generalization across initialization/viewpoint, cross-class generalization, etc. \n\n\n\nSmaller weaknesses\n* Figure 1 is too dense, and the symbols unintuitive. More text should be incorporated into the diagram, it’s really hard to figure out what’s going on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VVvM8P5eIz", "reviewer": "Reviewer_74wV", "summary": "This paper proposed a framework that aims to adaptively learn manipulation policy based on previous policy execution results. The framework is based on diffusion policy learning. The manipulation specifically targets articulated object manipulation such as safe and doors. Both simulation and real-world experiments are conducted.", "questions": "The authors can reply to the questions raised in \"Weakness\" section.", "limitations": "", "rating": 1, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The studied problem in this paper is interesting.\n- The paper presentation is good and clear.\n- The experiment results look strong and outperform related baselines.", "weaknesses": "- There isn't much technical novelty in this paper. It seems that the algorithm/model itself does not guarantee adaptive. The only \"adaptive\" component stems from adaptive demonstration collection. How is this \"adaptive\" policy different from vanilla diffusion policies?\n- The experiment shows that the proposed method is better than AdaAfford, which is quite surprising. If what the authors of AdaAfford argued is true, AdaAfford should also be able to solve the problem presented in this paper. Why is AdaAfford \"not so stable due to the capability limit of cVAE\"? Why is the method in this paper stable? How does \"stable\" defined? Is it because diffusion model vs. cVAE, or it's because output actions vs. point cloud affordance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SgbdLx21qp", "reviewer": "Reviewer_XbhV", "summary": "This work presents a new simulation environment and framework for adaptive manipulation of articulated objects with diverse mechanisms. The AdaManip environment includes nine object categories and five types of *adaptive* mechanisms. Using a novel adaptive collection method, the authors generate a dataset to support training for complex, multi-step manipulation tasks. The paper further introduces a 3D diffusion-based imitation learning framework to model multi-modal action distributions, enabling policies that adjust based on previous actions. Well-designed experiments demonstrate results in simulation and real-world scenarios indicating that AdaManip achieves high success rates and outperforms baseline methods in handling complex articulated objects.", "questions": "- Since the policy is a rule-based sequence, how does the proposed method compare with traditional planning methods? Specifically, please provide a brief comparison or discussion of how your method relates to or differs from traditional planning approaches for articulated object manipulation.\n\n- Is diffusion policy the only viable solution? Specifically,  maybe discuss potential alternative approaches and explain why you chose diffusion policy over other methods.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The AdaManip environment addresses a gap in existing robotic manipulation environments by focusing on objects with realistic, multi-step (adaptive) mechanisms, supporting a range of complex manipulation scenarios. This is valuable for advancing robot manipulation learning on real-world objects and increasing robots' learned policies' generalizability.\n\n- Adaptive Demonstration Collection: The method for collecting adaptive demonstration trajectories in a rule-based fashion is well-motivated.\n\n- The use of diffusion models for action distribution in policy learning is innovative and effectively models the multi-modal distribution necessary for tasks where visual cues alone are insufficient.\n\n- Extensive Evaluation: The paper provides detailed simulation and real-world experiments. Performance metrics such as success rates across nine object categories and comparisons with baseline methods, including ablation studies, illustrate the framework's effectiveness.", "weaknesses": "- Although the framework outperforms baselines like VAT-MART and AdaAfford, it would benefit from additional baseline comparisons, especially with RL methods (or more specifically, with dense reward annotation) or DAgger, that may be adaptable to adaptive manipulation tasks, or recent work, FlowBotHD, that incorporates history into flow in order to learn to adapt to past errors using a flow-based policy (which is your sequential rule-based policy but with flow representations).\n\n- More emphasis on how AdaManip differs from environments like PartNet-Mobility in complexity or adaptability would better explain the novelty of AdaManip. For example, you could talk about whether there are particular types of mechanisms or object categories that AdaManip includes that PartNet-Mobility does not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8cUzTq0qAS", "reviewer": "Reviewer_RR8i", "summary": "This paper presents a novel robotic simulation environment and a dataset which targets solving articulated object manipulation tasks with some adaptive mechanisms, adjusting the feedback from failed or successful past actions. The approach uses a 3D diffusion-based behaviour cloning method to model multi-modal action distribution. The experiments show that this method outperforms existing models in both simulated and real-world scenarios, illustrating its potential for real-world applications.", "questions": "1. How does your method provide significant methodological innovation beyond creating a novel environment and conducting real robot experiments?\n2. Why did you choose to use action history instead of observation history, which is more commonly used in reinforcement learning? Have you conducted experiments to compare the effectiveness of these approaches?\n3. How does your method perform compared to sampling-based methods that propose plausible actions and plan over them? Have you evaluated this as a baseline?\n4. Can you elaborate on how your approach scales to new, unseen objects in a zero-shot setting? What is the expected performance when combining multiple proposed mechanisms?\n5. How do you ensure that your method generalizes effectively across different categories of articulated objects?\n6. Could you provide more details on how your method handles potential overfitting, given its reliance on the quality of collected data and the \"learn everything at once\" nature of the diffusion model?\n7. What are the limitations of your adaptive policy learning approach when applied to more complex or real-world scenarios with more diverse mechanisms?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a new simulation setup with more varied and complex object interactions, which improves over existing datasets. This new setup helps robots learn and practice in a way that better matches real-world conditions, making their training and testing more realistic.\n2. The presented experimental results demonstrated the proposed policy outperformed the baseline on a significant level, proving its effectiveness and applicability.\n3. Real-world experiments are promising and impressive.", "weaknesses": "1. The paper's main contributions focus on introducing a new manipulation environment and demonstrating real robot experiments, but it lacks significant innovation in the underlying methodology, raising questions about its fit for this particular conference.\n2. The paper lacks comprehensive ablation experiments to justify design choices. For example, it does not explain why action history was chosen over observation history, which is more common in reinforcement learning research. Additionally, there is no comparison with sampling-based methods that first identify plausible actions and then plan over them.\n3. The paper does not discuss the scalability of the approach or its ability to generalize across different objects in a zero-shot setting. It also fails to explore how the method would handle combinations of the five proposed manipulation mechanisms.\n4. The method relies heavily on the quality of the collected data, and the diffusion model learns everything simultaneously, which could lead to redundancy and potential overfitting to the training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)