Learning to Open and Traverse Doors with a Legged Manipulator
Authors: Mike Zhang
Yuntao Ma
Takahiro Miki
Marco Hutter
Venue: CoRL 2024 (2024)
Keywords: Mobile Manipulation
Legged Manipulator
Reinforcement Learning
Door Opening
TL;DR: We train a legged manipulator robot to robustly open and pass through a variety of doors without knowing crucial properties such as if the door is push or pull a priori.

Abstract: Using doors is a longstanding challenge in robotics and is of significant practical interest in giving robots greater access to human-centric spaces. The task is challenging due to the need for online adaptation to varying door properties and precise control in manipulating the door panel and navigating through the confined doorway. To address this, we propose a learning-based controller for a legged manipulator to open and traverse through doors. The controller is trained using a teacher-student approach in simulation to learn robust task behaviors as well as estimate crucial door properties during the interaction. Unlike previous works, our approach is a single control policy that can handle both push and pull doors through learned behaviour which infers the opening direction during deployment without prior knowledge. The policy was deployed on the ANYmal legged robot with an arm and achieved a success rate of 95.0% in repeated trials conducted in an experimental setting. Additional experiments validate the policy's effectiveness and robustness to various doors and disturbances. A video overview of the method and experiments is provided in the supplementary material.

Learning to Open and Traverse Doors with a Legged Manipulator
Mike Zhang, Yuntao Ma, Takahiro Miki, and Marco Hutter Robotic Systems Lab, ETH Zurich, Switzerland
Figure 1:
We present a control policy that can open and pass through both pull (top) and push (bottom) doors by estimating the door properties during deployment.
Using doors is a longstanding challenge in robotics and is of significant practical interest in giving robots greater access to human-centric spaces. The task is challenging due to the need for online adaptation to varying door properties and precise control in manipulating the door panel and navigating through the confined doorway. To address this, we propose a learning-based controller for legged manipulator to open and traverse through doors. The controller is trained using a seacher-student approach in simulation to learn robust task behaviors as well as estimate crucial door properties during the interaction. Unlike previous works, our approach is a single control policy that can handle both push and pull doors through learned behaviour which infers the opening direction during deployment without prior knowledge. The policy was deployed on the ANYmal legged robot with an and achieved a success experimental setting. Additional experiments validate the policy's 's effectiveness and robustness to various doors and disturbances. A video overview of the method and experiments can be found at youtu. be/ be/tQDZXN_koBB _KNNU
Abstract:
of 95.0% in repeated trials conducted in
Keywords: Mobile Manipulation, Legged Manipulator, Reinforcement Learning
1 Introduction
Legged manipulator robots offer significant potential in combining the ability of legged robots to navigate varied environments with the potential for rich interactions afforded by the manipulator. An essential skill for these robots is the autonomous opening and traversal of doors, significantly expanding their reach in human-centered environments. While door opening seems a quotidian task, it poses challenge for control, especially for high degree-of-frreedmm system such as a legged manipulator. Moreover, doors can vary in ways that are not immediately observable, such as in their spring stiffness and whether they are push or pull. The task becomes more difficult when the robot must also pass through the door as must make additional decisions such as when to disengage from the handle or if it needs to hold the door open against the door's self-closing mechanism.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Existing methods for robot door opening have largely only considered being robust to properties that vary continuously such as the mass, spring resistance, or friction [1, 2,2,3), 3]. However, for determining the crucial property of the door opening direction, existing methods have relied on some hardcoded prior by either having user provide the opening direction [4] or using pre-programmed routines [5]. These approaches limit the autonomy and adaptability of the robot in unknown environments. Ide- ally, the controller is "plug-and-play" meaning it does not rely any priors and opens the door given only measurements of the door location by inferring the opening direction during the task, similar to how a person would open door they have not encountered before. In moving towards this ideal, this paper proposes learning-based legged manipulator control policy trained using the popular teacher-student approach [6] that can open and traverse through doors of varying properties via estimating these properties during deployment.
Reinforcement Learning (RL) with domain randomization is used to train the teacher policy as approaches such imitation learning model-based control are difficult to scale to doors of varying properties. The student policy is then trained to imitate the teacher and estimate the door properties using only the information available during deployment. To the best of our knowledge, our approach the first monolithic control policy that can handle both push and pull doors without being given this information a priori or relying on any hardcoded routines to determine the door opening direction.
Our main contributions are summarized as:
A control policy that can open and traverse through doors of varying characteristics (e.g. opening direction, dimensions, dynamical properties), given only the robot proprioception and the loca-
tions of the handle and doorway.
Real-world hardware experiments validating the policy's 's efficacy on different doors. Analysis of the policy's s ability to estimate the door properties from the task interaction.
2 Related Work
2.1 Model-Based Door Opening
Several works studied door opening for manipulator on wheeled base by tracking predefined reference velocities and forces [7, 8, 9, 10]. For legged manipulators, Model Predictive Control (MPC) has been applied for door opening [11, 2, 12]. MPC controllers rely on tracking reference trajectory, which necessitates using a planner and requires knowing the properties of the door such as its opening direction and precise dimensions. As such, the practical applicability of such controllers to unseen doors is limited. The commercially available Spot robot with the arm manipulator includes a door opening controller. While its details are not public, it is likely a model-based controller. According to available documentation [4], Spot's controller must be provided information priori such the door's opening and swing directions.
2.2 Learning-Basee Door Opening Control
RL is a natural approach for door opening as specifying the desired behavior through rewards for opening doors intuitive. Several works used door opening as a benchmark task for RL algo- rithms [13, 14, 15]. Urakami et al. [1] developed the DoorGym simulation environment for RL door opening with a specific focus on robustness and generalization via domain randomization and fixed-based manipulator. Schwarke et al. [3] trained and deployed an RL door opening policy on a wheeled-legged robot. However, these works produce control policies that can only handle a single opening direction as they treat push and pull doors as separate RL tasks. The works above have focused on opening the door and ignor- ing the subsequent task of passing through. Ito et al. [16] demonstrated a learning- approach for opening and passing through doors with a wheeled base mobile manipulator. Their approach combines separate control modules for handling opening and passing through. However, the robot leams its motions from human demonstrations via teleoperation which is difficult to scale to differ- ent doors. Moreover, the method requires two separate modules for handling push and pull doors.
demonstrated their trained policies in hardware experiments

---

Teacher Policy
Student Policy
RL Loss
Propriacaption
Propriacaption
MUP
MLP
MLP
RNN
Concat
Privileged Info
Door Exterocaption
Noise
Imitation
joint states
mass
taiquee
type
Estimaliin
Decodar
Exteroception
Door
Losses
Figure 2: Overview of the training method. Dashed lines indicate the flow of gradients from the losses. The teacher policy is first trained using RL operating on a privileged set of observations. The student policy operates only the observations available during deployment and is trained to imitate the teacher's behavior while also estimating the privileged information.
More recently, Kang et al. [5] combined RL and position-force controllers open and pass through both push and pull doors with wheeled base mobile manipulator. However, the RL controller is only used for push doors as the authors could not train a single RL policy to handle both opening directions. The method also relies on a hardcoded push-pul procedure after turning the handle to determine the door's opening direction.
2.3
Teacher-Student Distillation
Recent successes applying teacher-student training to legged robot locomotion [17, 18] have demonstrated remarkable robustness and the student policy's ability estimate properties of the environment. For example, in the locomotion task, the student policy can infer the height of the local terrain, even in the presence of corrupted exteroceptive sensor measurements by leveraging the knowledge of the feet's position. For door opening, the information necessary to complete the task can only be acquired through direct interaction with the door (e.g. push or pull). Given this, the teacher-student approach could be particularly beneficial for this task.
Method
An overview of our method is shown in Fig. 2. Both teacher and student policies are trained in a simulation environment Isaac Gym [19] as shown in Fig. 3. The robot modeled in simulation and used in real-world ex- periments is mal with an arm manipulator [20].
3.0.1 Door Model
We focus on hinged doors that consist single door panel with handle for unlocking the door. The door has two degrees of freedom corresponding the hinge and handle angles 0 and & respectively. To manipu- late the handle, the robot is equipped with hook end- effector. The handle must be turned to unlatch and open the door. Pretensioned spring resistances that act against opening the door and turning the handle are modeled as constant torques. The effect of a door closer system installed to regulate the speed of door closure was modeled as a damp- ing torque on the door hinge that scales linearly with the hinge velocity. We included additional damping that scales quadratically with the hinge velocity account for air resistance. The training environment contains four door types from the set (pull, push} (right, left). Where right and left denote whether the hinge is the right or left side of the door panel.
Figure 3: Overview of the training environ- ment in simulation. A hook end-effector is used for grasping the door handle.

---

3.0.2 Policy Actions
The policy directly controls the robot's arms and commands lower-level locomotion policy that governs the legs. We use the learned locomotion policy from Ma al. [21] which takes as input the planar linear velocities Ur and Uy. angular velocity about in the robot base frame, and the applied wrench on the robot base from the arm and its motion. Instead of estimating the applied wrench, we found it sufficient and simpler to set it to a fixed mean value. To prevent the policy from moving the base too fast, we clip the linear and angular velocity commands to the locomotion controller to maximum magnitudes of 0.5 m/s and I rads/s respectively. Conversely, commands below 0.1 m/s and 0.1 rads/s are to zero. The policy controls the arm by setting joint angle targets for the arm's joint-level PD) controllers. The commanded target angle for joint i is computed as
clip(sa; + di qi of./ / 45 + aT./ aT:/Kp)
where a; is the action, Ø is default position, and qi is the joint position for joint i. Finally, is an action scaling factor. For safety, the commanded target for joint i clipped based on a proxy for the actuator torque threshold computed from the joint's proportional gain K,, torque limit 7,, and saturation parameter a. We a to 0.7 for all arm joints. This action clipping applied during training and the real robot.
3.0.3 Rim- Considerations
The following were implemented the simulation training environment facilitate policy transfer onto the real robot. Detailed domain randomization parameters reported in the Appendix.
Randomize Initial Robot Configuration: The robot base starts an episode in a random location and yaw angle in front of the door with a random initial base velocity. The initial arm configuration is not randomized to avoid self-colliding configurations.
Randomize Arm Joint PD Gains: The PD gains for each arm joint resampled for each episode.
Randomize Door Dimensions: We generate door models while randomizing the handle locations on the door panel, the doorway width, and the door panel thickness.
Randomize Door Dynamics Properties: Resistance torques and the damping coefficients at the han- dle and hinge, the door mass, and the maximum handle turning angle resampled for each episode.
Zero Handle Contact Friction: Many door handles have slippery surfaces which are especially hard to turn with a hook end-efficctor. We address this by zeroing the friction coefficient of the handle and hook end-effector contact simulation. With non-zero handle contact friction, the policy tended to learn more aggressive behaviours that relied on the end-effector momentum to turn the handle.
3.1 Teacher Training
The teacher trained an Actor-Critic using Proximal Policy Optimization [22]. We separate the task into stages. The first is to approach and open the door and the second is pass through the door. This is done as the rewards are simpler to define when considering each stage in isolation.
3.1.1 Observations
The teacher observation includes measurements that are available to the robot during deployment along with privileged information only accessible in simulation. No noise is introduced to the teacher observation. The measurements available during deployment are the robot's proprioception and ex- teroceptive measurements the location of the doorway and handle relative to the robot. Propri- oception includes the base orientation, base linear and angular velocities, arm joint positions and velocities, and the previous step actions. The privileged observations include the door joint (hinge and handle) positions and velocities, the mass of the door panel, the applied torques at the door hinge and handle due to spring stiffness and damping, the door type, and task stage observation denoting if the policy is currently in the stage of opening passing through the door.

---

3.1.2 Rewards
The opening and passing stage rewards are denoted by ro and rp respectively. ro is further decomposed into rewards for handle manipulation, Plom, and reward for opening the door to target angle, Tod- Thm com- prises reward terms for moving the end-effector to the handle, grasping the handle, and turning the handle. The policy does not need to interact with the handle once the door has been unlocked and opened enough. As such, we Tim to its maximum value such that the policy ignores it once the door is opened enough. We considered the door opened enough at 30".
22
1
Figure 4: Additional rewards are used for pull doors to encourage the robot to move around the door panel. When the robot hase in 2, it will receive reward rz, and in 22 it will receive reward rz, + ---- These rewards are also given based on the end-effector location.
The opening stage transitions to the passing stage when the door has been opened more than 70% After transitioning to the passing stage, the policy receives the maximum possible value of ro in addition to rp-- This is done the policy can pursue Tp without trad- ing off ro as otherwise the policy can refuse to transition.
The passing stage reward r is given by the dot product of the base velocity va with the unit progress vector p. Before the robot has passed through the doorway, p the vector from the base to the doorway center. After the robot has passed through, p is the vector along the direction of the doorway. This dot product is normalized by the maximum commandable velocity of the locomotion controller || Tp is clipped to maximum of I, otherwise, the policy can learn undesired behaviors such as throwing its arm to move the base faster than [| max-
For pull doors, Tp alone insufficient for the policy to learn to move around the open door panel. Therefore, once pull door opened enough in the opening stage, we reward it for moving its base and end-effector around the door panel as shown in Fig. 4. We also require the base and end-effector to be behind the door panel to transition to the passing stage for pull doors.
Shaping rewards r, are used to regularize the behavior such that the policy respects the hardware limitations of the robot and is safe to deploy. r, is applied during both the open and passing stages and comprises terms such as avoiding collisions, minimizing arm motions, penalizing unwanted base motions, penalizing out-of-limit commands, and penalizing singular arm configurations.
Detailed definitions and scales of all reward terms are given in the Appendix.
3.2 Student Training
The student policy is trained to imitate the teacher policy's actions given only the proprioceptive and exteroceptive door observations that available during deployment. Gaussian noise is added to all of the student's 's observations. The student policy also receives supervision through estimating the privileged information of the door and handle locations relative to the robot, the door joint states, the door mass, the door hinge and handle torques, and the door type. The Smooth L1 Loss is used for losses except for the door type where the Cross Entropy Loss is used.
The student policy based on a recurrent neural network (RNN) with an architecture that follows the student policy for legged locomotion from Miki al. [18] with some differences. Specifically, the attention-gate decoder in the original architecture is replaced with a linear layer the decoder. This was done as unlike locomotion where it is possible to do blind, we assume that the exteroceptive measurements during door opening can be noisy but not degraded to the point of being useless.
We do not include all available measurements in the student policy observation similar Tan et al. [23]. Specifically, the previous actions and arm joint velocities are omitted. Training the student policy with much higher noise the arm joint velocity resulted in better transfer of the policy to the real robot. Given this, we completely removed the arm joint velocity from the observation.
5

---

irt
pushie
(a) Pull Right Door
(b) Pull Left Door
(c) Push Right Door
(d) Push Left Door
Figure 5: Real-world experiments of the policy deployed the real robot, traversing through doors of varying swing (left/right) and opening (push/pull) directions. The policy's estimated probability for each door type over time plotted below the corresponding experiment. The true door type is plotted as solid line while others are plotted as dashed lines.
In dceor spe 200
pull right
pull left:
0.6
10
commendee velocity
0.6
0.2
0.0
-15
-10
10 20
25
-044
Figure UMAP projection of the policy's hid- den state over trajectories collected in simula- tion with pull right and push right doors. All hidden state trajectories start at the central clus- ter and progress in separate directions depend- ing whether the door is push or pull.
0.0
0.5
1.0
2.0 time [s]
2.5
3.0
3.5
Figure 6: Correlation of the policy's estimated door type and its actions. At 2.7 seconds, the policy shifts its belief from pull to push and ac- cordingly commands the base forwards to push.
4 Results
The student policy is deployed directly onto the robot without further fine-tuning. The policy is both trained and deployed at 50Hz. We tasked the robot with opening and traversing through doors varying in their opening and swing directions, doorway and handle dimensions, door panel inertia, handle dynamics, and the presence
door self-closing mechanism.
Proprioceptive observations are provided by sensors onboard the robot, such encoders measuring the joint states. The robot also runs onboard lidar odometry [24] to estimate its base pose. For exteroception, the policy only needs to know the handle and doorway locations relative to the robot base. We provide these door measurements using either motion capture or AprilTags [25] as an external tracking system. The motion capture system provides accurate measurements but requires several fixed tracking cameras where as the AprilTags can be easily deployed on any door tracking tag the door panel with external camera. As the door measurements relative to the robot, they could also be obtained using onboard sensing, but leave this future work.

---

0.30
0.30
whh coor type estimation loss
RMN
0.25
whhout doer type estimatten loos
0.25
MLP
mnkaaio loe
0.20
Inkazien lose
0.20
6.15
6.15
250
500
750
1000
1250
1500
1750
2000
250
500
750
1000
1250
1500
1750
2000
Comparing imitation loss when training the student policy with and without the door type estimation loss.
Figure 8:
Figure 9: Comparing imitation loss when training student policy with and without a re- current module.
4.1 A Single Control Policy for All Door Types
We evaluate whether the policy can recognize the door type (opening and swing directions) during the task and open them without being given this information beforehand. Indeed our policy could successfully deduce the correct type and pass through for all door types as shown in Fig. 5. When the policy was uncertain of the opening direction, it shifted its estimate between push and pull. This shift of the policy's 's belief was correlated to its commanded actions as shown Fig. 6, where the policy learned to push and pull until it could move the door panel. We also study how the policy's belief of the door type is represented in its RNN hidden state by visualizing the hidden states trajectories using a UMAP [26] projection in Fig. 7. The figure shows that the policy learned to represent the opening direction (push/pull) in separate regions of the hidden state space. Moreover, the regions for the opening directions appear linearly separable in the projection.
4.2 Repeatability of the Control Policy
Repeated trials of the policy were conducted to evaluate its repeatability. A spring-loaded door was placed within the motion capture space and 20 continuous trials were performed for both the pull and push sides. The policy successfully traversed the door in 20/20 and 18/20 trials for the pull and push sides respectively, with an overall success rate of 95.0%. In all trials, the policy was successfully able to open the door. The two failed trials on the push side were caused by the robot getting stuck on the side of the doorway which was protruding due lack of walls around the door which differs from the simulation model. A video of the trials is provided in the supplementary material
4.3 Ablations in Simulation
We ablated the student policy training to study if supervision from the estimation task helps in learning the control task and if the recurrence of the student policy is necessary.
Door Type Estimation Loss: The student policy is trained without supervision from estimating the door type and we find that the imitation loss alone is sufficient for learning to distinguish between both push and pull doors. Moreover, removing the estimation loss has little effect on the training dynamics of the imitation loss as seen in Fig. 8, but its presence neither helps nor hurts the imitation learning. Given this, the estimation loss and decoder module are still included in final policy architecture they are useful for helping to understand the internal belief state of the policy.
Student Policy Recurrence: We replaced the RNN module of the student policy with an MLP and trained the policy with the imitation loss to study the effect of removing recurrence. Fig. shows that the student policy fails to imitate the teacher policy without recurrence. Specifically, the MLP student policy cannot imitate the teacher given the noisy student observations. As the MLP oper- ates on single-step observation, any noise the observation makes it difficult for the policy to disambiguate different states of the system and to take the appropriate action.
7

---

Chair blocking the door.
(b) Human pushing on the door.
Rope pulling on the robot.
Figure 10: Robustness of the policy to external disturbances unmodelled in the training environment.
5
Discussion
5.1 Emergent Robustness to Unmodeled Disturbances
The student policy showed robust recovery from disturbances not explicitly modeled during training in simulation. For example, the policy could handle pushing or pulling on the door or robot as seen in Fig. 10, even though random pushes/pulls on the robot were not used during training. can be expected that the teacher policy would be robust such disturbances it acts the current observation without regard to the history. As the student does not perfectly imitate the teacher during training, mistakes in the student's actions during training provide opportunities to learn recovery behaviours from the teacher. For example, if the student's action missed the handle grasp, the subsequent teacher's actions to imitate would be to retry grasping the handle.
5.2 Failure Cases
common failure case of the policy was not turning the handle enough to unlock the door. When this failure occurs, the policy becomes stuck moving the robot back and forth trying to push and pull on the door as it shifts its belief of the door type. The policy does not have direct observation which indicates that the handle has been turned. As such, can learn to incorrectly associate the robot's handle turning motion with the handle turning. Using visual or force sensing, which can directly observe the turning of the handle, could alleviate this issue.
The performance of the policy was notably degraded when using AprilTags track the door due to characteristics of the AprilTag measurement noise being unmodeled in simulation when training the policy. We note a failure case caused by the AprilTag latency where the policy lets go of the handle and attempts to grasp it again as it receives delayed measurements of the handle location.
Lastly, unmodeled door geometry in simulation causes failure cases such as the hook end-effector catching the door panel the robot base getting stuck the protruding sides of doorway as shown in the videos provided in the supplementary material.
Conclusion
This paper presented learned control policy for legged manipulator traverse through doors. The policy was trained in simulation using teacher-student approach such that it learned both robust behaviors and the ability to estimate properties of the door through interaction during the task. The latter allowed the policy to handle doors of varying properties, such as the opening direction, without being given this information a priori. The policy was deployed on the ANYmal robot with an arm and achieved a 95.0% success rate in trials on the push and pull sides of a spring-loaded door. Additional experiments demonstrated the policy traversing through doors of all opening and swing directions, various dimensions, and various hinge and handle dynamics, well as robustness and recovery from external disturbances. Future works include using onboard sensing for obtaining the door measurements, adding force sensing, and handling larger set of door handles such knobs.
8

---

Acknowledgments
We would like thank Eris Sako, Jan Preisig, Jia-Ruei Chiu, Kaixian Qu, and Turcan Tuna for help- ing with experiments and Andrei Cramariuc for feedback the paper. This work supported by the Max Planck ETH Center for Learning Systems, Intel, and NCCR digital fabrication. Moreover, this work was conducted as part of ANYmal Research, a community to advance legged robotics.
References
[1] Y. Urakami, A. Hodgkinson, C. Carlin, R. Leu, L. Rigazio, and P. Abbeel. Doorgym: A scalable door opening environment and baseline agent. NeurIPS 2019 Deep Reinforcement Learning Workshop, 2019.
[2] E. Arcari, V. Minniti, A. Scampicchio, A. Carron, Farshidian, M. Hutter, and M. N. Zeilinger. Bayesian multi-task learning mpe for robotic mobile manipulation. IEEE Robotics and Automation Letters, (00:3222-3229) 2023. doi:10. 100..110990....22...2222
[3] C. Schwarke, V. Klemm, M. Van der Boon, M. Bjelonic, and M. Hutter. Curiosity-driven learning of joint locomotion and manipulation tasks. In Proceedings of The 7th Conference Robot Learning, volume 229, pages 2594-2610. PMLR, 2023.
[4] Boston Dynamics Support. Door manipulation with the spot https://support. bostondynamica cs. http:///rrtcceee/eeooooo -it
[5] G. Kang, H. Scong, D. Lee, and D. H. Shim. A versatile door opening system with mobile manipulator through adaptive position-force control and reinforcement learning. arXiv preprint arXiv:2307.0444 04422 2023.
[6] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl. Learning by cheating. In Conference on Robot Learning, pages 66-75. PMLR, 2020.
[7] W. Meeussen, M. Wise, S. Glaser, S. Chitta, C. McGann, P. Mihelich, E. Marder-Eppsteinn, M. Muja, Eruhimov, T. Foote, J. Hsu, R. B. Rusu, B. Marthi, G. Bradski, K. Konolige, B. Gerkey, and E. Berger. Autonomous door opening and plugging with a personal robot. In 2010 IEEE International Conference on Robotics and Automation (ICRA), pages 729-736, 2010. doi:10. 10. 100/ROBOT.20105500000
[8] Y. Karayiannidis, C. Smith, F. E. Viña, Ogren, and D. Kragic. "open sesame!" adaptive force/velocity control for opening unknown doors. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4040-4047, 2012.
[9] H. Gao, C. Ma, L. Ding, H. Yu, K. Xia, H. Xing, and Z. Deng. Dynamic modeling and experimental validation of door-opening process by a mobile manipulator. IEEE Access, 7: 80916-80927, 2019. 100.1111000/ACCSS.00000....
[10] M. Stuede, K. Nuelle, S. Tappe, and T. Ortmaier, Door opening and traversal with an industrial cartesian impedance controlled mobile robot. 20/9 International Conference on Robotics and Automation (ICRA), pages 966-972. IEEE, 2019.
[11] J.RR. Chiu, J.PP. Sleiman, M. Mittal, Farshidian, and M. Hutter. collision-free mpc for whole-body dynamic locomotion and manipulation. In 2022 International Conference on Roboties and Automation (ICRA), pages 9812280.
2022. 10f1100109//CCRA4669990000
[12] J.-P. Sleiman, Farshidian, and M. Hutter. Versatile multicontact planning and control for legged loco-manipulation. Science Robotics, x 2023.
[13] S. Gu, Holly, Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipula- tion with asynchronous off- off-policy updates. In 20/7 IEEE international conference robotics and automation (ICRA), pages 3389-3396. IEEE, 2017.

---

[14] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. In Proceedings of Robotics: Science and Systems, Pittsburgh, Pennsylvania, June 2018. 88 10::10.15607/RS 200.RSSS.201..... 201....049..
[15] M. Sharma, J. Liang. J. Zhao, A. LaGrassa, and O. Kroemer. Learning to compose hierarchi- cal object-centric controllers for robotic manipulation. 2020 Conference on Robot Learning (CoRL), 2020. 2020.
[16] H. Ito, K. Yamamoto, H. Mori, and T. Ogata. Efficient multitask learning with embodied predictive model for door opening and entry with whole-body control. Science Robotics, (65)xeaxx8177, 2022.
[17] Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47))eabc5986, 2020.
[18] Miki, J. Lee, Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, :(62))aa 2022.
[19] V. Makoviychuk, I.. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simula- tion for robot learning, 2021.
[20] C. D. Bellicoso, K. Krämer, M. Stäuble, D. Sako, F. Jenelten, M. Bjelonic, and M. Hutter. Alma articulated locomotion and manipulation for a torque-controllable robot. In 2019 In- ternational Conference on Robotics and Automation (ICRA), pages 8477-8483, 2019. doi: 10.1109/ICRA.2019 299997.......3....
[21] Y. Ma, F. Farshidian, T. Miki, J. Lee, and M. Hutter. Combining learning- locomotion policy with model-based manipulation for legged mobile manipulators. IEEE Robotics and Automation Letters, 7(2):2377-2384, 2022. doi: 100.0.1101LRA222223443335
[22] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
[23] Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim- to-real: Learning agile locomotion for quadruped robots. arXiv preprint 2018.
[24] Khattak, Nguyen, F. Mascarich, T. Dang, and K. Alexis. Complementary multi-modal sensor fusion for resilient robot pose estimation in subterranean environments. 2020 Inter- national Conference Unmanned Aircraft Systems (ICUAS), pages 1024-1029, 2020. doi:
[25] Wang and E. Olson. AprilTag 2: Efficient and robust fiducial detection. In Proceedings of the IEFE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 2016.
[26] L. McInnes, J. Healy, N. Saul, and L. Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018.
10

---

Appendix
A Reward Definitions
A.1 Opening Rewards
The opening reward ro is composed of the reward for opening the door the target angle rod along with rewards for manipulating the door handle denoted by Thm- For pull doors, we additionally include reward for encouraging the robot to move its base and end-effector around the door panel denoted by Tadp- These rewards together compose the opening reward
To 3r + Thm
30
Film + 0.55rdd otherwise
When the door has been opened enough, set as 0 > 30,, Thm is set to its maximum value Fimm as it no longer necessary for the policy to interact with the handle to open the door further. radp is only applied once the door has been opened enough.
The handle manipulation reward is composed of the following components
Tho Tehd Tth Teho + 0.5 Tplg
All individual reward terms in r, are defined as follows.
Tehd (end-effector to handle): Minimizes the distance between the end-effector point e and the handle point h:
Tehd = exp(-lle hll2)
Tth (turn handle): Rewards increasing the handle turning angle 6:
@max
where @max the maximum the handle be turned.
Teho (end-effector grasp orientation): Rewards the end-effector for tracking desired orientation for grasping the handle.
Teho
where e, angular error between the end-effector orientation and the desired end-effector orienta- tion.
Thg (handle in end-effector grasp): Give a binary reward when the handle point h is within the grasp zone G of the end-effector.
(g( |le - h1|2 < 0
Thg
otherwise
(x) is the indicator function of value if x E A and otherwise. For the hook-end effector used in this work, we defined the grasp zone G as the region along the opening of the hook. This reward is only active when the end-effector point e close enough the handle (within m).
Tpg (penalize lost grasp): Give binary penalty when if the handle point the grasp zone step 1 and leaves the grasp zone
{
1gghhhhl 1g
|e e- 1111 1
Ppg
0
otherwise
Similar to Thg- Fplg is only active when the end-effector is close enough to the handle.
Rod (open door to target angle): Rewards opening the door to the target opening angle 0
0
Tod
where 0 is the door hinge joint angle. This reward can also be used train opening only policy that opens the door to 0. For training the opening and passing through policy 0 is set to 75".

---

Tadp (move around the door panel): This reward is only applied for pull doors.
Given zones 2 and 22 defined relative to the door panel shown Fig Tadp computed based on the locations of the base b and the end-effector
1 b E 2
c 21
Tadp
2 b E 22
E Z2
0
otherwise
otherwise
A.2 Passing Rewards
Cp (passing progress): Rewards the base velocity for moving along the unit progress vector
p VV
rp
max
1,
[Vvllmax
where ||v is the the max allowable commanded velocity of the locomotion controller.
A.3 Shaping Rewards
The shaping reward r, is defined as
T, 0.3r ma + 0.5r pbt + Tssa + 0.1 pr
Individual terms of r, are defined as:
Tma (minimize arm motion): Rewards minimizing the arm joint velocities and accelerations
exp(0.01 + cx000.000001
ma
where ÿ and C are the joint velocity and acceleration for the hth arm joint respectively.
Too (penalize base tilt): Penalizes large tilt the robot base. The base tilt angle ai can be com- puted from the projected gravity vector expressed in the robot base frame gi and expressed the world frame gw follows
gw gB
arccos
gw gs
Then Tpbe 1f > 3. where & tilt threshold, and 0 otherwise. We set as 8 Tea (penalize stretched arm): Penalize the from reaching too far prevent singular arm configurations.
e s (0.7 - 0.1) 0,11
Tpsa
-clip
0.1
where and are the locations of the end-effector and shoulder joint.
Teo (penalize command out of limits); As the arm PD target and locomotion commands clipped within certain bounds we penalize the policy for commands that exceed these bounds.
|asl a. 0,
Ppl Eeii
where a, ā, and o; corresponding to the action, action limit, and penalty ramp up speed for the tth component of the policy's output action. The action limits are discussed in Sec. 3.0.2.
pe (penalize collisions): Penalizes robot collisions.
{
Aell otherwise
Tpc
CEC
where A()) is the contact force robot link (-) and C is the set of robot links where collisions penalized including the base, thighs, and arm.
12

---

B Domain Randomization Parameters
duult
domner
The initial robot location relative the door is determined at the beginning of each episode by sampling dwalt and deenter.
Figure 1I:
The following randomizations are resampled for each new episode:
Initial Base Location: Set relative to the doorway by the distances dwolt and doenter as shown in
Fig. 11. dwall and deenter are sampled uniformly from [1,2] m and [-2, 2] m respectively.
Initial Base Yaw: We define a yaw of 09 as the robot facing forwards along the direction of the doorway. The initial yaw is sampled uniformly from [-180, 180)
Initial Base Velocity: The initial base velocity components and by are sampled uniformly from [-0.5, 0.5] m/s.
Door Panel Mass: Sampled uniformly from [15, 75] kg.
Door Hinge Resistance Torque: Sampled uniformly from [0, 30) Nm, set to 0 with probability 0.2.
Door Handle Resistance Torque: Sampled uniformly from [0, 3] Nm, set to with probability 0.2.
Door Hinge Damping Torques: The hinge damping torque comprises of the air resistance given by Kgr0² and the door closer mechanism damping given by K&-0 We sample Kÿ uniformly from [0, 4] Nms For most doors, the door closer's damping is tuned prevent the door from closing too quickly. To model this, we set Kde to be some multiple a of the hinge resistance torque, where a is sampled uniformly from [1.5, 3J s. The hinge damping torque is set to with probability 0.4.
Maximum Handle Turning Angle: Sampled uniformly from [15,90]*
Arm Joint Proportional Gain: Sampled uniformly from [40, 60).
Arm Joint Damping Gain: Sampled uniformly from [3, 6].
We generated door models with different dimensions
that are loaded into the simulation during initialization. The randomized door dimensions shown in Fig. 12.
hg
dw: Sampled uniformly from [0.8, 1.0] m.
ho
dr: Sampled uniformly from [0.02, 0.06] m.
h Sampled uniformly from [0.08, 0.12] m.
hu
hh: Sampled uniformly from [0.7, 1.3] m.
ho: Sampled uniformly from [0.03, 0.12] m.
Figure 12: Randomized dimensions of the
door and handle.
13

---

Additional Experiments
C.1 Teacher-Student Training Necessary?
To evaluate the necessity of the teacher-student framework for this task trained modified teacher policies see if a policy trained with RL directly the observations available the robot can learn the task. The first is given only the student observations without access to the privileged observations. The second is the same as the first except that noise is added to the observations at the same amount used for training the student. We evaluated the teacher policies in simulation on 4000 environments (a different randomized door each) for 10 episodes, each lasting 10 seconds, with all domain randomizations active. The success rates are reported Fig.
Opened Enough
Passed Through
100
success rate [%]
75
25
0
pull
push
pull
push
nominal teacher
w/ student observations
w/ noisy student observations
Figure 13: Comparing the nominal teacher policy against variants with only the student observations. The performance is reported as success rates on whether the policy could open the door enough and it could pass through the door. We consider the policy to have opened the door enough if the door was opened by least 30 within episode. The policy is considered to have passed through the door if the robot reaches position 0.5 meters behind the door wall.
We find that without the privileged observations, the teacher policy could not learn to pass-through for both opening directions, but surprisingly able to learn an opening behavior that can sometimes open both push and pull doors, albeit at a significantly worse success rate. Examining this behavior qualitatively suggests that without the door type privileged information, the policy learns to associate specific robot configurations with either push pull, learning move between these configurations until the door opens. Adding noise the observations further reduces the success rate for this type of behavior as disambiguating these specific robot configurations becomes more difficult.
14

---

C.2 Evaluating the Student Policy's Capabilities
The capability of the student policy is evaluated simulation by tasking it with traversing through doors of increasing hinge resistance torques. Increasing the hinge resistance makes the doors more difficult open as the robot must apply more force before the door moves, and more difficult to pass through the door closes faster after opening. The student policy tested whether it could open the door and if it could successfully pass through in episodes lasting 10 seconds with all domain randomizations active. We report these results in Fig.14.
opened door
passed through door
100
success rate (96)
50
25
pull
push
0
10
20
30
40
50
60
10
20
30
40
50
60
door hinge resistance [Nm]
door hinge resistance [Nm]
Figure 14: Evaluating the capability of the student policy on doors of increasing hinge resistance torques. The performance is reported as success rates on whether the policy could open the door enough and it could pass through the door. We consider the policy to have opened the door enough the door was opened by least 30 within an episode. The policy considered through the door if the robot reaches position 0.5 meters behind the door wall. During training, the policy did not experience hinge resistance torques greater than 30 Nm (highlighted in red).
have passed
The teacher and student policies were trained with a hinge resistance randomization range of 0 Nm Nm. As expected, the success rates for both opening and passing through the door decreases for increasing hinge resistance. For the doors with greater hinge resistances, the policy could still sometimes open them but passing through is more difficult with the success rate dropping zero for hinge resistances greater than 50 Nm. For push doors, the higher resistance makes the door panel more difficult for the robot to hold open and the robot can also get trapped in the door way by the door panel. For pull doors, the higher resistance causes the door to close faster, making it more difficult for the policy to move the robot around the door panel in time to pass through.
15


Reviews:
[{"review_id": "rgLkhNtc0Z", "reviewer": "Reviewer_26YG", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper addresses the problem of opening doors with a legged manipulator. The proposed method uses the student teacher framework to train the policy and demonstrates the effectiveness of the method with zero shot sim2real transfer.  Different reward functions are used for the two tasks of opening the door and passing through it along with other rewards for locomotion. Ablation studies are provided to show how the different input parameters and the chosen architecture impacts the performance.", "summary_of_recommendation": "The paper is very clear and contributions are novel.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "The reviewer thinks the paper is clear and well written.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1qkZAREyzc", "reviewer": "Reviewer_wfP1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present a Teacher-Student reinforcement learning framework to learn a single policy capable of opening different types of doors (left/right opening, push/pull). A key component that sets the paper apart from existing literature is that the robot not only opens the door but the same policy also guides the robot past the open door. The results are also evaluated on real world with impressive results. This is achieved through careful reward design in simulation and domain randomization which transfers to real robot well.", "summary_of_recommendation": "Accept", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1) At what frequency was the policy executed? Was it the same in simulation and real world. In general, more details on efforts to transfer the policy might be helpful. Was domain randomization sufficient? \n2) From what I understand, the 95% success rate reported was using the motion capture system for the door exteroception? Please correct me if Im wrong. The authors also mention that the performance degrades when using AprilTags, how much worse was the performance?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fW0hYTwcMF", "reviewer": "Reviewer_AGnv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper aims to address the challenging and long-standing problem of mobile manipulation, specifically door opening. The proposed approach employs teacher-student distillation policy training, enabling the robot to learn an open-and-traverse policy using reinforcement learning (RL) without human demonstrations. The learned policy is deployed and evaluated on real robot hardware (i.e., ANYmal), demonstrating its effectiveness across various door-opening tasks. It shows generalized capabilities for different opening directions (push and pull), door dimensions, masses, and other dynamic properties.", "summary_of_recommendation": "The paper is highly engineering-driven. While it does not introduce significant novelty from a research perspective, it demonstrates significant improvements in the robot's hardware capabilities.", "technical_quality": 5, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. The reviewer is curious about the circumstances under which the learned policy works best. For example, does it perform better when pulling or pushing doors, and what is the maximum door weight the robot can handle? Could these conclusions be supported by statistical analysis, such as conducting the same set of experiments multiple times or even in simulation?\n2. Can the author further justify the need for the teacher-student policy training framework through an ablation study?\n3. What is the reason behind the hardware design choice of a hook instead of a two-finger gripper?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for a legged robot with an arm that can traverse through a wide variety of doors. The reviewers were highly favorable. The method is still somewhat application specific (opening doors) but the results are an excellent contribution to the robot learning community.

Strengths:
- The policy can result in a legged robot opening doors and moving through the open door.
- There are substantial real world experiments.

Weaknesses:
- It requires careful reward design which might be hard to generalize to other scenarios.
- Further justification of student-teacher training could help for this particular pipeline. Also the technique itself is not novel as pointed out in the paper.
Decision: Accept