Curiosity-Driven Learning of Joint Locomotion and Manipulation Tasks
Authors: Clemens Schwarke
Victor Klemm
Matthijs van der Boon
Marko Bjelonic
Marco Hutter
Venue: CoRL 2023 Poster (2023)
Keywords: Curiosity
Reinforcement Learning
Wheeled-Legged Robots

Abstract: Learning complex locomotion and manipulation tasks presents significant challenges, often requiring extensive engineering of, e.g., reward functions or curricula to provide meaningful feedback to the Reinforcement Learning (RL) algorithm. This paper proposes an intrinsically motivated RL approach to reduce task-specific engineering. The desired task is encoded in a single sparse reward, i.e., a reward of “+1” is given if the task is achieved. Intrinsic motivation enables learning by guiding exploration toward the sparse reward signal. Specifically, we adapt the idea of Random Network Distillation (RND) to the robotics domain to learn holistic motion control policies involving simultaneous locomotion and manipulation. We investigate opening doors as an exemplary task for robotic ap- plications. A second task involving package manipulation from a table to a bin highlights the generalization capabilities of the presented approach. Finally, the resulting RL policies are executed in real-world experiments on a wheeled-legged robot in biped mode. We experienced no failure in our experiments, which consisted of opening push doors (over 15 times in a row) and manipulating packages (over 5 times in a row).

Curiosity-Driven Learning of Joint Locomotion and Manipulation Tasks
Clemens Schwarke," Victor Klemm," Matthijs van der Boon, Marko Bjelonic, and Marco Hutter Robotic Systems Lab, ETH Zurich, Switzerland {cschwarke, vklemm, mvander, markob, Bahutter]@etha. ch
Learning complex locomotion and manipulation tasks presents signif- icant challenges, often requiring extensive engineering of, e.g., reward functions or curricula to provide meaningful feedback to the Reinforcement Learning (RL) algorithm. This paper proposes an intrinsically motivated RL approach to reduce task-specific engineering. The desired task is encoded in a single sparse reward, i.e., reward of "+1" is given if the task is achieved. Intrinsic motivation enables learning by guiding exploration toward the sparse reward signal. Specifically, we adapt the idea of Random Network Distillation (RND) to the robotics domain to learn holistic motion control policies involving simultancous locomotion and manipulation. We investigate opening doors an exemplary task for robotic ap- plications. A second task involving package manipulation from table to a bin highlights the generalization capabilities of the presented approach. Finally, the resulting RL polícies are executed in real-world experiments on a wheeled-legged robot in biped mode. We experienced no failure in our experiments, which con- sisted of opening push doors (over 15 times in row) and manipulating packages (over 5 times in row).
Abstract:
Keywords: Curiosity, Reinforcement Learning, Wheeled-Legger Robots
Introduction
Recent advancements in Reinforcement Learn- ing (RL) have propelled legged and wheeled- legged rohots beyond the confines of the lab, empowering them to fulfill a broad range of practical applications [1, 2, 3, 4, 5, 6]. Nev- ertheless, tasks in environments designed for human interaction remain challenging as they frequently require coordination between loco- motion and manipulation. Research often aims to reduce task complexity by manually dividing the problem into multiple stages [7]. However, this approach lacks the exploration of holistic solutions that consider the interplay between different stages. Solving these problems end- to-end with minimal task-specific engineering, i.e., without including variety of dense reward terms that need to be tuned extensively, remains an open challenge.
Figure 1: A wheeled-legged robot picking up a package and opening door while balanc- ing on two wheels. Motions can be seen at https:/yooutu.be/obb2k IdLuw.
Shared first authorship.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

This work proposes an RL approach that solves the task as whole with little need for task mnsk-specific engineering. We adopt a sparse reward setting, rewarding "+1" when the desired task is achieved. Although simple its formulation, the discovery of sparse rewards in general challenging and requires strategies for the exploration of the environment. Because the reward only exists a small fraction of the state space, random exploration induced by, e.g., epsilon-greede policies is not sufficient pick up the reward signal in reasonable time [8]. In our work, intrinsic motivation is employed to guide the agent. For the first time, the concept of Random Network Distillation (RND) [9], which models intrinsic motivation, has been successfully applied and validated in real-world robotic task, as shown in Fig. 1. Our main contributions can be summarized as follows:
1. A curiosity-driven sparse reward RL approach for learning end-to-end locomotion and ma- nipulation tasks without laborious, task-specific engineering (Section 3)
2. The notion of curiosity states as guiding mechanisms, allowing focus curiosity on directly observable states, while maintaining simple deployment (Section 2.2)
3. An analysis of emerging learning behaviors (Section 4.1)
4. Successful real-world validation of RI policies on wheeled-legged robot in biped mode [10], by repeatedly opening a door and manipulating a package (Section 4.2)
1.1 Mechanisms to Guide Exploration
We group existing approaches that employ guided exploration to improve learning performance over random exploration into three main categories. Expert Demonstrations can be an effective tool to teach a desired skill [11, 12, 10, 13], but require predominantly hand-crafted demonstrations. Cur- riculum Learning [14, 15, 16, 17, 18] involves gradually increasing task difficulty during training, but the generation and efficient scheduling of intermediate tasks are still considered unsolved and subject ongoing research [17, 19, 20]. Intrinsic Motivation, more specifically curiosity, denotes the ability to learn without external rewards for the pure sake of knowledge gain. Integrating this mechanism into learning algorithms holds great promise, particularly due to its task-agnostic nature, which distinguishes it from expert demonstrations or curriculum learning.
Past research has explored three main approaches to modeling intrinsic motivation. In an initial endeavor incorporate curiosity into an RL algorithm, [21] introduced a reward based on the Eu- clidean distance between the prediction of a model learning the environmental forward dynamics and the observed transition, effectively rewarding surprise. The concept of measuring novelty using prediction error from a dynamics model raises a significant concern: it tends to favor stochastic tran- sitions [22], as they are difficult or even impossible to predict accurately. To address this challenge, [23] adopts a different approach by predicting a state embedding instead of the complete world state. Another line of work focuses on estimating and rewarding learning progress within specific re- gions of the state space [24, 25, 26]. However, measuring learning progress high dimensional continuous state spaces remains computationally infeasible [23]. Count-based exploration methods work by rewarding state novelty directly by keeping track of the number of visits for each state and prioritizing less visited states [27]. To tackle exploration in continuous state spaces, [28] proposes pseudo-counts, generalization of count- based exploration methods. Burda et al. [9] propose RND, a method based on predicting information about the current state to measure state novelty. They train a prediction model alongside the RL agent in a supervised fashion, improving its accuracy for visited states during the learning process. The prediction error can then be utilized as an intrinsic reward signal, where familiar states yield more accurate predictions compared less visited or un- visited states. Given focus on loco-manipulation tasks characterized by continuous state spaces and potentially complex task dynamics, we consider the third option the most suitable.
1.2 Loco-Manipulation
In recent years, approaches to solving complex loco-manipulation tasks dominated by RL, c.g., [6, 29, 30], with some works fully or partly relying model predictive control (MPC) [31, 32].
2

---

A common design choice to break down the complexity of loco- beco-manipulation is to divide the prob- lem into a locomotion and a manipulation task and to control them individually [32, 33). This introduces new challenges in engineering the communication between both controllers and leads to sub-optimal behaviors as synergies between different body parts can not be fully exploited [34]. Other works focus on manipulating objects through locomotion, e.g., pushing an object through walking forward [31, 35, 36]. More dynamic manipulation tasks are investigated by different body of work that studies soccer skills [30. 37, 38, 6]. Many of these works combine multiple low-level skills into more capable controller, mostly through hierarchical control framework and skill li- brary [30, 38, 35, 37]. This could be future research direction to combine the controllers proposed in this work. However, little work exists on how to leverage intrinsic motivation to learn complex loco-manipulation tasks in a lean task-independent framework.
In this study, the primary task under examination is opening doors. A quadrupedal robot by Boston Dynamics equipped with arm has demonstrated impressive performance in the task of opening doors [39]. However, limited information is available regarding the specific approach employed, only that it is model-based. Other works divide the door opening task into sub-tasks that can be solved in sequence [40, 7]. This requires task-specific modeling and tailoring of the control scheme, which can identified as common shortcoming in many of the mentioned works. the following.
we aim for a more holistic approach.
2 Curiosity Formulation
Structured exploration is a crucial factor for successful learning in sparse reward settings. RND [9] offers an intuitive and computationally efficient approach to intrinsic motivation, while the curiosity state proposed in this work focuses exploration.
2.1 Random Network Distillation
The RND module consists of two function approximators. A randomly initialized target network f encodes states s E S into an unknown embedding f(s). The target stays fixed during the whole training process. A predictor network f estimates the target's embedding, given the same input as the target. The predictor network is trained alongside the RL agent on the visited states a supervised fashion with Mean Squared Error (MSE) loss. The prediction error, i.e., the difference between the outputs of both networks serves as the intrinsic reward signal defined by
-
Tintrinsic
Intuitively, familiar state regions yield small prediction errors as the predictor is already trained on similar states. Not yet visited state regions lead to large errors and therefore large intrinsic rewards. As the agent visits an unfamiliar region repeatedly, the prediction error decreases.
2.2 Curiosity State
While [9] applies the RND module directly to the agent's observations instead of the state s, propose more flexible formulation where curiosity state s. (0s) is passed to the RND module to stay independent of the observations. The mapping &b can be freely chosen, long as the state can be implicitly inferred from the environment's feedback. This way, curiosity can be focused on the desired quantities, even though they might not be directly observable during deployment of the motion policy. The thereby introduced flexibility allows to leverage the curiosity module simulation while keeping deployment simple, thus showing practical adaption of the RND formulation the robotics domain. While this formulation allows for arbitrary mappings d, it suffices to consider & to select subset of the full state without further modification. A sketch of the implemented RND module is shown in Fig. 2.
3

---

3 -neeciC RL Formulation
We illustrate an effective sparse reward formulation on the exemplary task of door opening, detailing the chosen rewards, observations, and the introduced curiosity state. Subsequently, we adapt the formulation to the task of package manipulation demonstrate the straightforward generalization of the proposed approach to different tasks.
3.1 Rewards
The chosen reward function consists of the three reward terms Tintrinsic Trask + Tishpinng The first term Timrrnsic is defined by equation and motivates the agent to explore the rele- vant part of the state space. this work, we use Multi Layer Perceptrons (MLPs) with and 2 hidden layers with neurons and one- dimensional output for the target and predictor network, respectively. The second term Tusk is the only task-specific reward. For the task of door opening, it is defined intuitively by re- peatedly giving a reward of +1 while the task is achieved, i... when the door hinge angle Qhinge in desired interval, and no reward other- wise:
Target
liutriusic
Predictor
Random Network Distillation (RND) module. A mapping encodes the full system state s into the curiosity state &c. The target net- work with fixed weights (blue) embeds the curios- ity state se into a latent representation. The pre- dictor network (green) is updated during training and attempts to match the target's output. The dif- ference between outputs serves the intrinsic ward signal.
Figure 2:
Trask (1, if 1.5 < Ouinge < 2.1
(2)
0, otherwise
The inclusion of the third term I'shaping serves as an incentive for the robot to maintain stand- ing posture and adopt more refined, less forceful strategies. This emphasis on smoother and less aggressive policies is crucial for achieving effective sim-to-real deployment within the realm of RL control [15, 41]. For a comprchensive breakdown of the rewards and their respective weights, please refer to Appendix A.
3.2 Observations
In terms observations, our approach focuses on minimizing task dependency and avoids the use of complex exteroceptive information. The only observation that pertains specifically to the door the relative position стCи of the door handle origin in the robot's camera frame. We freeze the initial door handle position and provide it additional input ercH... to the policy determine the degree to which the door is open.
We provide a list standard observations that are not task-specific and have been covered in previ- ous works [10, 4] in Appendix A.1. We note that the observations are subject to empirical normal- ization for proper scaling of input variables.
3.3 Curiosity Implementation
An advantage of the proposed curiosity state 8, is its independence of the robot's observations. Since the RND module is only needed for training, it is possible to include quantities that easy to attain in simulation but hard to estimate in reality and thus unfit as observations. To focus the agent's curiosity on the door, we include door hinge and handle angles qdoor as well as their angular velocities Quoor in the curiosity state. Adding the distance between the robot and the door handle dch induces faster interaction with the door. To avoid an intrinsic reward signal for moving too far away from the door, the distance is clipped according to deu min(]|rcHi l|,, 2), and the curiosity state is defined
[
T
dcH]

---

3.4 Generalization to Package Manipulation
We show the task independence and generalization capability of the proposed approach by subjecting it to a second task requiring different locomotion and manipulation skills. We choose the exemplary task of package manipulation, i.e. grabbing, moving, and dropping package, as involves freely moving object, as opposed the fixed-bass articulated door. To encode the task, we define a similar sparse task reward Trask, given by
1 if packaage E Soin
Flask
{0,
otherwise
(3)
where I pappccage the position of the package in the inertial frame I and Soin includes the space in and above the bin. We include the space above the bin reduce reward sparsity as the agent still learns drop the package. It is however not necessary to include this space for successful learning. Instead of observing the door handle position, the agent now observes the relative package, table, and bin positions сTCP, сTCT, and стCB, respectively. As before, relative positions are observed in the camera frame. The last necessary change concerns the curiosity state. We define it 8; | T package dpp dep where z package the linear package velocity in the inertial frame. The distances between package and bin d and camera and package der again clipped.
Experimental Validation and Discussion
To validate our approach, we conduct experiments in simulation and the real world. For implemen- tation details please refer to the Appendix. All experiments are conducted with the wheeled-legged quadrupedal robot in Fig. 1 a dynamic robot that can perform hybrid motions between walking and driving. Recently, [10] discovered bipedal locomotion mode through RL, further increasing the robot's versatility. In bipedal mode, the front legs can as manipulate objects.
4.1 Simulation Results
Qualitative examples of the door opening and package-grabee grabbing motions can be seen in the supple- mentary video.? In Fig. 4, present quantitative results for all three tasks with different reward settings. In general, we noticed that there is divide between runs in which the agent learns to
Figure 3: Snapshot sequences (left to right) of the push door and package manipulation experiments. Opening the door takes 2.5 seconds, grasping and dropping the package 1.5 seconds.
To enable pull door experiments, equip the robot with basic hooks attached the front wheels.
5

---

Spure Reward (cwso))
Pudh Doors
Pull Doun
Original RND
Packages
fum (cK-100)
(cw-200)
(cw=100)
10
12
14
16
Succesdul Training Buss
Number training runs (out of 10 for each task) in which the agent successfully learned to accomplish the given task for different reward formulations and different intrinsic ward weights (denoted as cw). Every experiment was conducted for random seeds 1-10.
Figure 4:
complete a given task and others where the sparse reward is solely triggered through randomness. Only in rare cases, the sparse reward is discovered but remains small due to shaping rewards that hinder skill discovery. We consider training runs with final success rate greater than 25 % success- ful to exclude the aforementioned cases. Successful runs show success rates significantly above this threshold, as can be seen in Appendix A.3, where we also provide a more thorough analysis for the case of (out-of-distribution) environment disturbances of various magnitudes. The best-performing policies achieve success rates of 99 %, 92 %, and 99 %, for the push door, pull door, and package manipulation tasks, respectively. In the following, we discuss the relevant findings of learning with curiosity.
Training Evaluation: The need for guided exploration becomes clear when training with extrinsic reward signals only, as reported in Fig. 4. Indeed, the investigated skills are not learned. Instead, the learning process ends a local optimum and the agent learns to stand without moving. The added intrinsic reward signal guides the agent toward states that involve manipulating the door or package. An exemplary learning process is shown in Fig. 5 for a push door. The learning process for package manipulation evolves similarly. First, the robot discovers the package and moves it on the table to increase the intrinsic reward signal. After learning to grab the package with both wheels, the robot starts to wiggle the package and moves it closer to the bin. Once the sparse task reward is found, the intrinsic reward decreases as the agent optimizes its behavior to achieve the desired task.
Experiments show that network architecture of the RND module can kept minimal. While predictor of the same size as the target should be able to fully approximate the target, choosing larger predictor leads more consistent intrinsic reward curves over different training runs. The weight of the intrinsic reward does not require extensive tuning since learning is successful for large interval, can be seen in Fig. 4. Normalizing the reward empirically as in [9] does not improve training and prevents the reward from converging to values close to zero. Again, convergence to small values is necessary to model the loss of interest as the agent gets more familiar with the environment. Normalization scales reward with its standard deviation and thus leads larger rewards at the end of training. In contrast, normalization of the curiosity state is crucial for a proper decay of the intrinsic reward over the course of training. The curiosity state includes quantities that reach high magnitudes like velocities and is therefore subjected to em- pirical normalization, i.e., it is normalized with a running estimate of its mean and variance. While normalization improves the convergence properties of the intrinsic reward, it violates the theoret- ical foundation state visitation- based curiosity. During training, the same state keeps changing its normalized representation due to varying normalization parameters. Different states could have almost identical representations for different points in time and thus also almost identical target em- beddings. Even though this distorts the measure for state familiarity, curiosity state normalization does not seem to have negative influence exploration and is highly recommended improve convergence small reward signal.
RND Evaluation:

---

Task Rewand
0.2
0.0
lantssi Revard
2
250
500
730
1000
1250
1500
1750
2000
Btemtio
Figure 5: Task specific reward Fask and intrinsic reward Vintrinsic over the course of learning to open push door. The mean and standard deviation include 4 successful out of training runs for a curiosity weight of 100 and random seeds 1-5. After learning how to stand (A), the robot starts to play with the door handle and the intrinsic reward increases (B). Next, the robot opens the door slightly while still manipulating the door handle (C). The agent then discovers the sparse door opening reward and starts to optimize toward it (D).
Curiosity State Evaluation: To assess the relevance of the curiosity state notion, we probe the original RND formulation of [9], that applies the curiosity module to the entirety of the observation space. We note that skill discovery is not feasible for the investigated tasks due to two reasons. First, the large state space might cause an unreasonable amount of exploration to properly converge in reasonable time. Second, parts of the state space that would be crucial to explore might not be directly observable, e.g., the door handle angle. Using a curiosity state instead of the observation space allows the use of privileged information which might not directly observable in the real world, but available in simulation.
Emerging Behaviors: A key finding of this paper is the sensitivity of the learning process to small changes in the training environment. Changing the seed used to initialize networks or set randomized quantities is enough to alter the resulting policy significantly. RI. approaches usually rely on dense task rewards that heuristically steer the agent toward a feasible trajectory. This might bias the agent toward suboptimal behavior preventing discovery of the sparse reward. We evaluate naive implementation
dense reward setup of comparable complexity to our method. For simple tasks like opening push doors, the dense reward setup is able to discover the skill, but for the other, more complex tasks it is not. In contrast does not bias the agent toward any trajectory. Randomness in the exploration can thus lead to the discovery of different minima and trajectories, especially in contact-rich scenarios. As a result, different behaviors emerge as shown in Fig. 6 where the robot is holding the door open in a variety of poses. For the task of package manipulation, differences are more subtle. During task execution, policies mainly differ in the leg configuration and the stepping pattern of the robot.
dense task reward setting, sparse task reward setting
4.2 Real-World Results
We demonstrate the effectiveness of the proposed approach by opening a push door over 15 times in a row without a single failure in lab environment. Learned policies are able to let the robot stand and navigate toward the door. The robot reaches for the door handle with its right wheel and attempts to press it while pushing against the door. As soon the door is unlocked, the robot swings the door open and holds it open while standing still, as shown in the top row of Fig. 3. Sim-to-real transfer benefits from modeling the robot's Field of View (FOV) in training, explained in Appendix A.2. Since the real robot lacks the added hooks for pulling doors, we leave pull-door experiments to future work. For the task of package manipulation (shown in the bottom row of Fig. 3), the robot robustly grabs the package and drops into the bin over times in row. In one instance out of all tests, the robot lost its grip on the package. However, the robot quickly regrabbed the package and successfully delivered it. Policies exhibit highly dynamic behavior. If more gentle motion is desired, a task reward that does not favor quick task completion would be more appropriate.
7

---

4.3 Limitations
Depending on the desired task, the intrinsic mo- tivation approach might come with some trade- offs. On one hand, the sparsity of rewards en- courages the exploration of diverse behaviors (see Fig. 6). which can lead to new solutions, often neglected by dense approaches. On the other hand, it complicates the process of enforcing specific behavior. Although dense shaping rewards can be added to enforce the desired behavior, they limit exploration and re- duce training robustness. This can be seen in Fig. where the investigated skills are not dis- covered in every training run.
Figure 6: A wheeled-legged robot holding the door open in a variety of poses.
Secondly, since the intrinsic reward signal is not vanishing completely, the agent stays curious about its environment even after finding the task reward. In our case, this did not cause unwanted behav- iors. it does for other applications of RND, weight of the reward could be scheduled.
Lastly, the predictor network trained in supervised fashion, overfitting specific regions of the state space could occur. Although not observed in this work, this might be exploited by the RL agent in back-and-forth fashion. Subsequent switching to different state space regions would gain intrinsic reward repeatedly. Common methods to avoid overfitting such as regularization or dropouts could be employed in that case.
5 Conclusions and Future Work
We show that intrinsic motivation for exploration proves successful in simulation, yielding motion policies for complex tasks that involve locomotion and manipulation. Our RL method proves to generalize over multiple tasks requiring different sets of skills. We note that different behaviors emerge for small changes in the training environment. This phenomenon is explained by the absence of dense task rewards that bias the agent toward specific trajectories and is inherent to the sparse reward setting.
The introduced notion of curiosity state guides exploration toward the reward in efficient manner and allows learning of various tasks with basic task-dependent observations. We note that curiosity state normalization is crucial for proper reward convergence during the course of training, even though it distorts the measure of state familiarity.
To validate the proposed method, trained motion policies are executed on wheeled-legged robot in biped mode. Experiments show that the robot is able successfully and robustly open a push door in lab environment, over 15 times in without failure, well as manipulate package through grabbing, moving, and dropping motion over 5 times in a row without failure. We conclude that the investigated approach proves valuable for the robotics control domain as it enables the learning of highly complex skills with a minimal amount task-specific engineering.
Future research could involve further investigation into the chosen curiosity formulation, the no- tion of penalty-based surprise [42] could allow for gentle policies without the need for task-specific shaping rewards. Other potential continuations include investigating controllers that achieve multi- ple tasks by combining the control policies proposed in this work.
Acknowledgments
This project has received support from the European Union's Horizon Europe and H202 Framework Programme under grants agreement No. 101070596 and No. 852044 well as from the Swiss National Science Foundation (SNSF) part of project No. 166232.

---

References
[I] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26))5872, 2019.
[2] J. Lee, J. Hwangbo, and M. Hutter. Robust recovery controller for quadrupedal robot using deep reinforcement learning. arXiv e-prints, 2019.
[3] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62):2822, 2022.
[4] J. Lee, M. Bjelonic, and M. Hutter. Control of wheeled-legged quadrupeds using deep rein- forcement learning. In Robotics in Natural Settings: CLAWAR 2022, pages 119-127. Springer, 2022.
[5] N. Rudin, D. Hoeller, M. Bjelonic, and M. Hutter. Advanced skills by learning locomotion and local navigation end-to-end. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2497-2503, IEEE, 2022.
[6] T. Haarnoja, B. Moran, G. Lever. S. H. Huang, D. Tirumala, M. Wulfmeeer, J. Humplik, S. Tun- yasuvunakool, N. Y. Siegel, R. Hafner, et al. Learning agile soccer skills for bipedal robot with deep reinforcement learning. arXiv e-prints, 2023.
[7] H. Ito, K. Yamamoto, H. Mori, and T. Ogata. Efficient multitask learning with an embodied predictive model for door opening and entry with whole-body control. Science Robotics, (65):8177, 2022. 2022.
[8] J. Achiam and S. Sastry. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv e-prints, pages arXiv-1703, 2017.
[9] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. In Seventh International Conference Learning Representations, pages 1-17, 2019.
[10] E. Vollenweider, M. Bjelonic, V. Klemm, N. Rudin, J. Lee, and M. Hutter. Advanced skills through multiple adversarial motion priors in reinforcement learning. In 2023 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 5120-5126. IEEE, 2023.
[11] M. Kalakrishnan, L. Righetti, P. Pastor, and S. Schaal. Learning force control policies for compliant manipulation. In Proc. JEEE/RSJ Int. Conf. Intell. Robots Syst., pages 4639-4644. IEEE, 2011.
[12] J. Ramírez, W. Yu, and A. Perrusquía. Model-free reinforcement learning from expert demon- strations: a survey. Artificial Intelligence Review, pages 1-29, 2022.
[13] X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions Graphics (TOG), 40(4): 1-20, 2021.
[14] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pages 41-48, 2009.
[15] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning walk in minutes using massively parallel deep reinforcement learning, In Conference on Robot Learning, pages 91-100. PMLR, 2022.
[16] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Wiele, V. Mnih, N. Heess, and J. T. Springenberg. Learning by playing solving sparse reward tasks from scratch. In International conference machine learning, pages 1344-4353 PMLR, 2018.
[17] C. Florensa. D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning, pages 482-495. PMLR, 2017.

---

[18] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. To- bin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In Proceedings of the st International Conference on Neural Information Processing Systems, pages 5055-5065, 2017.
[19] C. Florensa, D. Held, X. Geng, and P. Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528. PMLR, 2018.
[20] S. Forestier, R. Portelas, Y. Mollard, and P.-Y. Oudeyer. Intrinsically motivated goal explo- ration processes with automatic curriculum learning, The Journal of Machine Learning Re- search, 23(1):6818 8-8888688888881 2022.
[21] Schmidhuber. possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals animats, pages 222-227, 1991.
[22] J. Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Trans. Auton. Mental Develop., 2(3):230-244 24000-2272222 2010.
[23] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In International conference on machine learning, pages
PMLR, 2017.
[24] P.-.. Oudeyer, F. Kaplan, and V. V. Hafner. Intrinsic motivation systems for autonomous mental development. IEEE Trans. Evol. Comput., 11(2):265-286, 2007.
[25] A. Baranes and P.-Y. Oudeyer. Robust intrinsically motivated exploration and active learning. In 2009 IEEE 8th International Conference on Development and Learning, pages 1-6. IEEE, 2009.
[26] M. Frank, J. Leitner, M. Stollenga, A. Förster, and Schmidhuber. Curiosity driven reinforce- ment learning for motion planning humanoids. Frontiers in neurorobotics, 7:25, 2014.
[27] G. Ostrovski, M. G. Bellemare, A. Oord, and R. Munos. Count-based exploration with neural density models. In International conference on machine learning, pages 2721-2730. PMLR, 2017.
[28] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Proceedings of the 30th International Conference Neural Information Processing Systems, pages 1479-14877 2016.
[29] J. Merel,, S. Tunyasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, V. Pham, T. Erez, G. Wayne, and N. Heess. Catch & carry: reusable neural controllers for vision-guided whole-body tasks. ACM Transactions on Graphics (TOG), 39(4)339, 2020.
[30] Y. Ji, G. B. Margolis, and Agrawal. Dribblebot: Dynamic legged manipulation in the wild. arXiv preprint arXiv:2304.01152 2023.
[31] A. Rigo, Y. Chen, S. K. Gupta, and Q. Nguyen. Contact optimization for oon-erehen loco- manipulation via hierarchical model predictive control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9945-9953 IEEE, 2023.
[32] Y. Ma, Farshidian, T. Miki, Lec, and M. Hutter. Combining Icarning- locomotion policy with model-based manipulation for legged mobile manipulators. IEEE Robot. Autom. Lett., 7()):237--2884, 2022.
[33] X. Cheng, A. Kumar, and D. Pathak. Legs as manipulator: Pushing quadrupedal agility beyond locomotion, arXiv preprint 2rXiv:2303.11330 2023.
10

---

[34] Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: learning unified policy for manipulation and locomotion. In Conference on Robot Learning, pages 138-149. PMLR, 2023.
[35] K. N. Kumar, Essa, and S. Ha. Cascaded compositional residual learning for complex inter- active behaviors. IEEE Robotics and Automation Letters, 2023.
[36] O. Nachum, M. Ahn, H. Ponte, S. Gu, and V. Kumar. Multi-agent manipulation via locomo- tion using hierarchical sim2real. In Conference Robot Learning, pages 110-121. PMLR, 2020.
[37] Y. Ji, Z. Li, Y. Sun, X. B. Peng. S. Levine, G. Berseth, and K. Sreenath. Hierarchical reinforce- ment learning for precise soccer shooting skills using quadrupedal robot. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1479-1486. IEEE, 2022.
[38] X. Huang, Z. Li, Y. Xiang, Ni, Y. Chi, Li Yang, X. B. Peng, and K. Sreenath, Creating a dynamic quadrupedal robotic goalkeeper with reinforcement learning. arXiv e-prints, pages arXiv-2210, 2022.
[39] Boston Dynamics. Testing robustness, 2018. URL hwtpht:://ootttaa http:y/uuuuubaaaauaaeeeee be/ ae/aaFAASO9.
[40] M. Stuede, K. Nuelle, S. Tappe, and T. Ortmaier. Door opening and traversal with an industrial cartesian impedance controlled mobile robot. In Proc. IEEE Int. Conf. Robot. Autom., pages IEEE, 2019.
[41] Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47):5986, 2020.
[42] S. H. Huang. M. Zambelli, J. Kay, M. F. Martins, Tassa, P. M. Pilarski, and R. Hadsell. Learning gentle object manipulation with curiosity-driven deep reinforcement learning. arXiv e-prints, pages arXiv-1903, 2019.
[43] V. Makoviychuk, I.. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu based physics simulation for robot learning. In Thirty-fffth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[44] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv e-prints, 2017.
[45] E. Olson. Apriltag: A robust and flexible visual fiducial system. In Proc. IEEE Int. Conf. Robot, Autom., pages 1400-3407, IEEE, 2011.

---

Appendix
In the following, we provide implementation details of the simulation and real-world experiments, as well as further quantitative evaluations of the investigated approach.
A.1 Simulation Setup
We train with NVIDIA's Isaac Gym [43] and employ Proximal Policy Optimization (PPO) [44]. A detailed description of the used training pipeline can be found in [15]. A full training run comprises 2000 policy updates to ensure reward convergence for all investigated tasks. takes one hour to train a policy on a single NVIDIA RTX 2080 Ti graphics card. Subsequently, we give a detailed description of the training environment.
Reward Formulation: The definitions and weights of the reward terms used for the door and the package task are detailed in Table 1. We decided to add two task-related shaping rewards for the task of package manipulation to improve the behavior for real-world tests. Namely, the agent receives penalties for generating high package velocities and exerting large contact forces onto the table. Notice that this choice does not violate the idea the proposed approach. Firstly, the added penalties are unrelated to the main task, which is still defined by a single sparse reward. Secondly, our approach first generates unbiased behaviors and can then be augmented for more pleasing results. In contrast, other formulations bias the agent as byproduct of defining the desired task in dense fashion. Penalizing table contacts and the package velocity, which is part of the chosen curiosity state, clearly increases the difficulty of discovering the desired skill. To compensate for this, we employ a simple reward scaling scheme. The first 1000 training iterations serve as a discovery phase, as most runs discover the sparse reward in that time. Shaping and standing rewards are active but scaled by a factor of 0.1. The second half of training acts as a shaping phase where the scaling factor is gradually increased to over the course of 500 iterations.
The corresponding observation definitions can be found in Table 2. All observations are subject to noise to account for uncertainties and sensor noise in reality. For more detail in that regard, please refer to [15].
Observations:
Randomization:
To improve generalization to different environments, as well as robustness against mismatches between simulation and reality, masses and friction coefficients randomized as de- tailed in Table 3. Additionally, the robot spawns in randomized pose, i.c., initial position, orienta- tion, and joint configuration vary. All randomized properties are sampled from a uniform distribution in the interval of (u Š 5] for every training environment.
Termination Conditions: Episodes terminate after seconds, resetting the environments to their initial state. An episode terminates early if either the robot is in collision, or if the robot's center too low, i.e., if the robot does not manage to stand and falls. The second condition accelerates training but is not necessary for successful learning. We also terminate an episode if the package is not in contact with either the table or the front wheels to prevent the agent from directly throwing the package. This termination condition is disabled in close proximity to the bin to allow the dropping of the package into the bin.
Door Model:
The considered doors feature standard lever door handles that need to be pressed to certain degree to unlock the door. In simulation, the handle needs to be pressed once to keep the door unlocked for the rest of the episode. Dynamics of the hinge and handle are modeled as spring-damper systems with a constant torque offset Tooost This is achieved by applying the torque
Tooor = Teonst + diag(k) Adoor diag(d) (d) Qdoor:
to the door joints. Constants Tooust k. and d are randomized by sampling from uniform distribution. Measurements on the lab door provide reference values for realistic door dynamics. Further details are provided in Table 3.
Field of View Simulation: To mimic the perception system of the real robot we simulate the FOV for egocentric vision, as introduced in simulation experiments in [29], resulting in behaviors that
12

---

Table 1: Rewards
Name
Formula
Weight
Intrinsic Reward
RND prediction error
((s.) - f(se)
100
Task Rewards
11 if 1.5 < Qhinge < 2.1 1.0
Door opened
0,
otherwise
1, if packkaee E Spin
Package delivered
1.0
0, otherwise
Standing Rewards
Height
T. Phase
0.5
//2 arccos( 7 e 7e
Upright base
0.5
x/2
Straight shoulder joints
Ishouldrss
0.5
Straight knee joints
exp Ikncos 11
0.25
Shaping Rewards
Joint torque
2
1.5 10-5
Joint acceleration
2
2.5-
Joint velocity
Q||
20--
Action difference
a aprey
1
1.0-10-2
Table contact force
Fe table
1.0-10-5
Package velocity
1 i package
12
1.0- 10-2
Table 2: Observations
Robot-related Observations
gubee E 3
Linear base velocity
BWaase E R3
Angular base velocity
sg E 3
Projected gravity vector
Neegs 12
Joint configuration without wheels
Obooks E R4
Hook directions (for pull doors)
q E 16
Joint velocity
Opeev 16
Previous actions
Door-related Observations
сTCH E 3
Relative door handle position Relative initial door handle position
сTC E 3
Package-related Observations
cPCp 3
Relative package position Relative table position
eTer E R3
CTCB E R
Relative bin position
13

---

Table 3: Randomization Parameters
Uniformly Randomized Property
Mean μ
Range
Unit
Global friction coefficient Robot position (æ, y) Initial robot yaw angle
0.75
0.75
0.6
m
1
rad
Initial joint angle deviation
rad
Added robot mass Package mass
10
kg
1.375
1.0
kg
Door torque offset Tconst
[10
0]
[10
0J
N m
N m
Door spring coefficient &
[0
5]
[0
5]
rad
Nms
Door damping coefficient d
[25 1]
[25
1]
rad
actively direct the robot's gaze. A visual marker, further explained in Section A.2, specifies the position of the door handle. Consequently, the observation c#CH only available the marker detected by camera. Always passing the door handle observation in the simulation would therefore capture the real system behavior. Instead, the observation is the camera's 's FOV. This way, the agent learns to approximately partition the observation space and reason about when it is necessary to observe the visual marker. The agent can develop behaviors to mitigate lost observation and to actively keep the marker in the FOV. An illustration of the approach is provided Fig. Note that the second door-related observation cTC is not set to because the initial door handle position static with respect to the incrtial frame. The observation can thus be bootstrapped with the onboard localization of the robot even the visual marker leaves the FOV.
if the visual marker leaves
14

---

A.2 Real-World Setup
We utilize AprilTaggs [45] to obtain task-related observations in the real world. The April'aag system features vision-based algorithm that determines the relative position and orientation of detected tags. Two visual markers attached the door provide the relative door handle position observation сrCи. If the robot does detect the tags, the observation is in simulation. The initial door handle position observation c.C is determined by two markers attached to the door frame. We make use of the robot's onboard localization to obtain an observation even if the tags leave the FOV of the camera. AprilTaags also provide relative positions of the package, bin, and table. We do not make use of the proposed FOV simulation for the package manipulation task for two reasons. Firstly, it increases the difficulty of learning the desired behavior because the robot tries to keep the package in the FOV by leaning over the bin and falling. Secondly, the package kept in the FOV naturally until the package dropped, rendering the additional FOV constraint unnecessary for this task.
achieve the same behavior
Furthermore, we note a few limitations with the current experimental setup. To achieve 100 % success rate the series real-world experiments, it was vital to get reliable door observations through the camera system. Especially for fast rotations, the used visual fiducial system suffered from image blur and low frame rates. Observations might also degrade over longer periods of time if the fiducials leave the FOV, the robot then purely relies its localization. With the remedies mentioned above, were able to resolve these issues, but longer horizon tasks might need more careful considerations.
Qhinge Ghinge
Qhandle, Thandle
cNCH dCH
CT CTCH init
Figure 7: Door setup and FOV simulation. Components of the curiosity state 8, are marked in red, while observations are marked in blue. The green cone represents the camera's 's FOV. A visual marker, attached to the door, is used to calculate the door handle observation стси- the vector from the camera to the visual marker (orange) leaves the FOV cone, the door handle observation is set to
15

---

A.3
Quantitative Results
A.3.1 Comparison to a Naive Dense Reward Setting
100
To highlight the benefits of the curiosity-driven approach, we draw a comparison to a basic dense reward approach that involves comparable engineering effort.² For the door opening tasks, define three dense rewards guidance toward the sparse task reward. These rewards are defined to minimize the distance from the wheel to the door handle, the door handle angle, and the door hinge angle. The hinge angle reward is clipped to en- sure that the robot does not open the door too far since we consider the task only fulfilled the door opened within specified angular window. We were able tune the reward weights deliver similar performance our approach for push doors, yielding an average success rate of 91 % in simulation, as seen in Fig. 8. We could not find weights that would result in successfully learning the pull door task. The package manipulation task is augmented with three dense rewards as well. These increase with de- creasing distance between the right wheel and the right side of the package, the distance between the left wheel and the left side of the package, and the distance between the package and the bin. Again, were unable to train policy that achieves the desired task, shown in Fig. 4.
Sucucs Ratz (8)
Pull Doses
Packages
Dense Rewards
Unrs (cw=100)
Our (ewm200)
Oun (cWm300)
Figure 8: Success rate mean and stan- dard deviation of successful training runs for different reward formulations, intrinsic reward weights (denoted as cw), and random seeds 1-10.
A.3.2 Intrinsic Reward Scale Sensitivity
The training process shows high level of robustness with respect to the scale of the intrinsic reward, as can be seen in Fig. 4. This can be explained by the reward's dynamic magnitude. At the beginning of training, the RND prediction error is large enough to overcome the local minimum imposed by shaping rewards. During training, the intrinsic reward shrinks and allows for optimization toward the task and shaping rewards. If the weight chosen too large, the intrinsic reward might not decay enough such that the sparse reward, although discovered, might get overlooked in the optimization.
A.3.3 Robustness Against Environment Variation
We investigate success rates in multiple simulation experiments to analyze the robustness of learned policies against the variation of different environment parameters, including parameter values that are out-of-distributton of the learning tasks. We report the results Fig. The success rates are determined by observing 1000 differently randomized environments for one episode. First, the robot's initial position and orientation are randomized over a larger interval than during training. Second, the door handle height randomized uniformly. Even though the handle height was not randomized during training, policies able adapt different heights. Last, the package position and the bin position randomized uniformly. Again, neither randomized during training. The package position is randomized for range up to 0.4 m, as this covers the entire table surface. Trained policies exhibit high level of robustness against the investigated disturbances.
We also investigated curriculum learning alternative. The (naive) task curricula consisted of spawning the door, with wheel touching the door handle), and then
the robot in favorable positions (e.g. very gradually increasing task difficulty by increasing distance to door. In our experiments, the robot was not able to successfully discover the desired behavior. We are convinced that the tasks could be solved with more intricate curricula, but these stand in no perspective with regard to the engineering effort to the proposed curiosity-driven driven approach.
curiosity
16

---

100
100
90
Stcrcce Rat (5)
95
80
90
70
Push Doors
85
Pull Doors
50-
Push Deors
Package Randomicatoo
Packages
Pull
1.0
2.0
2.5
5.0
0.0
0.1
0.2
0.1
0.4
0.00
0.25
0.50
0.75
1.00
Rondomizzziinn Mutippier
Randomizaiion Range Meters
Randomization Moters
(a)
(b)
(c)
Figure 9: Simulation success rates for different cut-of-distribution experiments. In (a). the initial robot position and yaw angle randomization range used during training is multiplied. In (b), the door handle height is randomized and in (c), the package and bin positions randomized uniformly.
17


Reviews:
[{"review_id": "xd6GNp6KOW6", "reviewer": "Reviewer_81RD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes to use curiosity-driven reward bonus to learn loco-manipulation tasks. A wheeled-legged robot is used in biped mode for experiments in the simulation and the real world on two tasks: door opening and package manipulation. Concretely, the vanilla RND is used as the exploration bonus. The authors also analyze the emerging behaviors and the sensitivity of the learning process. ", "summary_of_recommendation": "My recommendation reflects the current problems of the submission. Would be very happy to see future changes, and I'll update my score accordingly. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "- Can quantitive results be included? For both simulation and real world. \n- Can some environmental diversities be shown to validate that this method does not just overfit to a single task, environment, and initial state?\n- Can authors include a much more comprehensive discussion on the similarities and differences with the prior work that I listed in the above section?\n- Can comparisons with baselines (e.g. reward shaping, curriculum learning, etc) be included?\n- Can the limitation section include more related information about when the method fails? For example, difficulties in relying on a fiducial system, shown in the the supplementary material: \"We make use of the robot’s onboard localization to obtain an observation even if the tags leave the FOV of the camera\". ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Pros:\n- Interesting tasks are studied in this paper including using a wheeled-legged robot to open a door. \n\n### Cons:\n- This paper does not include any quantitative evaluations. The only descriptions regarding the performance of the method are \"over 16 times in a row without failure\" for the door opening task and \"over 6 times in a row without failure\" for the package moving task. The quantitative metrics can include measurements of success rates, time to completion, and robustness to different environments and objects.\n- It does not compare with any other baselines methods, including simple reward shaping, simple curriculum learning, or simple imitation. \n- One of the key motivations of the paper is using exploration bonus to alleviate heavy reward engineering. However, it is shown in the appendix that for each task, excluding the RND bonus term, 9 reward terms are used. As a result, a new natural question to ask is that why not just add several more reward shaping terms (e.g. L2 distance to the handle, etc) to solve these locomotion-manipulation tasks? \n- All the experiments focus on the exactly same sim and real pairs. For the door-opening task, the robot resets to the exact same initial position very close to the door, and the door is the same in all experiments. The same applies to the package-moving task. Hence, it makes me wonder does the method just overfit to a single environment without generalizing to other environments?\n- No limitations regarding the tasks, real-world robotics challenges or sim-to-real transfer are described in the limitation section. The limitation section solely focuses on describing the difficulties of using the RND exploration bonus, which is well-known. \n- This paper neglects most of (if not all) the prior work on manipulation-locomotion and locomotion-manipulation from the robotics community, The prior work section overwhelmingly focuses on exploration and RL training. However, this work does not introduce any new or change to the existing exploration method RND. In contrast, many existing works on manipulation-locomotion and locomotion-manipulation are not mentioned at all. These include but are not limited to:\n  - Human Motion Control of Quadrupedal Robots using Deep Reinforcement Learning. Sunwoo Kim, Maks Sorokin, Jehee Lee, Sehoon Ha. RSS 2022. \n  - Deep whole-body control: learning a unified policy for manipulation and locomotion. Zipeng Fu, Xuxin Cheng, Deepak Pathak. CoRL 2022. \n  - Hierarchical Reinforcement Learning for Precise Soccer Shooting Skills using Quadrupedal Robots. Yandong Ji, Zhongyu Li, Yinan Sun, Xue Bin Peng, Sergey Levine, Glen Berseth, Koushil Sreenath. IROS 2022. \n  - Legs as Manipulator: Pushing Quadrupedal Agility Beyond Locomotion. Xuxin Cheng, Ashish Kumar, Deepak Pathak. ICRA 2023. \n  - Cascaded Compositional Residual Learning for Complex Interactive Behaviors. K. Niranjan Kumar, Irfan Essa, Sehoon Ha.\n  - DribbleBot: Dynamic Legged Manipulation in the Wild. Yandong Ji, Gabriel B. Margolis, Pulkit Agrawal. ICRA 2023. \n  - Creating a Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning. Xiaoyu Huang, Zhongyu Li, Yanzhen Xiang, Yiming Ni, Yufeng Chi, Yunhao Li, Lizhi Yang, Xue Bin Peng, Koushil Sreenath. \n  - Learning agile soccer skills for a bipedal robot with deep reinforcement learning. Tuomas Haarnoja, et al. \n  - Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Real. Ofir Nachum, Michael Ahn, Hugo Ponte, Shixiang Gu, and Vikash Kumar. \n  - Contact optimization for non-prehensile loco-manipulation via hierarchical model predictive control. Alberto Rigo, Yiyu Chen, Satyandra K Gupta, Quan Nguyen. ICRA 2023. ", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "eH-KiXSQYq", "reviewer": "Reviewer_a9Bt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies the problem of learning joint locomotion and manipulation tasks. Using task-specific rewards can require labor intensive engineering and require re-tuning for a new task. An intrinsically motivated RL approach using Random Network Distillation (RND) is proposed to reduce task-specific engineering with sparse task rewards and intrinsic rewards. The method is shown to generalize to the door opening task and the packing task. Learned policies are evaluated on a wheeled-legged robot on two tasks and achieve 100% success rate.\n", "summary_of_recommendation": "After reading the updated paper and the rebuttal, most of the question I asked above is solved. My recommendation remains the same. The paper proposed an approach applying RND on learning joint locomotion and manipulation tasks. Comprehensive results analysis and discussion show the task-agnostic properties of the method. Some implementation details and insights are valuable to the community. Demonstration of the learned policies on the hardware is impressive.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Since the shaping rewards bring difficulty to the skill discovery. Have the author considered fine-tuning the policy with the shaping reward later by combining BC and RL? For example, policy 1 can be learned without the shaping reward. Policy 2 can be learned by using policy 1 as the teacher policy with BC loss and using the shaping reward as the RL loss.\n\nOn line 211, “The need for guided exploration can be seen in a training run with extrinsic reward signals only”. I couldn’t find this run in the paper or in the supplement.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The curiosity driven approach, RND, is successfully applied and validated on the two real robot tasks in this paper. Combining with the curiosity state, the proposed method show task-agnostic properties and generalize well to two different tasks. \n\nStrength: Results on the real robot are pretty impressive. The motivation of the paper is clear. The task-agnostic properties of the proposed method is thoroughly discussed and convincing. Results analysis of the experiments results are comprehensive and insightful. The paper also includes lots of implementation details regarding weights design and normalization with detailed discussion, which are very useful for the community. The emergence of different behaviors and its pros and cons are very interesting.\n\nWeakness: \n(1) The trained policy looks aggressive on real robot demonstration even with the shaping reward. It seems the shaping rewards may still require extensive tuning as it still contains task specific terms. \n(2) As mentioned by the author, the shaping rewards bring difficulty to the skill discovery. A reward scaling scheme is applied to compensate this. However, it’s unclear if this is sufficient for more complicated systems or longer horizon tasks.\n(3) There is no comparison between the proposed method and other baselines. On line 211, “The need for guided exploration can be seen in a training run with extrinsic reward signals only”. I couldn’t find this run in the paper or in the supplement.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "VWs5lw4m2ZO", "reviewer": "Reviewer_qvEX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a method for learning loco-manipulation skills with sparse rewards. They employ intrinsic motivation in the form of Random Network Distillation to provide a guiding signal for the agent to discover the sparse reward effectively. The based RND method is modified in two ways. First, a subset of task-relevant state features is used instead of the full observation. Second, the inputs for the RND networks are continuously normalised. The approach is successfully applied to both a door opening and package manipulation task with a wheeled-legged robot, and the resulting controllers are successfully transferred to hardware.", "summary_of_recommendation": "As the paper lacks quantitative results and the proposed method is very incremental compared to prior work, with no significant discussion of such, I find it difficult to vote to accept this paper. I'm voting for a weak reject instead of a strong reject due to the excellent presentation and the results on hardware.\n\nEdit after rebuttal: I have raised my recommendation after the authors' rebuttal due to the additional quantitative results and the rephrasing of the contributions. I would still urge the authors to move the quantitative results to the main text instead of appendix.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- What is the overall success rate for both tasks and across seeds in simulation, and ideally in real?\n- What is meant with \"succeeds 6 times in a row\"? Did it fail on the 7th? Were only 6 trials performed?\n- How sensitive are the results, quantitatively, to the scales of the various rewards?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "These days, training RL policies for robot control often involve complex, dense, heavily-shaped and task-specific rewards in order to solve the task. While the resulting behaviours are generally impressive and robust, this reward design requires significant tuning and effort for every task under consideration. Approaches that allow \"coarse\" reward definition by improving exploration behaviour are hence incredibly relevant for the robotics community. While I commend the authors for the clear presentation, professionally-montaged supplementary video and the hardware results, I have a couple of major concerns with this work:\n- There has been lots of prior work (e.g. https://arxiv.org/abs/1708.02190 from 2017) on applying intrinsic motivation to robotics settings. It would be appreciated to focus the related work section on this intersection and explicitly contrast the proposed approach.\n- The methodology is very incremental from base RND, and dare I say even removes some of the appeal by requiring only task-relevant state features to be used for the intrinsic reward. This effectively shifts some of the effort from reward design to feature design. This may be easier or less sensitive, but from the results in the paper it is hard to judge.\n- In a similar vain, a major issues with intrinsic exploration methods such as the one proposed, is the balance between intrinsic and task rewards, and in this case also regularisation rewards. This was also discussed in related work such as https://arxiv.org/abs/1903.08542. The authors touch on this and even note that it is important to tune reward scales, even with which the learned behaviours are still very dynamic (underscaling the regularisation reward). I would like to see more results on the sensitivity of reward scales to being able to solve the task and on the resulting behaviour.\n- While the behaviours shown in the figures and video are reasonably novel and interesting, I find the paper lacking quantitative results, only present in the caption of Figure 4 (4/6 seed success, missing from main text) and Section 5.2. There are no baselines that are compared to (e.g. other exploration mechanisms, dense rewards, model-based?). The evaluations should at least mention an overall task success rate in sim across many trials, and ideally so should the results in real (when the agent succeeds in 6 trials in a row in real on the package manipulation task, does that mean it failed on the 7th? Were only 6 trials executed?). While its great to see that the resulting controllers can be deployed on hardware, this is using existing approaches so in itself does not validate the proposed approach.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "E6OIR2nZd8r", "reviewer": "Reviewer_bDmA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper tackles the challenging joint locomotion and manipulation tasks — door opening and package manipulation — on a wheeled-legged robot. The paper proposes to train RL policies with an intrinsic reward to encourage exploration, where the intrinsic reward is adapted from previously proposed Random Network Distillation (RND). The simulation-trained policy is shown to transfer to real-world setup.", "summary_of_recommendation": "The paper tackles important and interesting tasks of joint locomotion and manipulation on wheeled-legged robot, while showing impressive real robot demos resulted from an RL policy trained purely in simulation. However, the paper lacks quantitative evaluations which make the technical contributions or experimental contributions unclear.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Section 5.1 mentions that training with extrinsic reward would fail, but no information or experiment of this can be found in the paper. Can this be further clarified?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper has following strengths:\n\n- The tasks as well as the real robot setup considered in this work are interesting, novel, and likely significant for future works.\n- The real-world robot demos are impressive.\n- The paper writing is clear and effectively conveys the contributions as well as the results of the work.\n\nDespite the strengths, there are a few major weaknesses:\n\n- While the tasks considered are interesting, the technical contributions of this work might be limiting. The main technical contribution of the paper seems to be an adaptation of the previously proposed RND method, where instead of using full observation, the paper proposes a new notion of “curiosity state”, which is a subset of full environment state and needs to be hand-selected for each task. By manually selecting the curiosity state to be a few “important parameters”, this essentially forms an implicit task curriculum that encourages the agent to explore within the manifold of those parameters. This does not seem to be an improvement of RND, where the whole premise of intrinsic reward is to reduce task-specific engineering.\n- To demonstrate technical contributions, an important and necessary way is to compare to prior works quantitatively, but this also is lacking from the paper. Except for the training plot that shows the progression of task-specific reward and intrinsic reward, no other quantitative results are seen from the paper.\n- Given the impressiveness of the real-world setup, the paper can also be shown to be of good value to the community if extensive experimental results are shown (e.g., across many more tasks and benchmarked against baselines), but the experimental results currently  in this work are lacking from this perspective.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There are strengths and weaknesses of the paper. For the strengths, the qualitative real robot results are quite nice. The biggest weakness before the rebuttal was the lack of quantitative results, though the quantitative results are much more thorough after the rebuttal. Now, it seems like the main limitations are the limited "generalization to different environments/scenes (testings are done on one door, one object, and relying on ARTags)" (reviewer 81RD), the significant revision needed to incorporate the new results (reviewer bDmA), and the limited technical contribution (multiple reviewers).

Overall, I think the strengths outweigh the weaknesses and recommend acceptance.
Decision: Accept (Poster)