Benchmarking Offline Reinforcement Learning on Real-Robot Hardware
Authors: Nico Gürtler
Sebastian Blaes
Pavel Kolev
Felix Widmaier
Manuel Wuthrich
Stefan Bauer
Bernhard Schölkopf
Georg Martius
Venue: ICLR 2023 notable top 25% (2023)
Keywords: offline reinforcement learning
robotic manipulation
dexterous manipulation
TriFinger platform

Abstract: Learning policies from previously recorded data is a promising direction for real-world robotics tasks, as online learning is often infeasible. Dexterous manipulation in particular remains an open problem in its general form. The combination of offline reinforcement learning with large diverse datasets, however, has the potential to lead to a breakthrough in this challenging domain analogously to the rapid progress made in supervised learning in recent years. To coordinate the efforts of the research community toward tackling this problem, we propose a benchmark including: i) a large collection of data for offline learning from a dexterous manipulation platform on two tasks, obtained with capable RL agents trained in simulation; ii) the option to execute learned policies on a real-world robotic system and a simulation for efficient debugging. We evaluate prominent open-sourced offline reinforcement learning algorithms on the datasets and provide a reproducible experimental setup for offline reinforcement learning on real systems.

Published as a conference paper at ICLR 2023
BENCHMARKING OFFLINE REINTORCEMENT LEARN- ING ON
HARDWARE
Nico Gürtler', Sebastian Blaes', Pavel Kolev', Felix Widmaier', Manuel Wüthrich', Stefan Bauer', Bernhard Schölkopf' and Georg Martius
Max Planck Institute for Intelligent Systems* Harvard University *KTH Stockholm
ABSTRACT
Learning policies from previously recorded data is a promising direction for real- world robotics tasks, as online learning is often infeasible. Dexterous manipu- lation in particular remains an open problem in its general form. The combina- tion of offline reinforcement learning with large diverse datasets, however, has the potential to lead to a breakthrough in this challenging domain analogously the rapid progress made in supervised learning in recent years. To coordi- nate the efforts of the research community toward tackling this problem, we propose a benchmark including: i) a large collection of data for offline learn- ing from dexterous manipulation platform on two tasks, obtained with capa- ble RL agents trained simulation; ii) the option to execute learned policies real-world robotic system and simulation for efficient debugging. We evaluate prominent open-sourced offline reinforcement learning algorithms on the datasets and provide reproducible experimental setup for offline reinforce- tes ment learning on real systems. Visit ht : wtw.tooo.ooo..eooooooooooo wwwgle.oo..oooooooooooooooooooo. com/ view/ real for more details.
INTRODUCTION
Reinforcement learning (RL) (Sttonn et al. 1998) holds great potential for robotic manipulation and other real-world decision-making problems interactions with the environment. When data can be collected during learning, RL in combination with high-capacity function approximators can solve challenging high-dimensional problems (Mnih) et al. 2015 Lillicrap et al., 2016, Silver et al. Berner et al. 2019). However, in many cases online learning not feasible because collecting large amount of experience with a partially trained policy is either prohibitively expensive or unsafe (Dulac-
solve tasks autonomously by learning from
et al.) (022)). Examples include
autonomous driving, where suboptimal policies can lead accidents, robotic applications where the hardware is likely to get damaged without additional safety mechanisms, and collaborative robotic scenarios where humans are at risk being harmed.
Offline reinforcement learning (offline RL or batch RL) (Lange et al. 2012) tackles this problem by learning a policy from prerecorded data generated by experts or handcrafted controllers respecting the system's constraints. Independently of how the data is collected, it is essential make the best possible use and to design algorithms that improve performance with the increase of available data. This property has led to unexpected generalization in computer vision (Krizhevsky 2012, He et al. 2016, Redmon et al., 2016) and natural language tasks (Floridi & Chiriatti 2020 Devlin al. 2018) when massive datasets are employed. With the motivation to learn similarly capable decision-making systems from data, the field of offline RL has gained considerable attention. Progress currently measured by benchmarking algorithms on simulated domains, both in terms of data collection and evaluation.
al.
"Correspondence ni colccogueee wo..terttloooooooomooo.o. wee.guuuettrueeeee errtuebi we@.ueeingennmmgggg.....

---

Published as a conference paper at ICLR 2023
Figure 1: The TriFinger manipulation platform (Withrich et al., 2021 Bauer et al., 2022). Left: The robot has? arms with 3 DoF each. The cube is constrained by a bowl-shapee wowl-shaped arena, allowing for unattended data collection. Right: A cluster of these robots for parallel data collection and evaluation.
Yet, real-world data differs from simulated data qualitatively and quantitatively in several as- pects (Dulac-Arnold et al., 2020). First, observations are noisy and sometimes faulty, Second, real-world systems introduce delays in the sensor readings and often have different sampling rates for different modalities. Third, the action execution can also be delayed and can get quantized by low-level hardware constraints. Fourth, real-world environments are rarely stationary. For instance, in autonomous robotics, battery voltages might drop, leading to reduced motor torques for the same control command. Similarly, thermal effects change sensor readings and motor responses. Abrasion changes friction behavior and dust particles can change object appearances and sensing in general. Fifth, contacts crucial for robotic manipulation but only insufficiently modeled in current physics simulations, in particular for soft materials. Lastly, physical robots have individual variations.
Since real-world data is different from simulated data, it is important put offline RL algorithms the test on real systems. We propose challenging robotic manipulation datasets recorded on real robots for two tasks: object pushing and object lifting with reorientation on the TriFinger platform (Wüthrich) et al. 2021). To study the differences between real and simulated environments, we also provide datasets collected simulation. Our benchmark of state-of-the-ar offline RL algorithms on these datasets reveals that they are able to solve the moderately difficult pushing task while their perfor- mance on the more challenging lifting task leaves room for improvement. In particular, there a much larger gap between the performance of the expert policy and offline-learned policies on the real system compared to the simulated system. This underpins the importance of real-world benchmarks for offline RL. We furthermore study the impact of adding suboptimal trajectories expert data and find that all algorithms are 'distracted" by them, i.e., their success rate drops significantly. This identifies an important open challenge for the offline RL community: robustness suboptimal trajectories.
Importantly, cluster of TriFinger robots is set up for evaluation of offline-leamedd policies for which remote access be requested for research purposes. With our dataset and evaluation platform, therefore aim to provide a breeding ground for future offline RI. algorithms.
2 THE TRIFINGER PLATFORM
We robot cluster that was initially developed and build for the Real Robot Challenge in 2020 and 2021 (Bauer et al. 2022). The robots that constitute the cluster are an industrial-grant adaptation of a robotic platform called TriFinger, an open-source hardware and software design introduced in Wüthrich et al. 202T), D, see Fig. The robots have three arms mounted at 120 degrees radially symmetric arrangement with 3 degrees of freedom (DoF) each. The arms are actuated by outrunner brushless motors with 1:9 belt-drive, yielding high agility, low friction, and good force feedback (details on the actuator modules can found in Grimminger et al. (0200). Pressure sensors inside the elastic fingertips provide basic tactile feedback. The working area, where objects can be manipulated, encapsulated by a high barrier to ensure that the object stays inside the arena even during aggressive motions. This is essential for operation without human supervision. The robot is inside closed housing that well lit by top-mounted LED panels making the images taken by three high-speed global shutter cameras consistent. The cameras distributed between the arms to ensure objects always seen by any camera. We study dexterous manipulation of a cube whose pose is estimated by a visual tracking system with 10 Hz.
2

---

Published as a conference paper at ICLR 2023
training
data collection
offline learning
Pybullet simulation
sim- datasets
Isaac parallel simulation
offline
evaluation
PPO
real- datasets
offline RL
domain randomization
real TriFinger
evaluation
Figure 2: Overview of our approach. Policies are trained with domain randomization in parallel simulation using Isaac Gym (Makoviychuk et al., (022)) and then deployed in the PyBullet (Coumans & Bai 2016) simulation and the real system without fine-tuning to collect the datasets. We train state-of-the-art offline RI. algorithms on these datasets and evaluate them on the respective system, i.e the simulator or the real-robot cluster.
To abstract away the low-level details, we developed software with a simple Gym (Brockman et al. 2016) interface in Python that interact with the robots maximal rate of kHz in position control or torque control mode (see (Wüthrich et al.) 2021) for details). We use a control frequency of Hz and torque control for this work. We have custom object tracking tool to provide position and orien- tation of a single colored cube in the environment, allowing algorithms to work without visual input.
On top of this interface, have developed a submission system that allows users to submit jobs to cluster of these robots for unattended remote execution. This setup was specifically adapted for ease of use in the offline RL setting and extensively tested. We will provide researchers with access the robot cluster, which will allow them to evaluate the policies they trained on the dataset proposed herein. To ease development and study the fundamental differences between simulated and real world data, we provide a corresponding simulated environment using PyBullet (Coumans & Bai 2016).
To summarize, the hardware and software have the following three key properties: i) physically capable of dexterous manipulation: ii) robust enough for running and evaluating learning methods, and iii) easy to use (robot hardware and simulator) and integrated in existing code frameworks.
3 THE TRIFINGER DATASETS
We consider tasks that involve a colored cube: pushing the cube a target location and lifting the cube to desired location and orientation. To create behavioral datasets for these tasks on the TriFinger platform, we need expert policies, which we obtain using reinforcement learning parallel simulation environment with domain randomization. Fig. Fig.2] visualizes the entire procedure from training to data collection to offline learning. In this section, we describe the tasks and the data collection while Sec. 4 is dedicated to benchmarking offline RI. algorithms on the collected data.
3.1 DEXTEROUS MANTPULATION TASKS
We consider two tasks on the TriFinger platform that require dexterous manipulation cube:
Push The goal is to move the cube to a target location. This task does not require the agent to align the orientation of the cube; the reward based only on the desired and the achieved position.
Lift The cube has to be picked up and moved to target pose in the air which includes position and orientation. This requires flipping the cube on the ground, obtaining a stable grasp, lifting it to a target location and rotating it to match the target orientation.
Following prior work (Hwangbo et al. 2019 Allshire et al. 2022) we define the reward by applying a logistic kernel & (xx) = (b - 2) (exp(a [=]] + + exp(-a) xxl)) to the difference between desired and achieved position (for the Push task) the desired and achieved corner points of the cube (for
3

---

Published as a conference paper at ICLR 2023
the Lift task). The parameters and b control the length scale over which the reward decays and how sensitive is for small distances 2, respectively. This yields a smooth, dense and bounded reward, see Appendix B.3 details. We define success in the pushing task reaching the goal position with a tolerance of 2cm For the lifting task we additionally require to not deviate more than 22 degrees from the goal orientation.
We note that pushing is more challenging than it may appear due to inelastic collisions between the soft fingertips and the cube which are not modeled in rigid body physics simulators. Furthermore, the performance of policies can be quite sensitive to the value of sliding friction. Lifting is, however, even more challenging as flipping the cube based on a noisy object-pose with a low refresh rate is error- error-prone, learning the right sequence of behaviors requires long- term credit assignment and dropping the cube often results in loosing all progress in an episode.
3.2 TRAINING EXPERT POLICIES
We train expert policies with online RL in simulation which we then use for data collection on the real system (Fig. 2 We build on prior work which achieved sim-to-real transfer for a dexterous manipulation task on the TriFinger platform (Allshire et al.] 2022). This approach replicates the real system in fast, GPU-accelerated rigid body physics simulator (Makoviychuk et al., 2027) and trains with an optimized implementation (Makoviichuk & Makoviychuk,2022 (222) of Proximal Policy Optimization (Schulman al. 2017) with a high number of parallel actors. We furthermore adopt their choice of a control frequency of 50 Hz and use torgue control to enable direct control of the fingers. The sensor input is the proprioceptive sensor information and the object pose. In order to obtain policies that are robust enough to work on the real system, domain randomization (Tobin et al.) 2017, Mandlekar et al.) 2017 Peng et al. 2018) is applied: for each episode, physics parameters like friction coefficinnts are sampled from distribution that is likely to contain the parameters of the real environment. Additionally, noise is added to the observations and actions to account for sensor noise and the stochasticity of the real robot. Furthermore, random forces applied to the cube to obtain policy that is robust against perturbations, as in (Andrychowicz et al. (020). The object and goal poses represented by keypoints, i.e., the Cartesian coordinates of the corners of the cube. This choice was empirically shown to accelerate training compared to separately encoding position and orientation in Cartesian coordinates and quaternion (Allshire et al. 2022).
To improve the robustness of the trained policies across the different robot instances and against other real-world effects like tracking errors due to accumulating dust, we modified the implementation of Allshire et al. (2022) in several ways. While the original code correctly models the 10 Hz refresh rate of the object pose estimate, we found it beneficial to also simulate the delay between the time when the camera images are captured and when they are provided to the agent. This delay typically ranges between 100 ms and 200 ms. We furthermore fully randomize the initial orientation of the cube and use hand-crafted convex decomposition of the barrier collision mesh to avoid artifacts of the automatic decomposition, to which the policy can overfit. For the pushing task we penalized rapid changes in the cube position and orientation sliding and flipping the cube transfers less well to the real system than moving it in a slow and controlled manner. We observed that policies trained with RL in simulation tend to output oscillatory actions, which does not transfer well to the real system and causes stronger wear effects (Mysore et al. 2022]. To avoid this, we penalized changing the action which led to smoother movements and better performance. For the Lift task we additionally consider a policy which was trained with an Exponential Moving Average on the actions as we observed that vibrations on the real robot can lead to slipping and dropping (see Fig. S14(b)) These vibrations might be caused by elastic deformations of the robot hardware and the complex contact dynamics between the soft fingertips and the cube that are not modeled in simulation. As also observed in Wang et al. (2022), the performance on the real system varies significantly with training seeds. We evaluated 20 seeds on the real system and used the best one for data collection. More details on training in simulation can be found in the Appendix C.1
3.3 DATA COLLECTION
We collected data for the pushing and lifting tasks both in PyBullet (Coumans & Bai 2016) simulation (Jshh et al. (020) and on the real system, as shown in Fig. 2] To ensure that the data collection procedures are identical, we run the same code with the simulator backend and with the
4

---

Published as a conference paper at ICLR 2023
Figure 3: An example behavior of our expert policy on the lifting task. The robot is grasping and reorienting the object to reach the desired target location and orientation (transparent cube).
real-robot backend on six TriFinger platforms (Fig. We observed that policies learned in simulation struggle with retrieving the cube from the barrier on the real robo [ To alleviate this problem, we employ a predefined procedure to push the cube away from the barrier between episodes. The resulting starting positions together with the target positions visualized in Fig. S12 During data collection, self-tests are performed at regular intervals to ensure that all robots are fully functional.
For the dataset to be useful to the community, it should be possible to evaluate offline-lcarned policies on the real robots. As machine learning models can be computationally demanding, we wait for a fixed time interval between receiving a new observation and starting to apply the action based this observation. This time budget is allocated for running the policy without deviating from the control frequency used for data collection. We choose 10 ms for the Push task and 2 ms for the Lift task as we found training with bigger delays difficult. Note that our expert policies run in less than 1 ms.
We provide as much information about the system as possible in the observations. The robot state is captured by joint angles, angular velocities, recorded torques, fingertip forces, fingertip positions, fingertip velocities (both obtained via forward kinematics), and the ID of the robot. The object pose is represented by position, a quaternion encoding orientation, the keypoints, the delay of the camera images for tracking and the tracking confidence. The observation additionally contains the last action, which is applied during the fixed delay between observation and action. The desired and achieved goal are also included in the observation and contain either position for the Push task or keypoints for the Lift task. Some observations provide redundant information; however, we believe this simplifies working with the datasets, as we provide user- seer-frienlly Python code that implements filtering as well as automatically converting observations publish a version of each dataset with camera images from three viewpoints. We helieve that directly learning from these image datasets is an exciting challenge for the community and that they are moreover valuable in their own right as large collection of robotic manipulation video footage.
flat array. Moreover, we additionally
For each task we consider pure expert data (Expert), mixed data recorded with a range of training checkpoints (Mixed), a combination of 50% expert trajectories and 50% trajectories recorded with a weaker policy with additive Gaussian noise on the actions (Weak&Expert) and the 50% expert data in Weak&Expert (Half-Experl) We run the same policies in simulation and on the real system. An exemplary expert behavior for the Lift task shown Fig. 3 Episodes last for 15 for the Push task and 30 s for the Lift task as reorienting, grasping and lifting the cube requires more time. For the Push task, we collect 16 h of interaction for each dataset, corresponding to 3840 episodes and 2.8 million transitions. For the more demanding Lift task, we collect 20 of robot interactions corresponding to 2400 episodes and 3.6 million transitions. The Half-Expert datasets contain the expert data that included in the corresponding WeaerExpert datasets to isolate the effects of reducing the amount of expert data and adding suboptimal trajectories. The average success rates are provided next to the offline learning results in Table T and 2 (see data column). In Appendix B we give detailed analysis along with a specification (Table S4) and statistics (Table of the offline RL. datasets.
4 BENCHMARKING OFFLINE REINFORCEMENT LEARNING ALGORITHMS
We benchmark offline RL algorithms on pairs of simulated and real datasets and study the impact of data quality their performance. We limit our evaluation to the best algorithms provided in the
Possibly hecause the object tracking performance is slightly worse at the harrier or the dynamics of the outstretched fingers aligns less well between simulation and reality.
2 At the time of writing cannot henchmark these datasets since the robot cluster does not provide GPU-access the moment. This will likely change in the near future.
5

---

Published as a conference paper at ICLR 2023
Push
Lift
Success return
Success return
1.0
1.0
0.5
0.0
0.0
CRR AWAC CQL
IQL
BC
CRR AWACT CQL' IQL'
Figure 4: Average normalized success rates and returns (light colors) on real robots. Each
quantity is normalized by the dataset mean and averaged over all Push or Lift datasets.
Table Pushing: Success rate on the TriFinger-Pushs Datasets. 'data' denotes the mean over the dataset. Average and standard deviation over five training seeds. A star indicates significance w.r.t.
all other methods using Welch's (-test with p < 0.05.
Push-Datasets
data
BC
CRR
AWAC
CQL.
IQL.
Sim- Sim-Expert
0.95
0.83 1 0.02
0.94 0.04
0.92
0.03
0.03 1 0.01
0.88
0.04
Sim-Half-Expe Exper-Expert
0.95
0.71 + 0.05
0.79
0.05
0.79 0.02
0.05 0.02
0.70 0.06
Sim- HeneWeakkEExerr
0.53
0.53 + 0.09
0.88
0.03 0.09
0.17
0.66 -
Sim-Mixed
0.76
0.53 + 0.04
0.10
0.84 0.06
0.02 + 0.01
Real-Expert
0.92
0.74 0.05
0.007
0.80 0.03
0.54 0.13
0.75 0.08
Real-Half-Fxpert
0.92
0.66 + 0.08
0.78 0.04
0.76
0.48 0.08
0.70 0.08
Fxpert
0.51
0.48 + 0.10
0.84 0.06"
0.69 0.06
0.14 0.04
0.68 0.05
Real-Mixed
0.49
0.29 + 0.06
0.30 0.06
0.61 - 0.09
0.02 + 0.02
0.66 0.08
open-source library d3rlpy (Seno & Imai 2021) keep the required robot time for the evaluation of the trained policies manageable. Namely, we benchmark the following algorithms: BC (Bain &
Sammut [995 Pomerleau 1991, Ross et al., 2011 Torabi et al. 2018). CRR (Wang al. 20200)
AWAC (Nair et al. (020)) CQL (Kumar et al. (020). and IQL. (Kostrikov et al. 2021b). We report
results for two sets of hyperparameters: The default values (except for CQL for which we performed grid search on Push-Sim--Exert the default parameters did not learn) and the results of a grid search on Lift-Sim-Weak&Baae Weak&&xpert (marked by !). Details about the hyperparamters and their optimization in Appendix C.3 We think that the performance the default hyperparameters is highly relevant for offline RL on real data as optimizing the hyperparameters without a simulator is often infeasible.
4.1 RESULTS
We train with five different seeds for each algorithm and evaluate with a fixed set of randomly sampled goals. Details about the policy evaluation the simulated and real set-up can be found in Appendix D We report success rates in the main text and returns in Appendix A
The benchmarking results for the Push task are summarized in Table Most of the offline RL algorithms perform well but the performance on the real data is generally worse. An exception to this pattern is CQL which performs poorly on the simulated Push task and gains some performance the real data, perhaps due to the broader data distribution of the stochastic real-world environment. As expected BC performs well on the expert datasets but cannot exceed the dataset success rate on the Wxpr& data, unlike CRR, AWAC, and IQL. On the expert data, CRR and AWAC match the performance of the expert policy in simulation but fall slightly behind on the real data. Interestingly. the performance of CRR is not negatively impacted by weak trajectories (e.g. 84% on Real-Weak& W compared to 78% on Real Real-Half-Eai Half-Exxprt, where the latter only contains the expert data portion). We provide insights into how the behavior policy compares to offline-trained policy for the Push task in Fig. 6 (b) and (c).
The more challenging Lift task separates the algorithms more clearly summarized in Table 2 CQL does not reach non-zero success rate at all, despite our best efforts to optimize the hyperparameters (see Appendix C.3) This is in line with the results reported in Mandlekar et al. (2021) where CQL failed to learn on the datasets corresponding to more complex manipulation tasks. Kumar et al. (002)) furthermore discusses the sensitivity of CQL to the choice of hyperparameters, especially data. While the best algorithms come close to matching the performance of the expert on Lift-Sim- Expert, they fall short of reaching the dataset success rate the real-robot data (Lift-Real- Feal--r..
robotic

---

Published as a conference paper at ICLR 2023
Table 2: Lifting: Success rate the TriFinger-Liff Datasets. 'data' denotes the mean the dataset. Average and standard deviation over five training seeds. Experiments with hyperparameters optimized Sim-Weak &eak are marked with a t. Stars and indicate significance w.r.t. second best Welch's (-test with p < 0.05 and p < 0.01, respectively.
Lift-Datasets
data
BC
CRR
AWAC
CQI.
IQI.
Sim-Expert
0.87
0.64 1: 0.00
0.80
0.03
0.75 1 0.04
0.00 i 0.00
0.47 0.06
Sim-Half-Expert
0.88
0.64 + 0.02
0.78 + 0.02"
0.69 = 0.05
0.00 = 0.00
0.04 0.01
Hon-Weak&EExpee
0.5
0.16 + 0.04
0.43 1:3=000 0.35
0.55 0.09
0.24 0.00
Sim-Mixed
0.68
0.01 + 0.01
0.19 0.06
0.41 0.09
0.00
0.00
0.00 + 0.00
Sim-Expert
0.87
0.80 1 0.04
0.84 0.01
0.84 1 1.00
0.01
000.0.000
0.80 0.03
Sim-Half-Expent
0.88
0.67 + 0.09
0.75 0.08
0.82
0.00
0.75
Sin-Weak&Expert
0.5
0.47 + 0.03
0.69 0.04
0.54 0.03
0.00
0.64 0.05
Sim-Mixed'
0.68
0.01 +: 0.00
0.20 0.07
0.32 + 0.04
0.00 1.00
0.01 0.01
Scal--Smothh-Expet
0.64
0.26 + 0.09
0.52
0.12"
0.24 0.00
Real-Expert
0.66
0.27 1 0.09
0.57
0.07**
0.24 0.004
0.00 0.00
0.29 1 0.00
Real-Half-Experr
0.68
0.15 + 0.01
0.38 0.08
0.12
0.12 0.06
Real-Weak Expee
0.40
0.02
+ 0.04
0.07
0.03 0.06
0.00 = 0.00
0.13
Real-Mixed
0.42
0.00 0.01
0.02 0.08
0.01
0.00 = 0.00
0.03 = 0.02
Real-Exper Expl-Expert
0.66
0.28 1 0.04
0.54 -.09
0.31 1 0.04
0.00 1 0.00
0.48 0.07
Soal-Half-Experee
0.68
0.25 + 0.07
0.37
0.36 0.06
0.01 = 0.01
0.30 0.06
Real- Expert!
0.40
0.09 + 0.04
0.07
0.12 0.06
0.00 = 0.00
0.15 0.03
Real-Mixed
0.42
0.00 + 0.00
0.18+0004"
0.01 0.01
0.00 0.00
0.02 + 0.01
Lift-Sim-Expert
Lift-Sim-Half-EEpeer
1.00
1.00
1.00
success rate
SUccess rate
success rate
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
100
200
100
200
100
200
epochs
epochs
epochs
BCt
CRR
AWACI
CQL
IQL'
Figure 5: Success rates during offline RL training on the simulated Lift task. The black dashed
line indicates the performance of the expert dataset, and the red dashed line depicts average success rate of the Weak&Expee dataset. Shaded areas indicate the interval between the 0.25 and 0.75 quantiles. Learning curves for all experiments are shown in Appendix A.6
On the Seal-SSmoth--Ewwe -xppert dataset the success rates are, on average, slightly lower, probably due to the non-Markovian behavior policy. The returns are generally higher, however, likely due to the smoothed expert reaching higher returns (see Table S2)
On the Lift-Weak&& Expert datasets all algorithms perform significantly worse than on the expert data. This effect is most pronounced on the real robot data and calls the ability of the algorithms to make good use of the 50% expert trajectories into question. To verify that the performance drop is not due to only half of the expert data being available, we also train solely on the expert trajectories contained in the Weak&Expert dataset. We refer this dataset as Half-Expert. We find that the resulting policy performs significantly better, ruling out that the performance drop on the Lift- & dataset exclusively caused by lack of data. Fig. [5] shows that this true for the simulated Lift datasets as well. We conclude that the benchmarked offline RI algorithms still exhibit BC-like behavior, i.e., they are distracted by suboptimal data. Their poor performance on the Mixed datasets further underpins their reliance imitation of consistent expert all algorithms struggle to learn from the data collected by a range of training checkpoints (see tables [ and 2 and learning curves in section A.6).
The Lift datasets, in particular those recorded the real robots, generally contain more useful sub-trajectories than apparent from the success rate, which is defined as achieving the goal pose at the end of an episode. For example, during 87% of all episodes in the Lift-Real-Exxn -xpe-- dataset the goal pose is achieved at some point but the best algorithm CRR achieves a success rate of only 57%. This suggests that these datasets have untapped potential for trajectory stitching. As the observations
7

---

Published as a conference paper at ICLR 2023
(a) Impact of noise
(b) Speed of the cube
(c) Momentary success
momentary success
success rate
0.75
cube speed (m/s]
0.50
0.5
2
0.25
0.00
0.0
0.0
0.5
1.0
1.5
2.0
0
2
5
10
relative noise scale
[s]
[s]
Expert
BCt
CRR
AWACI
CQL
IQL
Figure 6: Analysis of policy behavior. (a) Success rate for simulated lifting after training on data with varying relative environment noise scale (1.0 corresponds to the noise level of the real system). The dashed lines indicate the performance on the real system. (b) Average speed of the cube over a pushing episode on the real system. Some offline RL algorithms learn move the cube faster than the expert. (c) Momentary success during a pushing episode on the real system: the fraction of episodes during which the goal was achieved a given point in time (trained on expert data).
furthermore noisy and do not contain estimates of the velocity of the cube, training recurrent policies could potentially lead to an increase in performance.
To investigate the impact of noise the performance of offline RL policies and the expert, collected datasets in simulation for up to twice the noise amplitudes measured on the real system. Fig. 6 (a) shows that the performance of the expert and the offline RL policies degrades only slowly when iecrasiin the noise scale, ruling out that noise the sole explanation for the performance gap between simulated and real system. As delays in the observation and action execution are already implemented in the simulated environment, we conclude that other factors like more complex contact dynamics and elastic deformations of the fingertips and robot limbs are likely causing the larger performance gap between data and learned policies on the real robots.
To how well the policies learned from the datasets generalize over instances of the robot hardware, we evaluated on hold-out robot which was not used for data collection. We did not see significant difference in performance (see Appendix [A.3] for details on this experiment) suggesting that the datasets cover enough variations in the robot hardware to enable generalization to unseen robots.
In summary, CRR and AWAC generally perform best on the proposed datasets with IQI. also being competitive after hyperparameter optimization. The performance gap between expert and offline RL. in general bigger on the real system, perhaps due more challenging dynamics.
5 RELATED WORK
Offline RL:
The goal of offline RL (Levine al. 2020 Prudencio al. 2022) is learn effective
policies without resorting to an online interaction by leveraging large and diverse datasets covering sufficient amount of expert transitions. This approach is particularly interesting if interactions with the environment are either prohibitively costly or even dangerous.
Offline RL faces fundamental challenge, known as the Distributional Shift (DS) problem, originating from two sources. There is a distribution mismatch in training, as we use the behavioral data for training, but the learned policy would create different vist-viistttionn distribution. The second problem is that during policy improvement, the learned policy requires an evaluation of the Q-function on unseen actions (out of distribution). An over-estimation the Q-value on these -iitt--distribution samples leads to learning of a suboptimal policy.
Several algorithmic schemes were proposed for addressing the DS problem: i) constraining the learned policy to be close to the behavior data (Fujimoto et al., 2019: Kumar et al.) 2019b] Zhang ct al. 2021 Kostrikov et al. 2021a); ii) enforcing conservative estimates of future rewards (Kumar al. 2020 Yu et al. 2021 Cheng al. 2022); and iii) model-based methods that estimate the uncertainty via ensemble (Janner et al. 2019 Kidambi et al.) 2020).
Additionally, other approaches include: implicitly tackling the DS problem via (advantage- wedaantage-weighte))
variants of behavioral cloning (Nair et al.] 22000) Wang et al., 2020;; Chen et al., 22000] FFjimoto
& Gu 2021) completely bypassing it either by removing the off-policy evaluation and
8

---

Published as a conference paper at ICLR 2023
performing constrained policy improvement using an on-policy Q estimate of the behavior policy (Brandfonbrener et al., [022]. or by approximating the policy improvement step implicitly by learn- ing on-data state value function (Kostrikov al., 2021b). An orthogonal research line considers combining importance sampling and off-policy techniques (Nachum et al. 2019h at Zhang et al.) 2020 Xu al.) (00TT) and recently {Chen et al.] 2021: Janner al. 2021) investigated learning an optimal trajectory distribution via transformer architectures.
leaanin
The simplest strategy for learning from previously collected data is behavioral cloning (BC), which is fitting a policy to the data directly. Offline RL can outperform BC: i) on long-horizon tasks with mixed expert data, e.g., trajectories collected via an expert and a noisy-expert; and ii) with expert or near expert data, when there is a mismatch between the initial and the deployment state distribution.
RL for dexterous manipulation: Reinforcement learning was recently successfully applied to dexterous manipulation on real hardware (OpenAl et al. 2018
et al. 2022 Wang et al.
2022). These results rely training with online RL in simulation, however, and consequently limited by the fidelity of the simulator. While domain randomization (Tobin et al. 2017 Mandlekar et al. 2017; Peng al. (018) can account for mismatch between simulation and reality in terms of physics parameters, it cannot compensate oversimplified dynamics. The challenges of real-world environments have been recognized and partly modeled in simulated environments (Dulac-Arnodd et al. 2020). To overcome the sim-to-real gap entirely, however, data from real-world interactions is still required, in particular for robotics problems involving contacts.
Offline RT datasets: While offline RI. datasets with data from simulated environments, like D4RL (Fu et al. (2000) and RL Unplugged (Gulcehre et al. 22220) have propelled the field forward, the lack of real- world robotics data has been recognized (Behnke, (2066] /Bonsignorio & del Pobil
2015; Calli al., 2015 Amigoni al. 2015 Murali al.) 2019). Complex contact dynamics, soft deformable fingertips and vibrations are particularly relevant for robotic manipulation but not modeled sufficiently well in simulators used by the RL community (Todorov et al. [2012] Makoviychuk et al. [021] Freeman et al., 2021). Recently, three small real-world datasets with with gripper (using operational space control) have been proposed (Mandlekar et al., 202T). Two of them require only basic lifting and dropping while the third, more challenging task could not be solved with the available amount of data. For more challenging low-level physical manipulation, a dataset suitable for offline RL. is still missing. We therefore provide the first real-robot dataset for dexterous manipulation which sufficiently large for offline RL (one order of magnitude more data on real robots than prior work (Mandlekar et al., 202I and for which learned policies can easily be evaluated remotely on a real-robot platform.
human demonstrations for robot
Our hardware platform is open source. Other affordable robotic open-source platforms are, for instance, manipulator (Yang al., 2019). a simple robotic hand and quadruped (Ahn et al. 2020). Since it is hard to set up and maintain such platforms, we provide access to our real platform upon request, and hope that this will bring the field forward.
Affordable open-source platforms:
Remote benchmarks: For mobile robotics, Pickem et al. (2017 propose the Robotarium, a remotely accessible swarm robotics research platform, Kumar et al. (2019a) offer OffWorld gym consisting of two navigation tasks with a wheeled robot. Similarly, Duckietown (Paull et al. 2017) hosts the AI Driving Olympics (Al-DO-team) 2022).
CONCLUSION
We present benchmark datasets for robotic manipulation that are intended to help improving the sfate-of-the-at in offline reinforcement learning. To record datasets, we trained capable policies using online learning in simulation with domain randomization. Our analysis and evaluation on two tasks, Push and Lift, show that offline RI. algorithms still leave room for improvement on data from real robotic platforms. We identified two factors that could translate into increased performance datasets: trajectory stitching and robustness to non-expert trajectories. Further, our analysis indicates that noise and delay alone cannot explain the larger gap between dataset and offline RL performance on real systems, underpinning the importance of real-robot benchmarks.
We invite the offline RL community train their algorithms with the new datasets and test the empirical performance of the latest offline RL algorithms, e.g. (Kostrikov et al.) 2021a al Xu et al.
2021 Kumar al., 2021: Cheng al. 2022), on real-robot hardware.

---

Published as a conference paper at ICLR 2023
AUTHOR CONTRIBUTIONS
N.G., S.BI., P.K., M. W. S.Ba., B.S., and G.M. conceived the idea, methods and experiments. G.M. initiated the project, M.W. S.Ba. and B.S. conceived the robotic platform. F.W. implemented the low-level robot control and parts of the submission system. N.G. trained the expert policies and created the datasets. N.G., S.BI., and P.K. conducted the offline RL experiments, collected the results and analyzed them under the supervision of G.M. N.G. ran all experiments on the real systems. N.G. and F.W. wrote the software for downloading and accessing the datasets. N.G., S.BI., P.K., F.W. and G.M. drafted the manuscript, and all authors revised it.
ACKNOWLEDGMENTS
We are grateful for the help of Thomas Steinbrenner in repairing and maintaining the robot cluster. Moreover, feedback by Arthur Allshire training expert policies in simulation and by Huanbo Sun on domain randomization proved valuable. We acknowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 01IS18039B) Georg Martius is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645. Pavel Kolev was supported by the Cyber Valley Research Fund and the Volkswagen Stiftung (No 98 571). We thank the anonymous reviewers for comments which helped improve the presentation of the paper.
REPRODUCIBILITY STATEMENT
We publish the datasets we propose as benchmarks (sections 3.3 and B and provide access to the cluster of real TriFinger platforms (section 2) we used for data collection. Submissions to the cluster do not require any robotics experience and can be made in the form of a Python implementation of a RL policy (section B.5). simulated version of the TriFinger platform (Joshi et al.] 2020) and a low-cost hardware variant furthermore publicly available open source (Wüt Wüthrich et al. (021). For offline RL training we moreover use open-source software (Seno & Imai] [022]. Finally, we describe hyperparameter optimization in detail and provide the resulting hyperparameters in section
REFERENCES
Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 1300-1313, PMIR, 30 Oct-O1 Nov 2020.
AI-DO-team. The ai driving olympics (ai-do), httppp//ww..acckeettt...rrreeeee AI Dew-D-ivvin-oyyyp..... 2022.
Arthur Allshire, Mayank Mittal, Varun Lodaya, Viktor Makoviychuk, Denys Makoviichuk, Felix Widmaier, Manuel Wüthrich, Stefan Bauer, Ankur Handa, and Animesh Garg. Transferring dexterous manipulation from GPU simulation remote real-world trifinger. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 11802-11888 1809, 2022.
Francesco Amigoni, Emanuele Bastianelli, Jakob Berghofer, Andrea Bonarini, Giulio Fontana, Nico Hochgeschwender, Luca locchi, Gerhard Kraetzschmar, Pedro Lima, Mattco Matteucci. Pedro Miraldo, Daniele Nardi, and Viola Schiaffonati. Competitions for benchmarking: Task and functionality scoring complete performance assessment. IEEE Robotics & Automation Magazine, 22(3):53-61, 2015.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in- hnnd manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
10

---

Published as a conference paper at ICLR 2023
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Koichi Furukawa, Donald Michie, and Stephen H. Muggleton (eds.), Machine Intelligence 15, Intelligent Agents [St. Catherine's 's College, Oxford, UK. July 1995], pp. 103-129. Oxford University Press, 1995.
Stefan Bauer, Manuel Wüthrich, Felix Widmaier, Annika Buchholz, Sebastian Stark, Anirudh Goyal, Thomas Steinbrenner, Joel Akpo, Shruti Joshi, Vincent Berenz, Vaibhav Agrawal, Niklas Funk, Julen Urain De Jesus, Jan Peters, Joe Watson, Claire Chen, Krishnan Srinivasan, Junwu Zhang, Jeffrey Zhang, Matthew Walter, Rishabh Madan, Takuma Yoneda, Denis Yarats, Arthur Allshire, Ethan Gordon, Tapomayukh Bhattacharjee, Siddhartha Srinivasa, Animesh Garg. Takahiro Maeda, Harshit Sikchi, Jilong Wang. Qingfeng Yao, Shuyu Yang, Robert McCarthy, Francisco Sanchez, Qiang Wang. David Bulens, Kevin McGuinness, Noel O'Connor, Redmond Stephen, and Bernhard Schölkopf. Real robot challenge: A robotics competition the cloud. In Douwe Kiela, Marco Ciccone, and Barbara Caputo (eds.), Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, volume 176 of Proceedings of Machine Learning Research, pp. 190-204. PMLR, 06-14 Dec 2022.
Sven Behnke. Robot competitions-ideal benchmarks for robotics research. In Proc. of IROS-2006 Workshop Benchmarks in Robotics Research. Institute of Electrical and Electronics Engineers (IEEE), 2006.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06688 2019.
Fabio Bonsignorio and Angel P. del Pobil. Toward replicable and measurable robotics research [from the guest editors]. IEEE Robotics & Automation Magazine, 22(3):32-35, 2015.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off-policy evaluation. M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4933 4946. Curran Associates, Inc., 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint
2016.
Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols. arXiv:1502.03144 February 2015.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 15084-15097. Curran Associates, Inc., 2021.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. BAIL: hest-action imitation learning for batch deep reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18353-18363,
Curran Associates, Inc., 2020.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal, Adversarially trained actor critic for offline reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 3852-3878. PMLR, 17-23 Jul 2022.
Erwin Coumans and Yunfci Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet .org..ttt...o.......oo 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /11.:0480008 2018.
11

---

Published as a conference paper at ICLR 2023
Gabriel Dulac-Amold, Nir Levine, Daniel J. Mankowitz, Jenyy Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges real-world reinforcement learning. CoRR, abs/2003. 1881, 2020.
Luciano Floridi and Massimo Chiriatti. Gpt-3; Its nature, scope, limits, and consequences. Minds and Machines, 30(4):681-694, 2020.
C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax- a differentiable physies engine for large scale rigid body simulation, 2021. URL h
7/gi figithub... com goog e/ e/brax
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020.
Scott Fujimoto and Shixiang (Shane) Gu. A minimalist approach to offline reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 20132-20145. Curran Associates, Inc., 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2052-2062 PMLR, 09-15 Jun 2019.
Felix Grimminger, Avadesh Meduri, Majid Khadiv, Julian Viereck, Manuel Wüthrich, Maximilien Naveau, Vincent Berenz, Steve Heim, Felix Widmaier, Jonathan Fiene, Alexander Badri-Spröwitz, and Ludovie Righetti. An Open Forque-Controlle Modular Robot Architecture for Legged Locomotion Research. In International Conference on Robotics and Automation (ICRA), 2020.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 01477288-2599 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 770-778, 2016.
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, a(26))2a2, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 1273-1286. Curran Associates, Inc., 2021.
Shruti Joshi, Felix Widmaier, Vaibhav Agrawal, and Manuel Wüthrich.
com/
fingeriiiirrrreee pateeerssmmlattiioo 2020.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- based offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (cds.). Advances in Neural Information Processing Systems, volume 33, pp. 21810-21823, Curran Associates, Inc., 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 7783, PMLR, 2021a. URL htLp:tt::. http:///ooceeeiiie ngs. ngs /ttprocceccnooo.or.rrrrres/////s wwwpprss///1///ssrrrriss.
.hmml
12

---

Published as a conference paper at ICLR 2023
Iyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. CoRR, abs/2110.061166 2021b.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Pereira, CJ C Burges, L. Bottou, and K Q Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105, Curran Associates, Inc.. 2012.
Ashish Kumar, Toby Buckley, John Lanier, Qiaozhi Wang, Alicia Kavelaars, and Ilya Kuzovkin. Offworld gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research. arXiv preprint SEGG::1910.00000 2019a.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- Buch Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 1761-11771, 2019b.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Conference on Robot Learning, 8-11 November 2021, London, UK, volume 164 of Proceedings of Machine Learning Research, pp. 417-428. PMLR, 2021.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pp. 45-73. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01 2020.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.
Denys Makoviichuk and Viktor Makoviychuk. rl-games: A high-performance framework for rein- forcement
games May 2022.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo. Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:21086 10470, 2021.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Adversarially robust plausible perturbations. In IEEE Int'
policy learning through active construction of physically-plausbble Conf. on Intelligett Robots and Systems (IROS), volume 16, 2017.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni. Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin What matters in learning from offline human demonstrations for robot manipulation. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Conference on Robot Learning, 8- 3 November 2021, London, UK, volume 164 of Proceedings of Machine Learning Research, pp.
PMLR, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-523 533, February 2015.
13

---

Published as a conference paper at ICLR 2023
Adithyavairavan Murali, Tao Chen, Kalyan Vasudev Alwala, Dhiraj Gandhi, Lerrel Pinto, Saurabh Gupta, and Abhinav Gupta. PyRobot: An Open-source Robotics Framework for Research and Benchmarking, arXiv: 1906.08236, June 2019.
Siddharth Mysore, Bassel Mabsout, Renato Mancuso, and Kate Saenko. Regularizing action policies for smooth control with reinforcement learning. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 1810-1816. IEEE, 2021.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer. Florence d' Alché Buc, Emily B. Fox, and Roman Gamett (eds.). Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 2315-2325, 2019a.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. AlgaeDICE: Policy gradient from arbitrary experience. CoRR, abs/ 1bs//192.00274, 2019b.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. CoRR, abs/2006.09359) 2020.
OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning Dexterous In-Hand Manipulation. arXiv: 1XXiv:1808.0017. August 2018.
OpenAI, Ilge Akkaya, Marein Andrychowicz, Maciek Chociej. Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving Rubik's ' Cube with Robot Hand. arXiv:1910.0711 October 2019.
Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. Time limits in reinforcement learning. In International Conference Machine Learning, pp. 4054. PMLR, 2018
Liam Paull, Jacopo Tani, Heejin Ahn, Javier Alonso-Mora, Luca Carlone, Michal Cap. Yu Fan Chen, Changhyun Choi, Jeff Dusek, Yajun Fang, et al. Duckietown: an open, inexpensive and flexible platform for autonomy education and research. 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1497-1504. IEEE, 2017.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 3803-3810. IEEE, 2018
Daniel Pickem, Paul Glotfelter, Li Wang, Mark Mote, Aaron Ames, Eric Feron, and Magnus Egerstedt. The robotarium: remotely accessible swarm robotics research testbed. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1699-1706. IEEE, 2017.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991.
Rafael Figueiredo Prudencio, Marcos R. O. A. Maximo, and Esther Luna Colombini. A survey offline reinforcement learning: Taxonomy, review, and open problems. CoRR, abs/2203.01387, 2022.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Stéphane Ross, Geoffrey J. Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Geoffrey J. Gordon, David B. Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, USA, April 11-13, 2011, volume of JMLR Proceedings, pp. 627-635. JMLR.org, 2011.
14

---

Published as a conference paper at ICLR 2023
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library. CoRR, abs/211 11.071,,.788, 2021.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676)3354 339,339, 2017.
Richard S Sutton, Andrew G Barto, and Co-Director Autonomous Learning Laboratory Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ international conference intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017. doi: 10. 1..11099ROSS200 201R 2017.0118700 188221133.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Jérôme Lang (ed.), Proceedings of the Twenty-Seventh International Joint Conference Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pp. 4950-4957. ijcai.org, 2018.
Qiang Wang, Francisco Roldan Sanchez, Robert McCarthy, David Cordova Bulens, Kevin McGuin- ness, Noel O'Connor, Manuel Wüthrich, Felix Widmaier, Stefan Bauer, and Stephen Redmond. Dexterous robotic manipulation using deep reinforcement learning and knowledge transfer for complex sparse ewardd-basd tasks. Expert Systems, n/a(n/a):e1 2022.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh Merel, Jost Tobias Springenberg, Scott E. Reed, Bobak Shahriari, Noah Y. Siegel, Çaglar Gülçehre, Nicolas Heess, and Nando de Freitas. Critic regularizeed regression. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Manuel Wüthrich, Felix Widmaier, Felix Grimminger, Shruti Joshi, Vaibhav Agrawal, Bilal Ham- moud, Majid Khadiv, Miroslav Bogdanovic, Vincent Berenz, Julian Viereck, Maximilien Naveau, Ludovic Righetti, Bernhard Schölkopf, and Stefan Bauer. Trifinger: An open-source robot for learning dexterity. In Proceedings of the 2020 Conference on Robot Learning (CoRL), volume 155, pp. 1871-1882. PMLR, 2021.
Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. CoRR, abs/2110.07395. 2021.
Brian Yang, Jesse Zhang, Vitchyr Pong, Sergey Levine, and Dinesh Jayaraman. Replab: A repro- ducible low-cost arm benchmark platform for robotic learning. arXiv preprint MoXiv:9905.074.. 2019.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: conservative offline model-based policy optimization. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NewrlPS 2021, December 6-14, 2021, virtual, pp. 28954-28967 2021.
Chi Zhang, Sanmukh R. Kuppannagari, and Viktor K. Prasanna. BRAC+: improved behavior regularized actor critic for offline reinforcement learning. In Vineeth N. Balasubramanian and Ivor W. Tsang (eds.), Asian Conference on Machine Learning, ACML 2021, 17-19 November 2021, Virtual Event, volume 157 of Proceedings of Machine Learning Research, pp. 204-219. PMLR, 2021.
15

---

Published as a conference paper at ICLR 2023
Ruiyi Zhang, Bo Dai. Lihong Li, and Dale Schuurmans. GenDICE: Generalized offline estimation of stationary values. In 8th Interattonaal Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
16

---

Published as a conference paper at ICLR 2023
A ADDITIONAL RESULTS
This section contains additional experimental results. In particular values for the average returns achieved by the algorithms are reported. These were omitted from the main text as the success rates are easier to interpret.
A.1 RETURNS FOR THE PUSH TASK
We report the returns achieved on the Push task in Table ST Note that for the simulated datasets the best performing algorithm terms of return is not necessarily the same as the best performing one in terms of success rates (see Table For instance, AWAC has the highest average return on Sim-Expert while CRR achieves a higher success rate. This has two reasons. First, success measured only at the final step of the episode, while the return is computed as the cumulative reward over time. Second, the return is dense as it is computed from the distance between cube and target, while the success rate is sparse signal.
Table S1: Push: Returns on the TriFinger-Pure Datasets. 'data' denotes the mean over the dataset. Average and standard deviation over five training seeds.
Push-Datasets
data
BC
CRR
AWAC
CQL
IQL
Sim-Expert
674
585 + 19
636 + 20
657 + 14
184 + 23
631 + 18
fim-Half-xppert
674
535 + 18
576
586 +
226 + 12
565 22
Weak&Expent
512
460 = 50
613
603 23
21
543 + 75
Sim-Mixed
583
460 + 26
205 =05=100 100
636 20
138 11
588 - 25
Real- Real-Expert
660
563 + 34
638 16
17
37
592 + 29
Hii-Halff-Experr
660
546 + 29
627
587 46
26
573 23
Hell-Weak&
429
387 + 45
622 41
568 +9
346 - 68
555 + 16
Real-Mixed
419
335 +
31
569
206 =
600 30
A.2 RETURNS FOR THE LIFT TASK
In this section, we report the returns achieved on the Lift task (Table S2 achieves significantly higher return the Geal-Weak&&xppee dataset than CRR, its final success rate is lower. An analysis of the rollout statistics reveals that IQL often moves the cube close to the goal pose but not close enough satisfy the success criteria (defined in section B CRR, on the other hand, on average deviates further from the goal pose than IQL but has a bigger fraction of rollouts which satisfies the success criteria. In summary IQL does better on average on this dataset but lacks precision in matching the goal pose.
Interestingly. although IQL
A.3 EVALUATION ON THE HOLD-OUT ROBOT
To quantify how well the policies obtained with offline RL generalize to unseen hardware, we evaluate them on a holdout robot which was not used for data collection. Table [S3] shows the results for Prringer-Push--ee Real---al-----ee Expert and Litinngr--Lit-RRa-----a-a- the hold-out robot is within the performance margin of the other robots, suggesting that there is no significant difference between the different robots.
The performance of the algorithms on
A.4 IMPACT OF NOISE ON PERFORMANCE
As mentioned in section we studied the impact of noise on the performance of the expert and the considered offline RL algorithms by recording a sequence of datasets with increasing noise scale in simulation. A relative noise scale of corresponds to the variance measured on the actions and observations of the real system. Fig. ST shows the success rate and return as a function of the noise scale up value of 2. As the performance of the expert and the offline RL policies degrades only slowly when increasing noise, we conclude that the stochasticity of the real system cannot be the only reason for the performance gap between the simulated and the real system. As delays in
17

---

Published as a conference paper at ICLR 2023
Table S2: Lift: Returns the TriFinger-Lift Datasets. 'data' denotes the mean over the dataset.
Average and standard deviation over five training seeds.
Lift-Datasets
data
BC
CRR
AWAC
CQI.
IQL.
Sim-Expert
1334
1129 + 56
1246 10
1280 20
163 14
1133 + 41
Sim-Half- Expert
1337
1112 = 39
1231 =
1211 + 19
154 = 13
744 + 40
Sim-Weak& Fe--eeeex&xxpert
1133
791 43
727 447
1103 62
15
943 +
Sim-Mixed
1173
409 = 9
604 1 79
931 + 75
161 + 8
572 + 24
Sim-Expert
1334
1274 +: 17
1245 26
1319 + 11"
399 56
1286 + 20
Sim-Half-B -a--H-afff-xeettt
1337
1191 = 33
1153 +0
1303 + 14"
439 + 26
1229 + 45
Wepek&xperrt
1133
1040 22
1087 49
1120 23
467 34
1172 34
Sim-Mixed
1173
411 + 35
593 + 93
862 = 53
151 9
564 + 31
Seal-Smooth-Expee
1206
915 = 36
1059 = 54
1031 + 42
143 - 33
1002 + 19
Real-Expert
1064
711 + 92
1014 + 62
820 + 50
283 38
901 +
Real-Half-Eai -apert
1064
553 = 77
837 = 58
613 107
200 =
759 + 101
Real-Weak&Exxpee
851
346 + 21
633 + 59
397
298 16
827 + 105*
Real-Mixed
862
272 = 56
631 + 62
346
207 2
608 + 30
Real-Expert'
1064
676 + 47
889 39
747 44
289 + 33
899 + 40
Real- Real-Half--Be Expert"
1064
702 = 83
813 57
797 54
312 - 65
855 - 51
Real- &ell-Wea&&&aa Weak&Expert Expert!
851
437 = 47
707 + 20
481 + 54
269
33
574 + 67
Real-Mixed!
862
288 = 73
634 + 36
361 11 81
192 - 28
596 + 33
Table S3: Evaluation on hold-out robot. Success rate on the Real- Real-Expert datasets.
Dataset
BC
CRR
AWAC
CQL
IQL
0.80 = 0.04
0.91 + 0.08
0.84 0.06
0.61 + 0.05
0.83 + 0.09
Lift-Real-Fxpert
0.29 + 0.04
0.64 0.05** 0.31 + 0.08
0.00 + 0.00
0.24 + 0.07
the observation and action execution are already implemented in the simulated environment, hypothesize that other factors like more complex contact dynamics and elastic deformations of the fingertips and robot limbs are likely causing the larger performance gap between data and learned policies on the real robots.
AWAC and CRR perform consistently over wide range of noise scales with slight decrease in performance for high relative noise scales (probably due to large variance of the estimated object pose). BC and IQL seem to struggle with the narrow data distribution generated by deterministic environment but improve with increasing stochasticity. While the performance of BC drops significantly again for large noise scales, IQL becomes competitive in this regime.
0.75
return
1000
success
0.50
0.25
500
0.00
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
relative noise scale
relative noise scale
Expert
BCI
CRR
AWAC
CQL'
IQL'
Figure S1: Success rate and return for simulated lifting for varying relative environment noise scales (1.0 corresponds the noise level of the real system) when the algorithms trained data recorded with that noise scale using the expert policy. The dashed lines indicate the performance on the real system.
18

---

Published as a conference paper at ICLR 2023
A.5
BAR PLOTS
Fig. S2 and Fig. S3 show bar plots that summarize the performance of the algorithms on the two tasks in the simulated and real environment. Before averaging results from different datasets, normalized the algorithm performance by the dataset performance, i.e., reaching the average success rate return the dataset corresponds
value of
While AWAC (and IQL on the real data) can exceed the behavior policy on average on the Push task, all algorithms fall short matching the dataset performance on the challenging Lift datasets. While success rates on the Lift-Real datasets are particularly low, the returns indicate that CRR and IQL significantly outperform BC.
Since the hyperparameter optimization was done on the Lift-Sim- Weak&&Expert dataset, it has the biggest impact on the performance on the Lift-Sim datasets. IQL. in particular improves considerably there. The increase in performance on the Lift-Real datasets is considerably smaller, however. This suggests that optimizing the hyperparameters offline RL algorithms simulation may have limited benefits on real environments.
Push-Sim
Normalizer lizee success
Normalizz lizziidd return
1.00
0.75
0.75
0.50
0.50
0.25
0.25
BC
CRR
AWAC
CQL
IQL
BC
CRR
AWAC
CQL
IQL
Push-Real
Normalizer lzee success
Normalized return
1.00
1.00
0.75
0.50
0.50
0.25
0.25
0.00
BC
CRR AWAC CQL
IQL
BC
CRR
AWAC CQL IQL
Figure S2: Normalized performance for the Push task: Success rates at the end of episodes (or returns) were normalized to the dataset success rate (or mean return) and then averaged over all datasets corresponding to a task (treating simulated and real data separately). Default hyperparameters were used for the Push task.
19

---

Published as a conference paper at ICLR 2023
Lift-Sim
default hyperparameters
optimized hyperparameters
Normalized success
1.00
Normalized success
0.75
0.50
0.25
0.25
0.00
0.00
AWAC
CQL
IQL
BC
CRR AWACT CQL' IQL
Normalized return
Normalized return
0.75
BC
CRR
AWAC
CQL
IQL
CRR AWACT CQL' IQL'
Lift-Real
default hyperparameters
optimized byperparameters
Normalized success
Normalized success
1.00
0.75
0.50
0.50
0.25
0.25
0.00
0.00
BC
CRR
AWAC
CQL
IQL
BCT
CRR
AWACT CQLT
IQL
1.00
1.00
Normalized return
Normal lize return
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
CQL
IQL
BC
CRR AWACT CQL* IQL'
Figure S3: Normalized performance for the Lift task: Success rates at the end of episodes (or returns) were normalized to the dataset success rate (or mean return) and then averaged over all datasets corresponding to task (treating simulated and real data separately).
20

---

Published as a conference paper at ICLR 2023
A.6 LEARNING CURVES
In this section, we provide learning curves for the offline RL algorithms on all datasets. Since the evaluation of all training checkpoints on the real robots prohibitively expensive and time-consuming, we evaluate the checkpoints learned on the real data in the simulated environment. This gives an over-optimistic estimate of the learning performance on the real robots.
For the challenging Lift-Real-Weakee Expert task we performed grid search for each algorithm (BC, CRR, AWAC, CQL, IQL) and selected the best hyperparameterss. We present the grid search in the corresponding optimal hyperparameters in Table [8] and the default hyperparameters
Table S7
in Table $9 We note that due the poor performance of CQL, we expanded our grid search specifically for this algorithm (on the Push-Sim-Expert data) and selected the corresponding optimal hyperparameters as default parameters (see Figure (ST6). Our newly performed gridsearch, as mentioned above, was unfortunately not leading to improvements for CQL. Similar difficulties reported in Kumar et al. (2021) and are tackled via case distinction and appropriate regularization techniques. It would be interesting to test the preceding two algorithmic techniques our datasets, once their implementation is integrated in the D3RLPY library (Seno & Imai] (021).
We proceed by presenting the learning curves for the offline RL algorithms the datasets: for the Lift task section A.6 and for the Push task in section A.6.2 Shaded areas indicate the interval between the 0.25 and 0.75 quantiles.
21

---

Published as a conference paper at ICLR 2023
A.6.1 LIFT DATASETS
Optimized Hyperparameters!
Real-Expert
Real-Hall Hall-Half-Expertt
Real- Weak&Expert
Real-Mixed
0.6
0.5
06
0.5
0.0
0.0
0.0
0.0
150
epochs
opodis
spochs
spocts
1500
1530
1500
1500
1000
1000
730
750
750
750
500
530
500
500
130
202
150
100
150
200
Default Hyperparameters
Real- Dxpa--Expert
Real-Half- Half-Expert
Real- Weak&
Real-Mixed
1.0
1.0
10
1.0
0.6
0.6
0.6
0.6
0.4
0.0
0.0
0.0
0.0
#pochs
spochs
1500
1500
1500
1500
1.000
1000
1000
1000
750
750
750
750
BC
CRR
AWAC
CQL
IQL
Figure $4: Training curves for the Lift- Real datasets: Offline RL algorithms are trained on the real datasets and are evaluated in the simulated environment. Success rates and returns for the optimized hyperparamcters (in Table Table(S8) and the default hyperparameters (in Table S9). The selected hyperparameters, by the grid-search procedure (in Table S7), optimize the algorithm 's final average return on Sim- Weak& Expert. The black dashed line shows the dataset performance of Real- eeal-Expert and the red dashed line the performance the Real- Haal-Weak&& &xperE (first columns) and Real-Mixed (last column).
22

---

Published as a conference paper at ICLR 2023
Optimized Hyperparameters!
Sim- -im-Expert
Sim-Half-Eil Expert
Sim Weak&Expert
Sim-Mixed
0.6
0.6
06
0.5
0.2
0.2
0.2
0.0
0.0
100
150
200
1.00
150
200
1.00
150
200
50
1.00
150
200
#pocts
*pochs
sports
*pacts
1500
1500
1300
1300
1250
1230
1250
1230
750
250
750
750
500
500
500
500
100
250
100
200
100
100
e0ochs
spocth
#ponte
#002th
Default Hyperparameters
Sim- Lim-Expert
Sim- Sim--alf-Ball Half-Expert
Sim Weak&Expert
Sim-Mixed
0.6
0.6-
06
0.6
0.0
0.0
0.0
0.0
100
150
200
1.00
200
150
150
spochs
spochs
spachs
spacts
1550
1500
1500
1500
1250
1250
1250
1250
730
750
750
500
500
500
500
100
150
230
102
150
100
130
100
esochs
C50C/6
cooch6
BC
CRR
AWAC
CQL
IQL
Figure S5: Training curves for the Lift-Sim datasets: Offline RL algorithms are trained on simulated environment. Success rates and returns for
simulated dataset and are evaluated optimized hyperparameters (in Table (8) and the default hyperparameters (in Table (S)). The selected hyperparumeters, by the grid-search procedure (in Table S7), optimize the algorithm final average return Sim- Weak& Expert. The black dashed line shows the dataset performance of Sim fim-Expert and the red dashed line the performance of the Sim- Weak&Expert (first 3 columns) and Sim-Mixed (last column).
23

---

Published as a conference paper at ICLR 2023
A.6.2 PUSH DATASETS
Here, the default hyperparameters in Table 59 where for the CQL algorithm we selected hyperparameters optimized by a grid search (see Figure S16 for histograms), The success rates and returns are evaluated in the simulated environment.
Real-Expert
Real-Half-Expert
Teal-Weak&&En Expert
Real-Mixed
1385
0.6
0.5
06
0.5
0.2
0.2
02
0.2
0.0
50
100
150
200
50
100
150
200
50
100
150
200
50
100
150
200
epochs
opodis
spochs
spocto
500
500
150
500
25.0
250
250
250
50
100
150
200
50
1000
150
200
90
100
150
200
50
1.00
150
200
eoochs
opochs
opochs
4p30%s
BC
CRR
AWAC
CQL
IQL
Figure S6: Training curves for the Real datasets: Offline RL algorithms trained on the real datasets and are evaluated in the simulated environment. The black dashed line shows the dataset performance of Real-Expert and the red dashed line the performance of the Real- Real-Weak& & (first 3 columns) and Real-Mixed (last column).
Sim-Expe -in-Exxert
Sim- Weak&Expent
Sim-Mixed
0.8
0.8
rate
0.8
0.4
0.4
0.4
0.4
0.2
50
100 upochs
150
200
50
100
130
200
50
100
150
200
50
100
150
200
upochs
spuchs
750
750
750
750
30
150
200
30
100
150
200
100
130
200
330
200
spochs
epachs
spochs
BC
CRR
AWAC
CQL
IQL
Figure S7: Training curves for the Push-Sim datasets: Offline RL algorithms are trained on simulated dataset and are evaluated in the simulated environment. The black dashed line shows the dataset performance of Sim- Hun-Expert and the red the performance of the Sim- non-Weaak&Experr (first columns) and Mix-Mixed (last column).
24

---

Published as a conference paper at ICLR 2023
B DATASETS
In this section provide additional details on the datasets we collected. Section B. summarizes the data collection process, section [3.2 contains a discussion of the different notions of success we report, section B.3 is dedicated to the reward and the terminals, and section B.4 provides detailed analysis of the statistics of the datasets.
B.1
DATA COLLECTION
Data is collected by running jobs cluster of TriFinger platforms without human intervention. Before the recording of each dataset the platforms were cleaned from particle dust to ensure consistent object tracking performance. Each job consists of the following steps:
Move fingers to initial state and sample new goal.
2. Run one episode/rollout with the selected policy and store transitions.
3. Move the cube away from the barrier with pre-recorded trajectory of the fingers.
4. Repeat from 1. unless the desired number of episodes per job is reached.
5. Do self-check including cameras, pose estimation and joints.
6. Approximately reset the cube to the center of the arena.
The number of episodes collected per job is eight for the Push task (15 per episode) and six for the Lift task (30s per episode). If self-check fails, the corresponding robot is deactivated and the maintainers are alerted via mail. Finally, the data from all episodes is combined in a single HDF5 file following the conventions of D4RL (Fu et al. 2020)
The expert policies are obtained after the training Isaac Gym (see section 3.2 and section C.1, The weak policies are training checkpoints at 210 10 training steps for pushing and 288 100 training steps for lifting. Gaussian noise added to the actions with an amplitude of 0.2 Nm and an update frequency of 8 for pushing and 0.04 Nm with an update frequency of 4 for lifting.
The Mixed datasets are collected with policies from a range of training checkpoints where the total number of jobs was distributed as uniformly as possible over all training checkpoints. For pushing 22 checkpoints up to 668 10 training steps were used while for lifting 59 training checkpoints up to 1721 10 training steps considered. Fig. S8 shows the return and success rate of these checkpoints on the real TriFinger cluster.
B.2 THREE NOTIONS SUCCESS
We consider dexterous manipulation tasks which require the agent to match goal position (for the Push task) or a goal pose (for the Lift task) with a tracked cube. The tolerance for goal achievement is 2 for the position and 22 degrees for the orientation. We define momentary success achieving the goal at a single point in time, i.e., in an individual time step. This notion of success is rather weak, however, as achieving the goal pose for short amount of time is significantly easier than maintaining it. For the pushing task in particular, it is much more likely to move through the goal by accident than stabilize the cube after having reached it. When lifting the cube, it quite challenging maintain a stable grasp due to the variance of the object pose estimate which introduces a considerable amount of noise the observations. We therefore define success as achieving the goal at the end of the episode which is only likely to happen when maintaining the goal pose for an extended period of time.
From the perspective of offline RL, however, it is highly relevant whether parts of the trajectories in the datasets lead to success, even it is short-lived. As offline RL algorithms can, in principle, combine information from several trajectories, even trajectories that do not end with goal achievement may contain enough information to learn a successful policy. We therefore also report transient success which indicates whether the goal has been achieved at any time during the episode and mean momentary success which corresponds to the fraction of time steps during which the goal has been achieved.
25

---

Published as a conference paper at ICLR 2023
Miss-Reea-Mixed
Lift-Real-Mixx
600
return
1000
400
return
200
500
0.0
0.5
1.0
1.5
time steps
1e8
time steps
1e9
0.8
0.8
success rate
success rate
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
2
0.0
0.5
1.0
1.5
time steps
1e8
time steps
1e9
Figure S8: Return and success rates of the checkpoints used for collecting the Real-Mixed datasets. Note that the checkpoints were obtained by training Isaac Gym (see section 3.2 and section C.T and were evaluated on the real system. The shaded areas indicate the interval between the 0.25 and 0.75 quantiles of the return whereas the error bars represent the standard error of the mean of the success rate.
B.3 REWARD AND TERMINALS
As already mentioned in section of the main text, we use a logistic kernel Hwangbo et al. (2019); Allshire al. (2022)
b b+2 2 k(x)= exp(a||=[]) + 6 exp(-al|=)
(SI)
to define the reward where ||||| denotes the Euclidean norm. For the Push task, we apply it to the difference between the achieved and the desired cube position. Since also want to take the orientation of the cube into account for the Lift task, we apply & separately to the differences between the desired and achieved corner points (or keypoints) of the eube and average
the results Allshire et al. 2022). We use a = 30 and = 2. See Fig. S9 for visualization the reward function.
By default, the terminals in the dataset (a Boolean indicating whether terminal state has been reached) are never set. We chose this default setting to avoid problems due to state aliasing: Episodes
1.00
reward
0.75
0.50
0.25
0.00
-0.1
0.0
0.1
x [m]
Figure S9: Reward function of the Euclidean distance between the desired and achieved position (for the Push task) or keypoint (for the Lift task). The shaded grey area corresponds to cube centered at the goal and the green corresponds the goal achievement threshold (2 cm).
26

---

Published as a conference paper at ICLR 2023
Table S4: Overview of the TriFinger offline RL datasets.
task
dataset
overall duration [h]
#episodes
#transitions [10%]
episode length [s]
Sim- Sim-Expert
16
3840
2.8
15
Sim-Half- ExpertExpert
8
1920
1.4
15
S Expert
16
3840
2.8
15
Push-
Sim-Mixed
16
3840
2.8
15
Real-Exper
16
3840
2.8
15
Real-Half-Exper
8
1920
1.4
15
Real-Weak&&Exert
16
3840
2.8
15
Real-Mixed
16
3840
2.8
Sim-Fxpert
20
2400
3.6
30
Sim-Half-Expert
10
1200
1.8
30
Sim-Weak&EEpper
20
2400
3.6
30
Sim-Mixed
20
2400
3.6
30
Lift-
Real- Real-Smooth-Exper
20
2400
3.6
30
Real-Expert
20
2400
3.6
30
Real-Half HallHHaff-Exprrt
10
1200
1.8
30
Real- Rcal-Weak& &cak&Expert
20
2400
3,6
30
Real-Mixed
20
2400
3.6
30
last for a finite number of time steps H, and the RL objective is to maximize the discounted return (or cumulative reward)
H H-1
J Σ YT. t=
(S2)
where y denotes the discounting factor and rú the reward in step f. Markov state therefore has to keep track of how much time remains until the episode ends. The observation, the other hand, does not contain this information. This omission a deliberate choice as we want obtain a policy which is independent of the time step. This makes it impossible, however, to accurately estimate the value based on the observation y is too large. Intuitively, without access to the time step t, the agent cannot know whether cube far away from the goal corresponds to large expected return to go because it is the beginning of an episode or a small expected return to go because there is no time left to move the cube and accumulate reward. See Pardo al. (0188) for more detailed discussion of issues arising from training with a finite horizon.
practical solution this problem is to not the terminals and choose gamma which is appropriate for the time scale of the task (for offline learning we chose 0.99). This choice hides the episodic nature of the task from the agent which results in good performance while avoiding dependence of the policy on time. It may, however, sacrifice optimality in some corner cases like dropping the cube enough time to flip the cube over before lifting
close the end of the episode when there again. Nevertheless, as we provide flag to set the terminals at the episode ends and as it is straight forward augment the observation with the intra-episode time step, the datasets are also suitable for experiments with time-dependent policies.
B.4 DATASET ANALYSIS
Overview:
provide an overview of the various dataset types and their properties in Table S4
Statisties: Table S5 summarizes the statistics of the TriFinger datasets. While the success rate and the mean return constitute limit to what can achieved with pure imitation learning, the high numbers for the transient success rate (fraction of episodes in which the goal was achieved at least in one time step but not necessarily at the end of the episode) indicate that offline RL can potentially outperform the behavior policy significantly these datasets.
Impact of variations between robots: Fig. $10 compares the success rates of the expert policies on the individual robots used for data collection. While the performance differences between the robot instances are significant, the expert policies perform reasonably well on all of them, achieving least 80% success rate the Push task and 60% the Lift task on all robots.
27

---

Published as a conference paper at ICLR 2023
Table S5: Statistics of the proposed TriFinger offline RL datasets: TriFinger-Pust and TriFinger- Lift. Definitions of the success rates and the reward can be found in section B.2 and section B.3 respectively.
mean momentary success rate
transient success rate
TriFinger-Push
success
mean return
Sim-Expert
0.95
0.87
0.95
674
Sim-Half-Expert
0.94
0.86
0.94
667
Sin-Weak&xxpee
0.53
0.48
0.86
512
Sim-Mixed
0.76
0.68
0.77
583
Real-Expert
0.92
0.78
0.98
660
Real-Half-Eae Expert
0.92
0.78
0.98
660
Geal-Weak&Exppee
0.51
0.43
0.72
429
Real-Mixed
0.49
0.40
0.63
419
mean momentary success rate
transient success rate
TriFinger-Lift
success rate
mean return
Sim-Exper
0.87
0.77
0.97
1334
Sim-Half-
0.88
0.78
0.98
1337
Sin-Weak&Expert
0.50
0.44
0.93
1133
Sim-Mixed
0.68
0.60
0.84
1173
Sial-Smooth--Expe
0.64
0.53
0.82
1206
Real-Expert
0.67
0.52
0.87
1064
Real-Half--xpee
0.68
0.52
0.86
1064
Real-Weak& Weak&&xpert
0.40
0.30
0.66
851
Real-Mixed
0.42
0.32
0.65
862
1.0
1.0
SUccess rate
SUccess
0.0
0.0
7
7
robot
robot
(a) PerFinger-PushRRal--e
(b) TriFinger-Lift Realt---Ral-ppeerr
Figure S10: Success rates on the individual robots. The transitions recorded with policies trained with PPO in simulation. While there are significant differences between the success rates, the expert policies achieve at least 80% and 60% on every robot for TriFinger-Puur Pease-PPuu--eae Real--eee-- and TriFinger-Liff- Liltt-RRell-Eperrre respectively.
Distribution of returns: Fig. SII shows the distribution of the episode returns for the datasets collected on the real system. reveals that the expert policy on the Push task performed more consistently on the real robot than its counterpart for the Lift task. Qualitatively, this can be attributed to either (i) not being able to flip the cube to the approximately correct orientation, (ii) failing to establish stable grasp the cube, or (iii) dropping it after already having lifted it. Flipping the cube on the ground might fail due to incomplete modeling of interactions between the fingertips and the cube in the rigid body physics simulator or because it is sensitive to the value of sliding friction, the noise on the object pose estimate makes it difficult to maintain stable grasp after having lifted the object.
28

---

Published as a conference paper at ICLR 2023
Mush-Real-Expere
Lift-Real-Eperr
200
400
600
1000 episode return
500 episode return
1500
episode return
Push-Real- Real-Weak&&xxprt
LiftReal-Wecak&Experr
600
500
1000
1500
episode
episode return
Push-Real- Lich-Real-Mixed
Lift- Lift-Real-Min Real-Mixed
200
600
500
1000
1500
episode
episode return
Figure S11: Distribution of the episode returns in the datasets recorded on the real system.
Figure S12: Dataset visualization. Initial cube positions (blue) and goals (red) in the real datasets for the Push task (left) and Lift task (right). Note that we avoid initial positions close to the barrier.
Initial cube position and goal distributions: Fig. S12 visualizes the distribution of the initial cube position (blue) and the goal position (red) for Push and Lift tasks. The goal positions are sampled on the ground for the Push task and in the air for the Lift task. The distribution of the initial cube position results from the reset procedure which removes the cube from the boundaries by moving the fingers along pre-recorded trajectory.
Action statistics: Fig. S13 shows the distribution of actions of the expert policy that recorded and of a policy learned from this data by AWAC.
TriFinger-Push.
As mentioned in section 3.3 of the main text, we removed the cube from the barrier between episodes because observed that the expert policies trained in simulation struggled with retrieving the cube from the barrier. Fig. [S14a shows a histogram of the distance between the center of the cube and the arena center. While the reset procedure sufficiently stochastic
Reset procedure:
29

---

Published as a conference paper at ICLR 2023
finger joint 0
finger 0, Joint
finger O, joint 2
finger joint 0
finger 1, joint
finger joint
finger joint
finger 2, joint
finger joint
0.2 0.0 0.2
0.2 0.0
0.2
0.2 0.0
0.2
action [Nm]
action [Nm]
action [Nm]
Push-Real-Expent
AWAC
Figure S13:
Distribution action components, i.e., desired torques, for expert policy that recorded the PhlFiingr-Push-Rea---e dataset and for policy learned with AWAC from this dataset. Averaged over five offline RL seeds.
to randomize the initial cube position (see Fig. S12). it clearly removes the cube from the barrier in the majority of all resets. More precisely, only in 3% of all resets the cube center within cm of the boundary (the cube has a width of 6.5 cm).
momentary success
friquency drapping
pum feodforward smoethed
0.5
500
1000
1500
distance cube center to arena center
Uime step
[s]
Cube distribution after reset
(b) Dropping frequency
(c) Momentary success
Figure S14: (a) Histogram of the distance between the center of the cube and the arena center after the reset for the Sitt-Real-Expert dataset. In the shaded gray area the cube can potentially touch the barrier which begins at the vertical black line. (b) Dropping frequency over a real lift episode. 'Pure feedforward' refers directly applying the output of an MLP policy whereas 'smoothed' denotes policy with an Exponential Moving Average applied. (c) Comparison of the momentary success rate of policy learned from Push Real-Expert (AWAC in green) with the expert policy (PPO-Expert in blue) that recorded the data. The momentary success rate captures how often the policy achieved the goal at a time step averaged over all episodes. The momentary success rate of the weak policy used in Push-Real &eal--Waak&&peeeee is shown in orange.
Dropping frequency pure feedforward smoothed: One reason for the lower success rate the Lift task as compared to the Push task is frequent dropping of the cube. This can be the result of vibrations of the fingers which can occur when applying policies trained in simulation the
30

---

Published as a conference paper at ICLR 2023
real robots of shaking caused by the noisy pose estimate for the cube. We found that applying low-pass filter, more precisely an Exponential Moving Average, helps with both problems. Fig. [S14b) shows the frequency at which the cube is dropped over the course of lifting episode. Dropping the cuhe is defined as reaching the height of the goal pose up to a tolerance of 2 cm at some point during the episode and then dropping the cuhe flat the ground (up tolerance of cm). Smoothing the actions before applying them clearly helps with avoiding dropping, in particular later in the episode. Note, however, that the smoothing has be applied already during training in simulation for the policy to adapt to it. Despite the action smoothing the dropping rate does not reach zero. This can be partly attributed to unstable grasps that lead to slipping on the real robot.
B.5 USING THE DATASETS
We provide casy-to-use Python package that is compatible with popular collection of offline RL datasets Fu et al. (220)). To use the datasets, it is sufficient to clone the repository, instantiate the desired environment and call method that returns the dataset. The correct dataset is then downloaded automatically. We also provide options for filtering the observations and for converting them to flat arrays.
The following example code demonstrates how (parts of) dataset can loaded:
import gymnasium as gym
import numpy as np
import trifinger rl datasets
gym. gym.make(
disable env www_eeeeeeeeeeeeeeeeeeeeeeeeeeee True, disable_env_cheeeee visualization-Falsee
load data timesteps specified in array dataset_part env. www.get_dataset (indices=np array ([42, 1000, 5000)))
load data corresponding to range of timesteps dataset_pari part www.get_datee (rng" (1000, 2000))
load the whole dataset dataset env get_dat
For installation instructions and further details we refer to the repository of the Python package at https://github.c htt.:hhw..ooomeeaeee www.c///re-eeeeeaaanrarririeeeee
xl datasets
B.6 SUBMITTING A POLICY TO THE ROBOT CLUSTER
Access the robot cluster be requested https://weebaas wwweeedav..uebbiiiee mpg de/ trifinger/ The policy has to be implemented in a GitHub repository following a fixed in- terface. We recommend adapting the example package available https::////.. //gtthub//ttbb// com/ rr lew.ning/trifingeeee www.inneeerrrreemmee r1-example
31

---

Published as a conference paper at ICLR 2023
Table S6: Performance of the expert policies in the Isaac Gym environment. The numbers for the success rate and return are not directly comparable to those in the PyBullet simulator and on the real robot for two reasons: (i) In Isaac Gym Lift episodes last for 15s instead of 30 and (ii) the return in the Isaac Gym environment is contains auxiliary reward terms as described in Allshire et al. (2022)
and section 3.2
success rate
return
#training steps
Push
0.99
629
0.83 10
Lift
0.87
589
1.72 10
C TRAINING
C.I TRAINING EXPERT POLICIES IN SIMULATION
In addition to the modifications mentioned in section 3.2 we also ported training from Isaac Gym 2 to the current version Isaac Gym 3 (Makoviychuk et al. 2021). We furthermore increased the torque range from [-0.36 Nm, 0.36 Nm to 0.397 Nm, 0.397 Nm] make sure the fingers are strong enough to lift the cube when supporting from below. To make the training environment resemble the data collection setting on the real robots, we furthermore implemented an adjustable delay between when an observation is received and when the action based on this observation is applicd for the first time. The success rates and returns reached after training are shown in Table S6
Note that the success rates give for the lifting task the real robot not directly comparable to the ones reported in Allshire et al. (2022) as our lifting task more challenging. While Allshire et al.) (2022) evaluate success after 60 we evaluate after s. As policies usually need several attempts to flip the cube to roughly the correct orientation and for picking it up, the success rate after 30 is lower in general. We chose the shorter episode length because it is, in principle, sufficient solve the task and because we wanted to avoid episodes which consist, to a large part, of the cube being held in place. Moreover, unlike Allshire et al. (2022). we do not push the cube to the center of the arena before each episode but only remove it from the barrier.
C.2 TRAINING WITH OFFLINE RL ON THE DATASETS
We use the implementations of BC, CRR, AWAC, CQL and IQL provided by the open-source library D3RLPY (Seno & Imai 2021). The code is available at lhtps:: / httpg//ttt....tttttttot/ooo www.//ootttaaaunaeee d3rlpy and the documentation can be found at
greadthedocs.ion For experiments used versions 1.1.0 and 1.1.1 of D3RLPY. The used hyperparameters and the performed optimization are discussed in the next section.
C.3 HYPERPARAMETERS
We performed a grid search over hyperparameters for all algorithms as documented Table S7 hyperparameter setting with the highest performance in terms of final average return on Lift-Sim- Weak&Expert was selected, as listed in Table (88 the paper, the results with optimized parameters are marked with a 1, Otherwise, the default parameters were used, as listed in Table S9
The
The rest of the section contains a detailed analysis of the grid search.
Personal communication
32

---

Published as a conference paper at ICLR 2023
Table S7: Hyperparameter grid search (on dataset Lift-Sim-Weakkee SSin-Waak&EEpprr)) Each algorithm is trained with the same two seeds (sampled uniformly at random).
Algorithms
Parameters
AWAC
[(actor (aator__earning Jearning_rraaeeeeeee rate=R, critic leatic_eernnnn rate=R) (6(1555-4,3300 6.00-44):
batch sizee (256, 512); lume (0.3, 1.0,
BC
leaminngrate=(115EE44 3.0E-4; 6.0E 4); batch_sizee {256, 512)
CRR
{actor_leammnng rate=R, critic_lieanee Bateriin__rtt=RR.___ RE(1,55.4 3.0F=4,6.0F.-4 6.0F.4]]::
batch_sizeE {256, 512); betaE {0.25, 1.0,4.0) 4.0)
CQL
{aacoorlccaming lwwlllaaannnaaalleae ratc=R,t critic Jeariii__annnnnnn Jearning_ratc=3.0*
rate=3.0*R, R, initial_atn alpha=T[1],
alpha leara_llamnnnggg rate= R if T[O] else 0j, leamming rate=R, alpha threshold:T(2)).
RE (5.0E-5, 1.0E-4]. Te ([true, 1.0, 1.0]. [true. 1.0. 5.0). [true, 1.0, 10.0).
[false, 0.3, 10.0). [false. 1.0, 10.0). [false. 3.0, 10.0]] }:
sntervattvveweiihhee (2.5, 5.0, 10.0, 20.0)
IQL.
Battoo__laminnneeeeee rate=R, critie_leannnne Jearning_rattRR)) rate=R): RE( BE(1.5R-4 3.0F-4, 6.0F:4)]; batch_sizeE (256, 512); expectileE {0.7, 0.8, 0.9); weighr_tempe {3.0, 10.0)
Table S8: Optimized Hyperparameters using grid search Table S7
Algorithms
Parameters
AWACI
actor_ = critic_leamming.n 0.00015; size=255 lam=3.0
BCI
leaminggrrac==00001555 batch_size=512
CRR
actor Jeammin critic learning rate 0.00015: butch size=256; beta=1.0
CQL hooo___eaming_rate000000 conservative_weiiht=200..0 alpha rate=0.0; scaiorr minmax
initial initial_alpha=1..
IQL
actor actor_leaming rate critic_leane lerdie_llannnng_nee caaeninn_rateeiiaee 0.00015; amcch_size2550 expectile=0.9;; weight temp-33.
Table S9: Default hyperparameters. All algorithms except BC (with batch 1itch_size=100 and without critic) have batch size=256 and n critics=2. Note that due to bad performance we optimized CQL's parameters on Push-Sim-Expen with extensive grid search as shown in Fig. ST6 For all other algorithms, we used the default values of the implementation.
Algorithms
Parameters
AWAC
actor loarilcannnng-- = critic learning rate 0.0003; batch sizc=1024; lam=1,0
BC
learning_ratne rate=0.001: batch size=100 100
CRR
actor_leaming = critiic__eeeeemnn_rattee = 0.0003; batch_size=266; beta=1.0
CQL
actor acto__leaming_rntcc000000 critic Baaaminng_ate00.0000 initial initial_alpha=l alppaalpph000 conservative_weiihhze0..
action scaler: minmax
IQL
actor leuminc rate critic Jearning rate= 0.0003; batch size=256;
c
expectile=0.7;
33

---

Published as a conference paper at ICLR 2023
present the returns and success rates for each of the hyperparameter settings for the Lift task in simulation. We see that the different algorithms have very different sensitivity to their hyperparameters. For CRR large fraction of parameters leads good results. For AWAC the sensitivity is a bit higher. IQL seems to degrade more gracefully with changed parameters. For CQL were unable to find good hyperparameters, despite running 48 configurations. The performance of the individual runs over training time are shown in Fig. S17
In Fig. S15
22.5
30-
30.0
3.3
503
1.000
5:00
1000
5.00
20000
500
1200
500
1000
return
naturr
retuum
retum
2
Frouurcc
Froqusncy
1.0
20
0.5
0.3
0.00
0.00
5.75
0.25
0.75 1.00
BC
CRR
AWAC
CQL
IQL
Figure S15: Hyperparameter grid search: Histogram of returns and success rates for the Lift-Sim- Weak&Expert task.
On the Push task, we ran even larger hyperparameter scan with 405 configurations for CQL, as presented in Fig. [11 Even for this much simpler task, we see that the majority of parameters yield low success rates. In addition, and due to this, we conducted our main grid-search with alpha Ir = 0.0. Then, we chose the best hyperparameter configuration the default setting for COL.
note that the training became quite unstable for alpha Ir > 0.0
Returns
Success rate
80
150
Parameters
60
actor_ir E (3.0E-5, 1.0E-4, 3.00-4
trequency
frequency
critic Ir E (3.0E-5. 1.0E-4, 3.0E-4);
100
40
temp_Ir E [3.0E-5, 1.0E-4, 3.0E-4):
cons weight E (1.0, 5.0, 10.0, 15.0, 20.0); batch_size E (256, 1024, 4096);
20
50
alpha_Ir= 0.0:
1000
2000
3000
1.0
return
success_rate
Figure S16: Hyperparameter grid search the Push-Sim-Expert task for COL. Shown is the histogram of returns and success rates for the 405 hyperparameter settings, as defined on the right.
34

---

Published as a conference paper at ICLR 2023
Returns
BC
CRR
AWAC
CQL
IQL
1250
1793
12:0
1000
1602
1.60
1066-
790
106
1.00
202
104
350
200
1:00
50
102
160
200
116
150
203
upeda
Success-rates
BC
CRR
AWAC
CQL
IQL
1.00
150
202
30
100
250
200
54
1.00
150
200
30
350
202
57
100
150
200
Figure S17: Lift-Sim-Weaa&&xppe Wxxer&xprr&. Hyperparameter grid search. Returns (top row) and success rate (bottom row). Every curve corresponds to one parameter configuration (averaged over seeds). The black dashed line shows the dataset performance of Sim-Expert and the red dashed line the performance of the Sim- Weak&Expert.
D POLICY EVALUATION
D.1 EVALUATION ON THE REAL SYSTEM
On the real platforms, we report numbers for 48 trials for the Push task and 36 trials for the Lift task, each for 5 training seeds. Each algorithm and seed is evaluated on 5 to 6 robots with the same set of 6 (for the Lift task) to (for the Push task) goals per robot (computed from the robot ID). The success rate and return of each algorithm and seed is computed from the resulting trials. The mean and standard deviations of the success rates and returns of each algorithm is computed across seeds.
D.2 EVALUATION IN THE SIMULATOR
In the simulated environment, we perform 100 testing episodes per final policy for independent training runs.
35


Reviews:
[{"review_id": "oR2r70FgKM", "reviewer": "Reviewer_RMbA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a real robot RL benchmarking platform for dexterous manipulation and, as a main contribution of this paper, collect two offline RL datasets and benchmark current popular Offline RL algorithms on these two datasets. This provides a point of reference to track future progress in Offline RL algorithms on these tasks.", "summary_of_the_review": "I think for a field such as robotics having a set of well-established benchmarks is important, especially when we are talking about real world robotics and not sims. Currently each lab doing real-robot RL has its own hardware and RL problem definitions, which makes is hard to track progress in the similar manner to how we were able to track progress on Atari, MuJoCo, ImageNet etc. On one hand this work does not offer any novel scientific contributions, but on the other hand it also isn't the goal here. The goal is to get as many people as possible aware of the benchmark. Which is kind of the purpose of conferences as such - to make the community aware of things. With all that in mind I leaving a recommendation of \"weak accept\". In order to make a stronger recommendation the range of tasks would have to be wider and more diverse.", "strength_and_weaknesses": "The paper reads well and leaves no confusion about what was done, how and why.\n\nThe purpose of this work is simple and is executed well, so that leaves little room for weaknesses as such. However what I would bring forward is the fact that in here we only deal with two datasets on only one platform. This set of tasks covers only a small portion of possible real-robot hardware tasks we (as a robot learning community) would like to benchmark our algorithms on. And within the subset of tasks that this work does address (dexterous manipulation) only a small set of challenges are presented. \n\nFor related work - here is another real robot platform that provides remote access for RL training https://arxiv.org/abs/1910.08639", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gXo74gnSKO2", "reviewer": "Reviewer_KUtS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a benchmark for offline Reinforcement Learning algorithms, which includes simulated data, real-world data and the possibility to execute the learned policies on a real-world robotic system. The robot application is dexterous manipulation, the specific tasks are Push and Lift, and the robotic platform TriFinger. Moreover, the paper presents a benchmark of prominent open-sourced offline RL algorithms (CRR. AWAC, CQL, and IQL), and a detailed analysis of the obtained results.", "summary_of_the_review": "This is a good paper. The proposed benchmark of ORL algorithms is very valuable and will allow the RL community comparing ORL algorithms and studying their applicability.", "strength_and_weaknesses": "The benchmark of offline Reinforcement Learning (ORL) algorithms is a very relevant issue for comparing them and studying their applicability. The paper addresses the key issue of providing valuable data for comparing ORL algorithms, but also the possibility to execute the learned policies on a real-world robotic system. This is very relevant because the comparison of this kind of algorithms requires evaluating the so-called \"distribution shift\", which from my point of view requires experimenting in the real-world. This is the main paper's strength, in addition of being very clear and well written.\n\nI don’t see any relevant weaknesses.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "T9_7WFs6yV", "reviewer": "Reviewer_Yamp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new benchmark for Offline Reinforcement Learning (RL). The benchmark contains two tasks, pushing a block and lifting it to a desired position. Rewards for each task are based on the object's pose (and distance to the goal pose). The hardware used is the TriFinger robot, available as a cluster. There is a pybullet simulation component to this benchmark as well. The datasets for offline RL have a mixture of expert, weaker and random policies, similar to other offline RL benchmarks. The paper shows evaluation of many SOTA offline RL benchmarks such as CQL, CRR, AWAC and IQL as well as behavior cloning. ", "summary_of_the_review": "Overall, I think this paper tackles an important problem of benchmarking in robotics and presents a good and useful solution to it. The benchmarks seems like it is ready for public use, and would be a benefit to the community. There are details on the exact implementations and setup of the benchmark missing, and adding the ability for users to do even more types of analysis and ablations would be interesting. ", "strength_and_weaknesses": "Strengths: \n\n- I think this paper aims to solve a very important problem in robotics, which is the lack of proper comparisons and benchmarks. \n- This paper presents an easy to use benchmarks with multiple tasks and multiple baselines \n- The experiments conducted are thorough and insightful\n- The paper is well written and very clear \n- To my knowledge, no other such offline RL for robotics benchmarks exist that can allow for running the hardware directly\n\nWeaknesses: \n\n- While I think the experiments are interesting, implementation makes a big difference in the performance of RL algorithms. It would be good to include details about how these approaches are implemented should be included in Section 3. \n\n- Additionally, more information should be included on the standardization protocol i.e., how each type of dataset is created exactly, and how to add other types. In general, it would be good to see more types of datasets with a visualization of the distributions of each. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5F7_P3-Eks", "reviewer": "Reviewer_zztz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a novel benchmark for offline RL algorithms on real-robot hardware. The authors present two tasks, a pushing and a lifting tasks, which can be performed on the TriFinger platform. For each task an expert policy is learned on a simulator with extensive domain randomization. Afterwards the expert policy is used to gather two datasets, one from a simulated environment and one from a real-robot platform.\nThe authors then evaluate several SOTA offline RL algorithms on all four datasets and test the effect of including \"weak trajectories\" as well as reducing the amount of available expert data.", "summary_of_the_review": "Since this is a pure benchmark paper the requirements concerning an extensive experimental validation are very high. In my opinion though they are fulfilled in this case. Additionally, I find that there is a considerable need for a real-robot benchmark in the offline RL community, which this paper contributes with great reproducibility. Therefore I vote to accept this paper.", "strength_and_weaknesses": "Strengths:\n- The experimental evaluation is extensive and very detailed.\n- The reproducibility of the results seems very good. In particular the authors announce that they will publish the datasets and they will offer access to the real-robot platform for validation\n\nWeaknesses:\n- This is a pure benchmark paper. There is no technical contribution in terms of novel models or methods. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%