Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
Authors: Shivakanth Sujit
Pedro Braga
Jorg Bornschein
Samira Ebrahimi Kahou
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning (RL) has shown great promise with algorithms learning in environments with large state and action spaces purely from scalar reward signals. 
A crucial challenge for current deep RL algorithms is that they require a tremendous amount of environment interactions for learning. 
This can be infeasible in situations where such interactions are expensive, such as in robotics. 
Offline RL algorithms try to address this issue by bootstrapping the learning process from existing logged data without needing to interact with the environment from the very beginning. 
While online RL algorithms are typically evaluated as a function of the number of environment interactions, there isn't a single established protocol for evaluating offline RL methods.
In this paper, we propose a sequential approach to evaluate offline RL algorithms as a function of the training set size and thus by their data efficiency.
Sequential evaluation provides valuable insights into the data efficiency of the learning process and the robustness of algorithms to distribution changes in the dataset while also harmonizing the visualization of the offline and online learning phases. 
Our approach is generally applicable and easy to implement. 
We compare several existing offline RL algorithms using this approach and present insights from a variety of tasks and offline datasets.

Published in Transactions on Machine Learning Research (11/2023)
Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
Shivakanth Sujit ETS Montreal, Mila Quebec
shivakanth. sujit. etsmtl.cc eteseettttl.....cccc
Pedro H. M. Braga
plomby plomba@cii...o ufpe. br
Universidade Federal de Pernambuco, Mila Quebec
Jorg Bornschein DeepMind
bornschein det.peeiidddddeeedddddd.domm
Samira Ebrahimi Kahou
samiro, www.heeb. www.simm.kahouu@mmall.co
ÉTS Montréal, Mila Quebec, CIPAR Chaär
Reviewed on OpenReview: https: h/tp///ppeeeeeeeeeeeee net/ forum? id- J3veZdVpts
Abstract
Reinforcement learning (RL) has shown great promise with algorithms learning in envi- romments with large state and action spaces purely from scalar reward signals. crucial challenge for current deep RL algorithms is that they require tremendous amount of environment interactions for learning. This can be infeasible in situations where such in- leractions are expensive, such in robotics. Offline RL algorithms try to address this issue by bootstrapping the learning process from existing logged data without needing to interact with the environment from the very beginning. While online RL algorithms are typically evaluated as a function of the number of environment interactions, there isn't a single established protocol for evaluating offline RL methods. In this paper, we propose sequential approach evaluate offline RL algorithms as a function of the training size and thus by their data efficicncy. Sequential evaluation provides valuable insights into the data efficiency of the learning process and the robustness of algorithms to distribution changes in the dataset while also harmonizing the visualization of the offline and online learning phases. Our approach is generally applicable and easy to implement. We compare several existing offline RL algorithms using this approach and present insights from a va- riety tasks and offline datasets. The code to reproduce our experiments is available at https://sshiaaaantssujitt hit.iit.ottttbooooogo..... eval.
1 Introduction
Reinforcement learning (RI.) has shown great progress recent years with algorithms learning to play highly complex games with large state and action spaces such as DoTA2 (=104 valid actions) and StarCraft purely from scalar reward signal (Berner et. al. 2019 Vinyals al. 2019). However, each of these breakthroughs required tremendous amount of environment interactions, sometimes upwards of 40 years of accumulated experience in the game (Schrittwieser et al.) 2019 Silver et al. 2016 2018). This can be infeasible for applications where such interactions are expensive, for example robolics.
Offline RL methods tackle this problem by leveraging previously collected data to bootstrap the learning process towards a good policy. These methods can obtain behaviors that maximize rewards obtained from the system conditioned on fixed dataset of experience. The existence of logged data. from industrial applications provides ample data to train agents in simulation till they achieve good performance and then

---

Published in Transactions on Machine Learning Research (11/2023)
can be trained on real hardware. The downside of relying on offline data without any interactions with the system is that the behavior learned can be limited by the quality of data available (Levine et al. 2020).
Levine et al. (2020) point out that there is a lack of consensus in the offline RL community on evaluation protocols for these methods. The most widely used approach is to train for a fixed number of epochs on the offline dataset and report performance through the average return obtained over a. number of episodes in the environment. However, this style of training does not provide insights into the data efficiency of offiine RL algorithms. It also does not reveal how algorithm reacts to changes in the dataset distribution, for example with better dataset- gennrating policies. In this paper, propose to evaluate algorithms function of available data instead of just reporting final performance or plotting learning curves over a number of gradient steps. This approach allows us to study the sample efficiency and robustness of offline RL algorithms to distribution shifts. It also makes it easy to compare with online RL algorithms as well as intuitively study online fine-tuning performance. We call this approach of evaluation Sequential Evaluation EvoqEval) and present settings where SeqEval can be integrated into the evaluation protocol of offline RL algorithms. We also propose a style of model card (Mitchell et al. 2018) that can be designed for offline RL algorithms to provide relevant context to practitioners about their sample efficiency to aid in algorithm selection.
2 Background and Related Work
In Reinforcement Learning (RI.) we are concerned with learning a parametric policy To(als) that maximizes the reward it obtains from an environment. The environment is described as a Markov Decision Process (MDP) using- < S, R, P >, where S represents the states of the environment, represents the actions that can be taken, R. is the set of rewards and P denotes the transition probability function S x A + S. The objective is to learn optimal policy, denoted by Th, as T arggmar (Eio 7'ra] where is the discount factor and Tr is the per timestep reward. The Q value of a policy for a given state-action pair is the expected discounted return obtained from taking action a at state st and following the policy w afterwards. By estimating the Q value, we can implicitly represent a policy by always taking the action with the highest Q value. The Q value network is trained by minimizing the temporal difference (TD) error, defined as (Q(se, ae) - (r. + QQ(+++ , Q is called the target Q network and is usually a delayed copy of the Q network or a exponentially moving average of the Q network.
[0,
Offline RL. The simplest form of offline RL is behavior cloning (BC) which trains an agent to mimic the behavior present in the datasot, using the dataset actions as labels for supervised learning. However, offline datasets might have insufficient coverage of the states and operating conditions the agent will be exposed to. Hence BC agents tend to be fragile and can often perform poorly when deployed in the online environment. Therefore, offline RL algorithms have to balance two competing objectives, learning to generalize from the given dataset to novel conditions without deviating too far from the state space coverage in the dataset. The former can be enabled by algorithms that learn to identify and mimic portions of optimal behavior from different episodes that are suboptimal overall, as pointed out in Chen et al. (2021). The latter issue is addressed by constraining the agent's policy around behavior seen the datasct, for example, through enforcing divergence penalties on the policy distribution (Peng et al., 2019 Nair et al. 2020). Another way of penalizing out of dataset predictions is by using a regularizer
the value, for example Conservative Q Learning (CQL) (Kumar et al., 2020) prevents actions that have low support in the data distribution from having high Q values. Decision Transformer (DT) (Chen et al.) 2021) does not learn Q estimate and instead formulates the RL task as a sequence modelling task conditioned on the return-to-go from a given state. The agent is then queried for the maximum return and predicts actions to gencrate a sequence that produces the queried return. This a very brief overview of oflline RL methods, and we direct readers to Levine al. (2020) for a broader overview of the field.
Metrics and Objectives. The paradigm of empirical risk minimization (ERM) (Vapnik 1991] is the prevailing training and evaluation protocol, both for supervised and unsupervisod Deep Learning (DL). At its core, ERM assumes a fixed, stationary distribution and that we are given a set of (i.id.) data points for training and validation. Beyond FRM, and especially to accommodate non-stationary situations,
2

---

Published in Transactions on Machine Learning Research (11/2023)
Buffer D
Logged date
Agent
Logged
Agant
(a)
Minibatch
(b) Sequential Evaluation
Figure 1: Comparison of traditional training schemes with Sequential Evaluation
Algorithm Algorithm for Sequential Evaluation in the offline setting.
Input: Algorithm A, Offline data D {st, at, Tr, 8641 1 increment-size 7, gradient steps per increment K, evaluation frequency fe
2: Replay-buffer B <- {st, Qz, Pt,, 8t41 To
3: 1 + To
4: while / << do
5: Update replay-buffer B BU {st, ae, rt, $441 137y
Sample training batch, ensure new data inchuded: batch B Perform training step with A on batch.
t t ++
for 1, do
10:
Sample a training batch ~ B
11:
Perform training step with on batch.
12: end for
13: if % fe then
14: Evaluate in the environment and log performance.
15: end if
16: end while
different fields have converged to alternative evaluation metries: In online learning, bandit research, and sequential decision making in general, the (cumulative) reward the reyrel of central interest. The regret is the cumulative loss accrued by agent relative decisions. For learning in situations where single, potentially non-stationary sequence of observations is given, Minimum Description Length (MDL) (Rissanen 1984 provides a theoretically sound approach to model evaluation. Multíple, subtly different formulations of the description length are in use, however, they are all closely related and asymptotically equivalent to prequential MDL, which is the cumulative log loss when soquentially predicting the next observation given all previous ones (Rissanen, 1987: Poland Hutter 2005). Similar approaches have been studied and are called the prequential approach (Dawid & Vovk 1999) simply forward validation. A common theme behind these metrics is that they consider the agents' ability to perform well, not only in the big-data regime, but also its generalization performance at the beginming, when only a few observations are available for learning. The MDL literature provides arguments and proofs why those models, that perform well in the small-data regime without sacrificing their big-data performance, are expected to generalize better to future data (Rahhannerr & Hutter 2011
optimal agent when sequentially making
With these two aspects in mind, that a) RL deals with inherently non-stationary data, and b) that sample efficiency is theoretically and practically desirable property, propose to evaluate offline RL. approaches by their data efficiency.
3 Sequential Evaluation of Offline RL Algorithms
As mentioned above, one approach for offline RL evaluation is to perform multiple epochs training over the datasct. We contend that there are a few issucs with this approach. Firstly, this approach does not provide much information about the sample efficiency of the algorithm since is trained on all data every epoch. This means that practitioners do not how the algorithm scale with the dataset size,
3

---

Published in Transactions on Machine Learning Research (11/2023)
or if it can achieve good performance even with small amounts of logged data. Furthermore, there can be distribution changes in the quality of the policy in the dataset, and evaluating as function of epochs hides how algorithms react to these changes. Finally, there is a disconnection in the evaluation strategies of online and offline RL algorithms, which can make it difficult to compare algorithms realistically.
Instead of treating the dataset as a fixed entity, we propose that the portion of the dataset available to the agent changes over time and that the agent's performance is evaluated as function of the available data, We term this as sequential evaluation of offiine algorithms (SeqEval). This can be implemented by reusing any of the prevalent replay-buffer with sampled trajectories from the currently learned policy, we insert prerecorded offline RL data. That is, the offline dataset replaces the enviromment as the data generating function. We alternate between adding new samples to the buffer and performing gradient updates using mini- batches sampled from the buffer. The number of samples added to the buffer at time is denoted by y and the number of gradient steps performed between each addition to the buffer is denoted by K. A concrete implementation of the approach is outlined in Alg. and it is illustrated in Fig.
hoplay-buffer-basee training schemes from online deep RL. Instead of extending the
This approach of evaluation addresses several of the issues with epoch-style training, By varying 7 and K we can get information about the scaling performance of an algorithm with respect to datasct size, which tells us if data is the bottleneck for further improvements, and how quickly the algorithm can learn with limited data. We can visualize how the algorithm behaves with shifts in dataset quality directly from the performance curves. y and K together affect the rate at which an offline RL algorithu is given access to the data and hence can be studied together depending on the needs of the practitioner. Online RL methods are evaluated as function of the number of environment steps, which is measure of the amount of data available to the agent. In the offline RL setting this would translate to evaluating the algorithm in terms of the amount of data it has access to in the replay buffer. We can also seamlessly evaluate the performance of the algorithm in online fine-tuning by adding samples from the environment once the entire offline dataset is added to the replay buffer. A benefit of the sequential approach is that it does not require a complete overhaul in codebases that follow existing training paradigms. For the baselines that we present in this paper, we were able to use the sequential evaluation approach with less than 10 lines of changes to the original codebases.
The ratio K is the replay ratio (RR), a hyperparameter commonll found in RL algorithms which use replay- 7 buffers. The RR determines how many gradient steps are performed for each added data point. Increasing RR allows some algorithms to extract more utility from the data it has received, potentially improving sample efficiency. By varying RR we can identify whether computation is the bottleneck for improving performance, or the amount of available data.
Implementation Details To ensure that the algorithm sees each data point in the dataset at least once, when a new batch of data is added to the buffer, the algorithm is trained on that sample of data once before K mini-batches are sampled from the buffer for training. If the batch added is smaller than the mini-batch size, then it can be made part of the next mini-batch that is trained on. In practice, we found that setting y and K to 1 worked well in all datasets tested. This means that the x-axis of all plots directly corresponds to the number of samples available for training and the number of gradient updates performed. Additionally, we study other values of the RR and report these results in Appendix C We observe that RR of 1 generally performs well across datasets and higher RRs lead to overfitting and worse performance. The changes made to the codebase of each algorithm are as follows: Each codebase had a notion of a replay buffer that was being sampled, and the only addition required here was a counter that kept track of up to which index in the buffer data points could be sampled from to create mini-batches. The counter was initialized to To = 5000 so that there were some samples in the buffer at the start of training. The second change that needed to be made was changing the outer loop of training from epochs to the number of gradient updates and incrementing the buffer counter by every K update. This way, the amount of data the algorithm was trained on sequentially increased to the full dataset over the course of training. Finally, we shuffle the order of the dataset on each scod so that the training curves are not speefic to only one particular ordering of the dataset. From our experiments we see that the dataset order does not have a significant impact on the performance of the algorithm.
4

---

Published in Transactions on Machine Learning Research (11/2023)
4 Experiments
4.1
Baselines and Benchmarks
We evaluate several existing offline RL algorithms using the sequential approach, namely Implicit Q Learning
(IQL) [Kostrikov et al., 2022), CQL [Kumrr et al., 2020), TD3+BC (Fujimoto & Gu 2021 AWAC (Nair
et a.. 2020), BCQ (Fujimoto et al.) 2019), Decision Transformer (DT) Chen et al., 2021 and Behavior Cloning (BC). These algorithms were evaluated on the D4RL benchmark (Fu et al. 2020], which consists of Hallcheetah-v2,, Walker2d-v2 and Hopper-v2. For each environment, we evaluate four versions of the offline dataset: random, medium, medium-expert, and medium-replay. Random consists of IM data points collected using random policy. Medium contains 1M data points from policy that trained for one-third of the time needed for an expert policy, while medium-replay is the replay buffer that was used to train the policy. Medium-expert consists of a mix of 1M samples from the medium policy and 1M samples from the expert policy. These versions of the dataset are used to evaluate the performance of offline agents across a spectrum of dataset quality.
three environments:
We also created a dataset from the DeepMind Control Suite (DMC) (Tassa et al. 2018 environments following the same procedure as outlined by the authors of DARL. We chose the DMC environments because of their high dimensional state space and action space. Specifically we chose cheetah run, walker run and finger turn hard and trained Soft Actor Critic (SAC) (Haarnoja et al. 2018) agent these environments for 1M steps. A medium and expert version of the dataset was created for cach of the three environments using policies obtained after 500K and 1M steps respectively.
Finally, to study algorithms in visual offline RL domains, we used the v-d4rl benchmark (Lu et al. 2023 which follows the philosophy of D4RL and creates datasots of images from the DMC Suite with varying difficulties. We use the cheetah run, humanoid walk and walker walk domains and the expert, medium and random versions of these domains. [u et al.] (2023) open sourced implementations of DrQ (Kostrikov et al. 2020 variants along with the dataset and we compare these algorithuns using SeqEval.
4.2
Experimental Settings
We study varied ways in which SeqEval can be integrated into offline RL training. The first setting is the 'Standard' setting in which samples are added periodically to the buffer during training. The results of the Standard setting
subset of datasets are given in Fig. 2 and the complete set of datasets, along with experiment comparing curves with larger K are available in the appendix. Secondly, to highlight how SeqEval can visualize how the algorithm reacts to changing dataset quality during training, we create a "Distribution Shifts' setting where a mixed version of each environment is created. In this dataset, the first 33% of data comes from the random dataset, the next 33% from the medium datasct and the final 33% from the expert dataset. From the performance curves given Fig. G we can see how each algorithm adapts to changes in the dataset distribution. We then show how SeqEval can be utilised in multi task offline RL in a similar fashion. Instead of providing the entire datasct of new task to the agent at once, we periodically add new samples of the task to the agent over time. This lets study the amount of samples required for the agent to transfer to the new task. Finally we also show how Seqlval supports seamless integration of online fine-tuning experiments into performance curves. In this setting, once the entire offline dataset is added to the replay buffer, the agent is allowed to interact with the online simulator for a fixed number of steps (500k steps in our experiments). Since the curves are function of data samples, we can continue evaluating performance as before. The results on a subset of datascts are given in Fig. 7 and curves for all datasets are available in the appendix.
For each dataset, we train algorithms following Alg. D initializing the replay buffer with 5000 data points at the start of training. We set and K cach to 1, that is, there is one gradient update performed on a batch of data sampled from the environment every time a data point is added to the buller. The x-axis in the performance curves represents the amount of data in the replay buffer. In each plot, we also include the performance of the policy that generated the dataset as a bascline, which provides context for how much information each algorithm was able extract from the dataset. This baseline is given as horizontal dotted line.
5

---

Published in Transactions on Machine Learning Research (11/2023)
4.3 Model Cards
Since Eeq evaluates algorithms as a function of data, it opens up the possibility of quantitatively analysing how algorithms scale with data. We propose a few evaluation criteria in addition the training curves. The first one is Perf@50%, which measures the aggregate performance of algorithms when half the dataset is available to it for training. To further analyze the role of data scaling, we studied two measures, Perf050% normalised with respect to performance obtained with the whole dataset (Perf0100%) and the difference between Perf&100% and Perf050% These three metrics give end users valuable insights that can aid in offline algorithm selection. They can be presented as part of an overall model card (Mitchell et al. 2018 available the user.
4.4 Results
Standard We present the results of the Standard setting in Figs. and One striking observation from the learning curves in Figs. 2 and 4 is how quickly the algorithms converge to a given performance level and then stagnate. This is most evident in the medium version of each environment. With less then 300K data points in the buffer, each algorithm stagnates and does not improve in performance even after another 500K points are added. That is, there are diminishing returns from adding data beyond 500K points to the buffer. This highlights that most of the lested algorithms are not very data-hungry. That is, they do not require a large data store to reach good performance, which is beneficial when they need to be employed in practical applications. The experiment highlights that the chosen datasets might lack diversity in collected experience since most algorithms appear to need only fraction of it to attain good performance. This phenomenon is made more stark in Fig. Most algorithms reach within 90% of final performance with just 50% of the data. In other words, doubling the data in the DARL benchmarks provides an aggregated uplift of 10% at best. This indicates that more effort could be put into designing a harder benchmark. The effect is less pronounced in the DMC datasets, where the algorithms continue learning till about 50% of the dataset is added and then learning begins to stagnate.
In majority of the datasets tested on DARL, CQL reaches better performance than other methods carlier in training and consistently stays at that level more data is added. TD3+BC exhibits an initial steep rise in performance but levels out at a lower score overall, or in the case of Halfchootal-medim--syy in performance as training progresses. In the visual domain, there is a similar trend with steep increases in performance for the first 100K points and much more modest gains as the dataset size doubles. These experiments highlight the need to further study the effect of dataset sizes in offline RL benchmarks since a
degrades
current algorithms appear to learn very quickly from limited data.
The learning curves for the "mixed" dataset in the D4RL benchmark and DMC are given in Fig. D In the mixed dataset experiment, most of the algorithms do adapt to the new data continually improving each time there is a shift in the dataset quality. However rarely are they able to do so in every single environment. While CQL adapts quickly in both HalfCheetah and Walker2d, it fails to learn at all in Hopper. TD3+BC outperforms all other methods in HalfCheetah, but is the worse performing in Walker2d. DT is able to consistently adapt to new data in each environment. A surprising result is how well BC performs in each environment, with BC nearly being the second best performing algorithin in all environments. We hypothesise that since there is good coverage of the state spaces and optimal behavior in the dataset, behavior cloning is able to learn effectivcly. It does not have to 'stitch' behavior from different episodes and instead just need to replicate the behavior in the dataset. This is in line with the findings of Chen et al. (202T) which found that 10%BC, which trained BC on only the top 10% percentile of episodes (based on episodic return), was competitive with specialised offline algorithms on the D4RL benchmark.
Distribution Shifts
Online Finetuning In online fine-tuning, shown in Fig. 7 and Table 3 CQL reaches higher scores com- pared to the other algorithms showing the versatility of CQL in handling both offline datasets and online interactions with the system. While in the random datasct there a jump in performance after online finetuning, there aren't large gains in other datasets. This is evident from Fig. 5c where we see that in general there are modest gains from online finetuning in these algorithms.
6

---

Published in Transactions on Machine Learning Research (11/2023)
Halfcheetah Medium Expert-V. V2
Halfcheetah Medium V/
Hopper Medium Expert-V2
100
Reward
50
20
25
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
166
1c5
166
Hopper- dium-V2
Baikker2D-Meduum-eee -xppertt-222
Walker2D- Minerr/D-Mddimm-2
80
100
Roward
40
50
2
0.0
0.5
10 Samples
1.5
Samples
165
166
Samples
165
NWAC
COL
DT Dotaset
TD3+BC
Figure Performance curves on the D4RL benchmark as a function of data points seen. Shaded regions represent standard deviation across seeds.
Cheetah Run- Nun-Expert-Sac
Cheetah Run-Med Lun-Medium-Sar
Finger Turn Hard Tard--xeet-Sae
800
1000
750
600
500
500
250
200
0-
0-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
le6
166
166
Finger Turn Hard-MMdium--S5
Walker Run-Expert-Sac
Walk Run-Medium-Sac
1000
800
750
600
600
400
200
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0,2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
1e6
le6
Samples
leb
8C0
AWAC
COL K0L
DT
TD3+0C
Datasel
Performance curves the DMC Dataset function of data points seen. Shaded regions represent standard deviation across seeds.
Figure 3:

---

Published in Transactions on Machine Learning Research (11/2023)
Expert
Cheetah Medium
Cheetah
40
20-
02
0.4
0.6
0.3
1.0
0.0
0.2
0-4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1e5
1e5
Offline Humanoid Walk Expert
Offiline Humanoid Walk Medium
Offline Humanoid Walk Random
1.5
20-
20
0.5
0.0
0.0
0.6
0.8
Offline Walker Walk Expert
Offiine Walker Walk Medi um
Offline Walker Walk Random
750
400
20
250
0.6
165
165
1d5
DIO
DIC+BC
Figure 4: Performance curves the v-d4rl benchmark function of data points seen. Shaded regions represent standard deviation across seeds.
IQM
IQM
IQM
BCQ
BCQ IQL COL
BCQ
IQL- CQL BC
IQL
CQL
BC
BC
AWAC TD3 BBC
AWAC
AWAC
TD3+BC
TD3+BC
0.90
0.95
1.00
0.00
0.03
0.06
0.09
0.00
0.03
0.06
0.09
Final Normed Score
Max Normed Difference
Uplift during Finetuning
Per @000% 1110%%
(b) Perf@100% Perf@50%
(c) Finetuming Uplift
Figure Analysis of data scaling the D4RL benchmark. PerfäX% refers to performance when the algorithm has seen X% of the dataset. a) and b): Standard Setting e) Online Finetuning setting.
4.5 Discussion of Results
A surprising insight from our experiments was how the algorithms studied learnt with a fraction of the dataset size and learning stagnated after 20-30% of the dataset was added. Further addition of data did not lead to improvements in performance. This observation was consistent across environments and algorithms. We can view these results two ways.
One, the current benchmarks/datasse for offline RL algorithms need re-evaluatod. an algorithm can perform similarly with 20% of the data 100% of the data, then these datasets might not be sufficiently
8

---

Published in Transactions on Machine Learning Research (11/2023)
Hallcheetth--lixne--2
Hopper-Mixed-22
Nalker2D-Mixedie
100
50
20-
Sampre
Sarpppes
AWAA
COL
(a) D4RL
Cheetah Rur-Mixed-Sac
Fing et Hard- Mixee--ac
Walker Run-Mixed-SS
1000
800
750
500
500
400-
250
200
Swrmp
800
AMACC
COL
DT
(b) DMC
Figure 6: Performance curves for Mixed datasets with varying dataset quality. Dotted line indicates where there is a change in the datasot generating policy distribution. Shaded regions represent standard deviation across 5 seeds.
HallEcheetah--eediian---ee
Hellch wwwlkheetaab-Beeeeee waikkeewttt---edddreeeeppp-.
Halkchest heakcheethhhkkk Medkemmme22
75-
40-
25
20
0.75
123
0.25
166
165
166
BC0
COL
K0
Figure 7: Performance curves for online fine-tuning. Each algorithm is given 500k steps in the simulator after sequential evaluation of the oflline dataset. Dotted line indicates where online fine-tuning begins. Shaded regions represent standard deviation across seeds.
difficult to properly evaluate algorithms. We believe that effort should be invested in designing harder benchmarks for this domain.
The second takeaway is that the algorithms tested were surprisingly data efficient. The model cards Figs. 5a and 5b show that most algorithms reach around 90% of final performance with less than 50% of the dataset. Presenting such summary statistics make it easy for practitioners to 1) perform algorithm selection, and 2) evaluate the benefits of additional data collection. The latter can be useful in practical domains where data collection is expensive and time consuming and the model cards can provide insights into the trade-off between better algorithmic performance and increased cost. data collection.
4.6 Comparison with Mini Batch Style Training
We compare how algorithms perform when given access the same amount of compute in the SeqEval setting and the mini batch setting. The results are given in Fig. g In the mini batch style training, the algorithm is given access to the entire dataset at all times during training. The axis in Fig. is the

---

Published in Transactions on Machine Learning Research (11/2023)
Hallcheetah-Mediun--
Halftheetah-Random-Vy
Hoppai-Medium-12
15
40
40-1
0.0
0.2
0.4
0.8
1,0
0.2
0.4
0.6
0.8
1.0
1.00
1:5
105
Hopper/Raddo--22
Walker2D Medium
Malker2D-RRnddoman
10.0
80
5.0-
40-
0.0
0.0
0.6
0.8
1.0
Gpps
GradtemSSeps
GradentStens
MirdBltch
Sectial
Daceset
Figure 8: Comparison of SeqEval training and mini batch style training.
number of gradient steps, while in the previous figures for SeqEval the axis represents the number of data points available for training. However when = 1 and K 1, the x axis can also represent the number of gradient steps, hence we plot the SeqEval and mini batch training curves together. So while the algorithm has performed a similar amount of computation when comparing the curves at a given gradient step, the data they have access to at these points can be very different. As a result, SeqEval would exhibit slower rise in performance per gradient step compared mini batch, but that because it simply does not have access to the same amount of data mini batch style training. We do that SeqEval closes the gap towards the end of training.
4.7 Additional Metrics for Evaluation
While have used the evaluation episodic return the performance metric in experiments, it is not necessary to do so. For example, if there are environments where simulator is not available for periodic cheap evaluation, off-policy evaluation metrics TThomas et al. 2015 Wang et. al. 2020 Munos & Szepesvári 2008) can be used. We provide an example of this in Fig. 9 where instead of monitoring the episodic return during training, we utilize Fitted Q Evaluation (FQE) score (Wang et al.) 2020). We followed the same implementation as given
Wang al. (2020), training the FQE estimntor from scratch each time during an evaluation phase. The FQE estimator was given access to the entire dataset during its training, while IQL still followed the SeqEval style of training. The exact FQE metric used was [E[((so, (55))], i.e. the predicted Q value of the start state distribution.
We would like to emphasize that SeqEval is independent of the evaluation metric used during training. SeqEval is style of training and can be used with different evaluation metrics depending on the constraints, for example simulator return or off- off-policy evaluation metrics such as FQE scores. The experiment above is a concrete example of how EeqEval can be used in situations where there is limited access a simulator evaluate performance.
5 Conclusion
In this paper, we propose a soquential style of evaluation for offline RL methods so that algorithms are evaluated as function of data rather than compute or gradient steps. In this style of evaluation, data is added sequentially replay buffer time, and mini-batches are sampled from this buffer for training.
10

---

Published in Transactions on Machine Learning Research (11/2023)
Hallcheetah--eedumm.
Hopper-Meduum- Mww.eddlml-------22222
400
330
200
Gov. ntxel
200
200
100
0.0
0.5
1.0
1.5
2.0
1
145
1+5
145
Hopper-Medium--y
Wallee/DD Medium Bandllu--Roeppe leeplpapal
Walker2D Medium-V2
150
Gis. nhall
100
100
0.0
0.5
1.0
1.5
2.0
3.0
Sump es
Samples
Samp
146
Figure 9: Performance of IQL in the SeqEval setting using FQE as the evaluation metric.
This is analogous to online training to deep RL and allows us to measure the data scaling and robustness of offline algorithms simultancously from the training curves. We compared several cxisting offline methods using sequential evaluation and showed how their training curves allow for algorithm selection depending on data efficiency or performance.
We believe that sequential evaluation holds promise established method of evaluation for the offline RL community. Future work in this domain could explore the effect of y and K on algorithms and their ramifications on sample efficiency.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. ar preprint w:Xiv:1912.06688 2019.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprinl Xrx::22006.00 01345, 2021.
A. Philip Dawid and Vladimir G. Vovk. Prequential probability: principles and properties. Bernoulli, 5(1): 125-162, 1999.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datascts for deep data- driven reinforcement learning, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. In Thirty- Fifth Conference on Neurol Information Processing Systems, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2062, 2019.
11

---

Published in Transactions on Machine Learning Research (11/2023)
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.); Proceedings of the 35th International Conference on Machine Learning, 1CML 2018, Stockholmsmássan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856- 1865. PMLR, 2018. URL http::/prroceeiing.. mmlr..ssmm..r.....m.........m mlr. wttp:/ss.......e////e//eoa... ppress//80/haarroja188 html
lyy Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. orXiv preprint Xxxx:2004.19443 2020.
Rya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement. learning with implicit q-learning. In The Tenth International Conference on Learning Representations,, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL httts:://ppenreviiett /ttp://ss/////aaaeeoroo//oo/o//a
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Conference on Newral Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https:// Abstract cesstract.html.
proceedings spoceedings..neuips..o www..euuw..s.p.pprrrreeeeeeeeeee ptce/paper/22220 20wp:/200//aaaa http//aas///eee 02w.0/22602022222222222221211110
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv: Arziv-2005.0100 2020.
Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack worker-Holder, Michael A Osborne, and Yee Whye Teh. Chal- lenges and opportunitics in offline reinforcement learning from visual observations. Transactions on Ma- chine Learning Research, 2023. ISSN 2835-8856. URL httpps net//foruum?it--fQ]0GGG0
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, B. Hutchinson, Elena Spitzer, Inioluwa. Deborah Raji, and Timnit Gebru. Model cards for model reporting. FAT, 2018. doi:
00.1145//287560.3287575
Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(27):815 857, 2008. URL htttp://jmlr http://tpe//mml..og///ppppppp pttp:/pppeeees///mmmsooa...a.. html
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arii preprint arXiv:2006.0933 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightet regression: Simple and scalable off-policy reinforcement learning. CoRR, abs/1910.00177,, 2019. URL https://arxiv.
09107
Jan Poland and Marcus Hutter. Asymptotics of discrete MDL for online prediction. IEEE Transactions Information Theory, 51(11):3780-3795, 2005.
Samuel Rathmanner and Marcus Hutter. A philosophical treatise of universal induction. Entropy, 13(6): 1076-1136, 2011. ISSN 1099-4300. doi:
Rissanen. Universal coding, information, prediction, and estimation. IEEE Trans. Inf. Theory, 30(4): 629 636, July 1984.
Jorma Rissanen. Stochastic complexity. Journal of the Royal Statistical Socieby: Series B (Methodological), 49(3):223 239, 1987.
Julian Schrittwieser, Ioannis Antonoglou, T. Hubert, K. Simonyan, L. Sifre, Simon Schmitt, A. Guez, Edward Lockhart, D. Hassabis, Graepel, T. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 2019. doi: 10.103/s445566-02 0200030---4
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panncershelvam, Mare Lanctot, Sander Dieleman, Do- minik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Ko- ray Kavukeuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural
12

---

Published in Transactions on Machine Learning Research (11/2023)
networks and tree search. Nature, (897748))4844488, Jamuary 2016. doi: 10.1038/naturea URL https://doi.org// 10. L038/nature163000
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362:1140 1144, 2018. doi: 10.1126/ 10.1126/science.an arrre httwiire.rrg/conee//////o///
URL htttps:/ssynicattiin
wcieecceeece......ecceiieeeeeeec 4ar404
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite. arXiv preprint Xiv: Arcio-1801.00690 2018.
Philip Thomas, Georgios Thcocharous, and Mohammad Ghavamzadeh. High confidence off-policy evalu- ation. AAAI, 2015. URL https://al..cc..ooo
28881134
V Vapnik. Principles of risk minimization for learning theory. In Proceedings of the 4th International Conference on Newral Information Processing Systems, NIPS'91, pp. 831-838, San Francisco, CA, USA, December 1991. Morgan Kaufmann Publishers Inc.
Oriol Vinyals, Igor Babuschkin na, Wojeiech M. Czarnecki na, Michaël Mathieu na, Andrew Dudzik na Junyoung Chung na, David H. Choi na, Richard Powell na, Timo Ewalds na, Petko Georgiev na, Jun- hyuk Oh na, Dan Horgan na, Manuel Kroiss na, Ivo Danihelka na, Aja Huang na, Laurent Sifre na, Trevor Cai na, John Agapiou na, Max Jaderberg, Alexander S. Vezhnevets, Rémi Teblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukeuoglu, Demis Hassabis, Chris Apps na, and David Silver na. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 2019. doi: 00088//55566-01---11
CRL https://doi. 1tt.:///....000//111-001111111-0
Jie Wang, Rui Gao, and Hongyuan Zha. Reliable off-policy evaluation for reinforcement learning. arXiv preprint ar Xiv: Arzie-2011 101004 2020.
13

---

Published in Transactions on Machine Learning Research (11/2023)
A Description of datasets
The DARL locomotion benchmark consists of three environments with varying data quality. The size of each version and the performance of the policy that generated is given in Table
Table 1: Dataset sizes in D4RL
Dataset
Size
Average score Dataset
halfcheetab-randommyy
999000
-0.07
hallcheetal-mmedimme
999000
40.60
haffchoetaa-mmciun--eppa---
201798
27.03
baffehtertab-mmedduuneeeeerryr
1998000
64.29
halfchoetab-exxeert-22
999000
87.91
waller22f-random--22
999999
0.01
walker2d--medium--2
999322
61.94
walker22t-mmdduuu--eepaaay2
301698
14.81
wallker2d-mdddum--xpprr.....
1998318
82.55
walker2dd-xxpert-v22
999000
106.90
hoppeer-aandom--22
999999
1.19
hoppper-nedium--22
999998
44.28
hopper-medduum--pplayyy22
401598
14.97
hopper-modduun-cxpee--222
1998966
64.78
hopper-expert-v2
999061
108.24
B Performance on D4RL Benchmark
We present complete training curves on all twelve datasets that were used in Fig. I1 and final performance in Table In addition to the curves, compare the algorithms the end of training with scores aggregated across environments. This is done using the rliable (Agarwal el al. 202T library to plot interval estimates of normalized performance measures such as median, mean, interquartile mean (TQM) and optimality gap. The optimality gap is a measure of how far an algorithm is from optimal performance aggregated across environments. So, lower values are better. The scores in each dataset are normalized with respect to the maximum score, which is 100. These results are given in Fig. [3. Tn both the performance curves and aggregated scores we can see that CQL outperforms other tested methods by a clear margin. A curious phenomenon observed was that AWAC (Nair al. (0200) is unable learn al all in the Walker2d environment the medium version achieving very low rewards. As sanity check we set
with either the medium-exper to the size of the dataset and reran experiments and the method achieved results similar to those reported in the paper, indicating it was not an implementation problem. This is surprising since AWAC is proposed as an algorithm that can work for online fine-tuning following offline pretraining, but among all the methods tested, had drastic changes performance when using sequential evaluation. The final performance in the online fine-tuning task and the mixod version of the environment also given in Tables and 4 The Perf@50% and Perf@100% results given in Fig. 10
Moreover, Fig. 14 show how the algorithms performed when K was set to 2. This experiment studied if we had not trained the methods for enough gradient steps and if additional performance could be extracted from the data. However, performance remained essentially the same or even degraded some instances, showing that this was not the case.
C Effect of Replay Ratio in SeqEval
The experiments in the paper studied replay ratio (RR) of The results are in Figs. 15 and 16 We
can directly compare the training and evaluation curves for different RRs and conclude whether compute (gradient-steps) training data are the limiting factor. In main experiments study the setting where
14

---

Published in Transactions on Machine Learning Research (11/2023)
Optimality Gap
Optimality
DT
DT
IQL
COL BC AWAC TD3
COL BC
AWAC
0.5
0.5
0.8
0.4
0.5
Score
Score
(a) Perf450%
(b) Perf&1000%
Figure 10: Comparison of performance when a) half the dataset is available b) full dataset is available
Table 2: Performance of each algorithm on the D4RL Benchmark
Datasct
BCQ
TD3+BC
AWAC
BC
CQL
DT
IQL
malicheeta--modhum-co
92.25
6.83
59.04
63.44
83.45
71.6
86.0
halfchhetth--mdduum--reaaaaa
37.54
44.26
43.37
36.3
43.54
30.91
41.97
halfchostah-mdduun-v2
45.47
48.45
46.58
43.12
49.13
42.65
46.96
hallcheeela--raadddmm--2
8.16
10.09
2.26
2.26
24.43
2.12
8.41
hopper--eddium-xperrr
108.8
69.92
111.88
44.53
111.0
111.38
49.18
hopper-madhum-rrpaay--222
23.9
63.04
65.47
14.02
88.51
74.94
69.09
hopper-neddium-v2
53.39
49.77
52.8
55.84
70.53
62.06
67.49
hopper-rancoom--2
7.16
8.13
9.18
2.26
6.2
7.41
7.63
wallerr2d-medduun--eppett--
110.36
108.33
1.98
107.58
109.98
108.23
98.51
walkerzsd-mccduu---pppp---22
59.94
76.8
82.54
24.63
73.31
55.04
63.12
walker22d-medium-22
76.85
83.4
1.76
78.78
83.16
65.79
80.85
walker2d-random--22
0.11
0.49
3.48
0.63
-0.12
2.48
7.82
Table 3: Performance of each algorithm the online fine-tuning task on the D4RL Benchmark
Dataset
BCQ
TD3+BB
AWAC
BC
CQL
IQL
88.02
74.36
83.6
61.77
96.72
87.74
fheetue-halkchhttaa-mmdiu---
47.86
51.9
52.89
42.86
55.29
49.24
Ineeuue-ballcheetaaaraaddou
22.29
48.53
30.67
2.26
34.17
50.87
thetuue-hopppe--medii-.eee
43.19
112.71
111.9
48.01
100.25
110.66
fueluuee-oopprr-mnddunx>2
52.35
63.03
45.45
58.92
94.31
73.59
finetune-hopper-randnn-22
14.31
9.85
8.78
2.47
4.56
12.05
108.28
107.77
1.41
108.35
110.73
110.81
fueetune-walker22knnddduu
71.26
85.01
21.32
66.47
83.7
84.24
Inetune-wwllerr2traadddoo
1.42
8.6
1.67
0.5
0.33
10.56
Table 4: Performance of each algorithm in the mixed version of the D4RL Benchmark
Dataset
BCQ
TD3+BC
AWAC
BC
CQL
DT
IQL
halfchhetahh-mixed-222
66.98
80.65
29.55
57.38
93.47
81.51
68.04
hopper-mixed-v2
55.29
112.42
110.36
86.67
0.75
111.74
51.37
walker2d-mixed--22
102.77
8.15
98.36
108.79
109.71
107.63
109.83
RR We utilized this sctting since we obscrved that increasing the compute budget did not improve performance. Lower RRs do not extract as much information as possible from the datasel and do worse. These curves provide valuable insights into the combined data and compute scaling of an algorithm.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Halfchoctah
Halfchoctah
Halfchectah
60-
23
20
0.2
0.5
1.
1.5
0.5
0.5
1.0
155
2
Hopper-Medlum Ridpliil----a---222222 Replay-V2
Hogper-andonn
75
50-
als
1/
145
Walker2D Mcdium Export
kor2D Modium Replay V2
Walker2 Medium
Walker2D Random
143
Bared
1.0
2.5
3.0
0.7
0.5
1.0
3.5 Somultes
2.0
Samsllss
Samretss
Sampls
Figure 11: Performance curves the D4RL benchmark of oflline RL algorithms function of data points Shaded regions represent standard deviation across 5 seeds.
16

---

Published in Transactions on Machine Learning Research (11/2023)
halfcheecan-Meeduu--Eeeerv-
Halfchestah-Meelum-zz
Hallcheetah-Random-VZ
40-
40
50
25
0.0
0.5
1.0
1.5
2.0
0.00
0.25
0.50
0.75
1.00
1.25
0.00
0.25
0.50
0.75
1.00
1.25
166
1e6
166
Hopper-Medium- Experrr-V2
Hopper- Medium-V2
Hopper-Random-22
60
25
0-
0-
0.5
1.5
2.0
0.00
0.25
0.50
0.75
1.00
1.25
0.00
0.25
0.50
0.75
1.00
1.25
1c6
1e6
Walker 2D Medium Me
Walker2D-Medlum--2
Walker2D Random-VV WalkkerDDRanddomWV
100
50
0-
0.5
1.25
0.25
0.75
1.00
1.25
Samples
1e6
Samples
1e6
Samples
1e6
AWAC
COL
TD3+8C
Figure 12: Performance curves for online fine-tuning. Each algorithm given 500k steps the simulator after sequential evaluation of the offline dataset. Dotted line indicates where online fine-tuning begins. Shaded regions represent standard deviation across seeds.
Optimality Gap
IQM
Optimality Gap
BCQ
BCQ
COL BC
COL
BC
BB
AWAC
AWAC
TD3
0.30
0.60
0.48
0.60
Normalized
(a) Only offline Dataset.
(b) Online fine-tuning
Figure 13: Performance aggregated across environments using rliable. For TQM higher is better, while for Optimality gap, lower is better
17

---

Published in Transactions on Machine Learning Research (11/2023)
Halicheecah-Meelumm-Eeeervv
Hallchhetta-MMediu--eeeeee
Hallcheeeahh-Meluum-2
40-
75
50-
10
0.5
1.5
0.0
1.5
le6
Hopper-MediumEExperr-2
Hopper-Medium-RRapaayyy2
Hopper-Medium-V2
100
60
20
0.0
0.5
1.5
166
165
1e5
Waalker2D-Mediunee Expert-V2
Walker2D Medium Replay-V2
Walker2D-
00-
100
50
20
0.
0.5
1.5
0.0
0.5
1.0
15
3.0
Samples
1e6
1e5
BCQ
AWAAC BC
COL 1QL
DT Dotaset
TD3+BC
Figure 14: Performance curves the D4RL benchmark with K increased to Shaded regions represent standard devialion across 3 seeds.
18

---

Published in Transactions on Machine Learning Research (11/2023)
Expn-Exppeeeee 42
Hallchostah Medium-Rapaayyeeee
Medicum
75
Rowond
20
0.5
1.0
15
0.0
0.5
1.0
1.5
Ieb
Hopper-Medurmg Expert-V2
Hopper-Medium--iee Rapala---2
Hopper-Medium-V Medium--2
100
40
25-
Whileer2D-Modiun-Rpllyyy-
Walker2D-Modlumm-2
40
40
20-
20
0.5
1.0 Samples
1.5
2.0
165
Samples
165
165
00-1
RR-10
BR-5
Decimer
Figure 15: Comparison of replay ratios greater than 1.
19

---

Published in Transactions on Machine Learning Research (11/2023)
Expertiv2
Haltcheetah-Medlnne Roplay/V2
30-
Rewon
0.5
1.0
0.0
0.5
1.0
2.0
Hopper-Meddum--B Expert-122
Hopper-Medium-22
Walkerr2 D-Mediium----- -Fpp-rr--2222
Wolker2D-Medlum-Rellay
Walker2D.Medie
300
25
1.0
0.5
1.0
255
Samplees
1e6
165
1e5
8R-0.00
80-00.0
RR-2.05
RR-0.1
RR-02
RR-1
Detteee
Figure 16: Comparison of replay ratios less than 1.
20


Reviews:
[{"review_id": "0Ny266iPjI", "reviewer": "Reviewer_gGYN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nS1: the paper is tackling an unsexy issue whose resolution has massive potential impact on the field. Kudos!\n\nS2: it’s a straightforward reutilisation of relatively simple and exhausting evaluation methods in RL, and thus easy for researchers to use.\n\n## Weaknesses \n\nW1: many of the papers cited in the lit review utilise a form of sequential eval where an offline policy is periodically evaluated out-of-training-dataset while training. This feels like very similar to the main contribution of the manuscript.\n\nW2: “distributional shift” seems like a poor way of describing the variance in data / policy space for an arbitrary dataset. The manuscript seems to assume some notion of underlying quality for an arbitrary rollout (or SARS tuple?), which gets used as some measure of change in distribution. However, that’s really quite arbitrary, given how dependent this is on how difficult the task is, what are the data (and MDP) dynamics, the dimensionality of learning policy, etc. — Ultimately, this doesn’t seem like a useful “feature” to target for a general evaluation method, unless it can really demonstrate to be useful to gain consistent insights across a variety of (env, algorithm, model) sets.\n\nW3: the method uses two params, which seem fairly important to understand the experimental dynamics. However they are immediately fixed in all experiments, and they are not explored at all. The manuscript doesn’t seem to discuss what this particular parameterization would bring to the table.\n\nW4: the fine tuning part seems out of scope wrt. standardising offline RL eval. It’s not clear this is a strength of the method, since one could very well imagine using different algorithms with different eval requirements.\n\nW5: the model card criteria don’t seem to control for different kinds of dataset sampling (e.g. uniform vs online prioritised vs pre-prioritised). Given that it’s a major source of complexity in offline RL algos, I’m not sure it is reasonable not to model it explicitly.\n\nW6: I think the rationalizations of performance of the various tested algorithms are too bold. E.g. “this highlights that most of the tested algorithms are not very data-hungry” is hard to defend, given that it’s likely a function of the task + model dimensionality + algo being used. Other statements in the manuscript have similar issues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents SeqEval, an experimental methodology for offline RL (+ finetuning) that attempts to provide a way to measure an algorithm’s dependency on dataset size and data distributional shift. \n\nSeqEval is demonstrated by comparing a few recent baselines (CQL, DT, TD3+BC, …) on a subset of DMC and D4RL environments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "C1: I think it is necessary to compare SeqEval against the evaluation setups used in the baselines (a subset is fine). The goal is to figure out how SeqEval improves the understanding of each algo as presented by their own manuscripts. It would also highlight negative trade-offs.\n\nC2: adding a sound justification for why the fine tuning bit is a good requirement to spec for (this could be done e.g. by lit review to see how many of the baselines also had finetuning experiments).\n\nC3: SeqEval’s parameterization needs to be justified both empirically as well as philosophically to justifiy the significant addition in complexity. What is the objective of its two parameters? How would researchers choose them, or even compare between two instances of SeqEval?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jSTzrvfGJZ", "reviewer": "Reviewer_krMq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n- The paper is well written and almost clear.\n- The motivation makes sense to me and I believe SeqEval will be a very useful evaluation approach to offline RL community.\n- Experiments are diverse with different settings (standard, distribution shift, online finetuning) and multiple metrics. The experimental results showcase the abilities of well-known offline RL algorithms from the angles of data efficiency and robustness.\n\n&nbsp;\n\n### Weaknesses:\n\n- Although SeqEval is easy to implement, the hyperparameters $\\gamma$ and $K$ (maybe $T_{0}$ also) introduced are not well investigated and discussed in this work. I am also aware that this may be something to be standardized or customized in the future.\n- Some experimental details are missing. Please see my Requested Changes below.\n- Some offline RL algorithms are missing in some plots without explanation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work propose a new approach of evaluation called Sequential Evaluation (SeqEval) for offline RL. SeqEval is a sequential approach to evaluate offline RL methods as a function of the training set size, rather than using the final performance trained on the whole offline dataset.\n\nSeqEval is useful to evaluate the data efficiency and the robustness of algorithms to distribution shifts. Moreover, it bridges the gap between offline and online evaluation protocols.\n\nIn the experiments, SeqEval is adopted for the evaluation of representative offline algorithms, e.g., CQL, DT, TD3+BC, BCQ and etc, on both a standard setting and a distribution-shift setting with D4RL and DMC environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I am curious about how different choices of $\\gamma$ and $K$ influence the evaluation results. Would there be different ranks obtained from different choices of $\\gamma$ and $K$. It seems that only $K=1,2$ and $\\gamma=1$ are considered in this paper. Personally, I think this is quite limited. I would like to see some more choices maybe like $\\gamma \\in \\\\{1, 10, 50, 100\\\\} $ and $K \\in \\\\{1, 2, 5, 10\\\\}$. I noticed the authors mentioned “we found that setting $\\gamma$ and $K$ to 1 worked well in all datasets tested”, what is meaning of “worked well”?\n2. Why are there some offline RL algorithms missing in specific plots, e.g., CQL is missing in Figure 3 and Figure 5(b), DT is missing in Figure 11? \n3. I would like to know whether the samples in a medium-expert dataset are maintained in order like first the medium data then the expert data or fully mixed in random. Can I consider that even in the standard setting, SeqEval on the medium-replay and the medium-expert (if samples are in order like mentioned before) datasets contains distribution changes?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "F5Y2Qv3e7c", "reviewer": "Reviewer_ZR7g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- It's a new approach for measuring the performance of offline RL that also considers the data-efficiency, which is a bit overlooked point of view in offline RL.\n- The main idea is very simple and clearly motivated.\n- Writing is clear and easy to understand the main idea.\n\nWeaknesses\n- **W1:** It seems like how to choose the part of the buffer is very critical design choice, but this is not thoroughly investigated.\n- **W2:** As a paper that introduces a new metric, there is a limited evaluation on few datasets. \n- **W3:** Considering that the promise of offline RL is to learn the behavior using the fixed logged dataset, it's not clear how the proposed metric can be used for more practical scenarios where evaluation is prohibitively costly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new evaluation methodology for offline RL, which measures the performance of algorithms in terms of data used in a similar manner to online RL, instead of measuring the performance after some arbitrarily set epochs. This is easy-to-implement because existing offline RL methods implements a replay buffer with fixed logged dataset and what the method should do is to use part of the buffer instead of using all of them. The proposed metric is investigated in D4RL benchmark and also the custom-made DMC benchmark and provide a new insight into the data-efficiency of existing offline RL algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It could be nice to provide a further detailed discussion on how to shuffle the fixed dataset for every run. Moreover, a kind of sensitivity analysis across each runs could be a nice addition.\n- Providing metrics on more datasets (e.g., other datasets on D4RL, pixel-based dataset) could make it much more easier to use the metrics proposed in the paper. And this could make the benefit of using the proposed metric more persuasive.\n- I think more investigation using the off-policy evaluation metrics and empirically showing its benefits for analyzing the performance of offline RL methods is crucial, considering the weakness W3. I noticed that authors also mentioned that `For example, if there are environments where a simulator is not available for periodic cheap evaluation, off-policy evaluation metrics Thomas et al. (2015); Wang et al. (2020) can be used.`. I think addressing this point would be very crucial for me to recommend acceptance of this paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "14mzza73w5", "reviewer": "Reviewer_wRve", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The issue investigated in this article holds significant importance. Currently, the evaluation of most offline RL algorithms necessitates interaction with the environment, which contradicts the fundamental premise of offline RL. This aspect indeed calls for improvement.\n2. The approach is generally applicable and easy to implement.\n\nWeaknesses\n1. The proposed method, which sequentially adds the existing offline data to the agent's replay buffer and periodically evaluates agent performance as a function of the available data, still requires interaction with the real environment. It does not fundamentally resolve the current issues in evaluating offline RL algorithms.\n2. There is another paper that proposes an enhancement to evaluating offline RL algorithms [1]. That paper estimates policy performance by constraining the number of online evaluation iterations, thereby reducing the frequency of interactions with the online environment to some extent. That approach appears to be more aligned with practical applications. I would appreciate the author's insights on that paper and their perspective on its potential contributions to the field.\n3. The evaluation method proposed in this article introduces additional parameters (gamma and K), which may vary for different offline RL algorithms. These parameters require extra time for tuning and introduce an additional burden.\n\n\n[1] Kurenkov V, Kolesnikov S. Showing your offline reinforcement learning work: Online evaluation budget matters[C]//International Conference on Machine Learning. PMLR, 2022: 11729-11752.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a method for evaluating offline RL algorithms, referred to as SeqEval. This approach assesses the algorithm's performance as a function of the available data, aiming to address limitations in past evaluations of offline RL methods. With this method, it becomes possible to investigate the sample efficiency and robustness of offline RL algorithms, while also enabling comparisons with online RL algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness part.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the paper tackles an important task. The main critique focused on the lack of a thorough investigation due to a limited set of experiments. 
After the clarifications and additional results provided by the authors, the majority agreed that it is a paper that should be accepted. 

This requires the addition of a detailed discussion of the new results, and I therefore recommend _Accept with minor revision_. 

The authors should
- include their clarifications and comments made by the reviewers on the theoretical basis of the model
- include and discuss the additional results they provide, especially the role of $K$ and $\gamma$.
Decision: Accept (poster)