No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL
Authors: Han Wang
Archit Sakhadeo
Adam M White
James M Bell
Vincent Liu
Xutong Zhao
Puer Liu
Tadashi Kozuno
Alona Fyshe
Martha White
Venue: Accepted by TMLR ()

Abstract: The performance of reinforcement learning (RL) agents is sensitive to the choice of hyperparameters. In real-world settings like robotics or industrial control systems, however, testing different hyperparameter configurations directly on the environment can be financially prohibitive, dangerous, or time consuming. We focus on hyperparameter tuning from offline logs of data, to fully specify the hyperparameters for an RL agent that learns online in the real world. The approach is conceptually simple: we first learn a model of the environment from the offline data, which we call a calibration model, and then simulate learning in the calibration model to identify promising hyperparameters. Though such a natural idea is (likely) being used in industry, it has yet to be systematically investigated. We identify several criteria to make this strategy effective, and develop an approach that satisfies these criteria. We empirically investigate the method in a variety of settings to identify when it is effective and when it fails.

Published in Transactions on Machine Learning Research (07 (022))
No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL
Han Wang*
han&@ualberta.co ca
Archit Sakhadco*t
sakhadeo Cuabbera..c
Adam White
amw.8vaalerta.ccm ca
James Bell
jbell1 @ualberta.co ca
Vincent Liut
oliul @ualberta.co
Xutong Zhaot
mutong&ualbertte ca
Puer Liu
puer@valberta.cr ca
Tadashi Kozuno
kuzuno@ualerrta.co
Alona Fyshet
alona ca
Martha Whitet
whilem@rulberta..com cu
Reviewed on OpenReview: https: // openreview. net/ formm? id= Ad0344434
Abstract
The performance of reinforcement learning (RL) agents is sensitive to the choice of hyperpa- rameters. In real-world settings like robotics industrial control systems, however, testing different hyperparameter configuratons directly on the environment can be financially pro- hibitive, dangerous, or time consuming. We focus on hyperparameter tuning from offline logs of data, to fully specify the hyperparameters for an RL agent that learns online the real world. The approach is conceptually simple: we first learn a model of the environment from the offline data, which we call calibration model, and then simulate learning in the calibration model to identify promising hyperparameters. Though such natural idea. is (likely) being used in industry, it has yet to be systematically investigated. We identify several criteria to make this strategy effective, and develop an approach that satisfies these criteria. We empirically investigate the method in variety of settings to identify when it effective and when it fails.
1 Introduction
Reinforcement learning (RL) agents are extremely sensitive the choice of hyperparameters that regulate speed of learning, exploration, degree of bootstrapping, amount of replay and so on. The vast majority of work RL is focused on new algorithmic ideas and improving performance in both cases assuming near-optimal hyperparameters. Indeed the vast majority of empirical comparisons involve well-tuned implementations or reporting the best performance after a hyperparameter sweep. Although progress has been made towards eliminating the need for tuning with adaptive methods (White & White, 2016; Xu al., 2018; Mann al.,
"These authors contributed equally to this work. i "Computing Science, Alherta Machine Intelligence Institute (Amit), University of Alberta, Edmonton, Alberta, Canada.

---

Published in Transactions on Machine Learning Research (07/2022)
2016; Zahavy al., 2020; Jacobsen et al., 2019; Kingma & Ba, 2014; Papini et al., 2019), widely used agents employ dozens of hyperparameters and tuning is critical to their success (Henderson et al., 2018).
The reason domain specialization and hyperparameter sweeps are possible and perhaps why our algorithms are so dependent on them is because most empirical work in RL is conducted in simulation. Simulators are critical for research because they facilitate rapid prototyping of ideas and extensive analysis. On the other hand, simulators allow us rely on features of simulation not possible in the real world, such as exhaustively sweeping different hyperparameters. Often, it is not acceptable to test poor hyperparameters a real system that could cause serious failures. In many cases, interaction with the real system is limited, or in more extreme cases, only data collected from human operator is available. Recent experiments confirm significant sensitivity to hyperparameters is exhibited on real robots as well (Mahmood et al., 2018). It is not surprising that one of the major roadblocks to applied RL extreme hyperparameter sensitivity.
Fortunately, there is an alternative evaluating algorithms on the real system: using previously logged data under an existing controller (human or otherwise). This oflline data provides some information about the system, which could be used to evaluate and select hyperparameters without interacting with the real world. Hyperparameters general, and can even include a policy initialization that is adjusted online. We call this problem Data2Online.
This problem setring differs from the standard offline or batch RT setting because the goal is to select hyperparameters offline for the agent to use to learn online in deployment, as opposed to learning a policy offline. Typically in offine RL a policy is learned on the batch of data, using a method like Fitted Q Iteration (FQT) (Ernst et al., 2005; Riedmiller, 2005; massoud Farahmand al., 2009), and the resulting fixed policy is deployed without being updatod. Our setting is less stringent, as the policy contimually adapts during deployment. Inluitively, selecting just the hyperparameters for further online learning should not suffer from the same hardness problems as offline RL (see (Wang et al., 2021) for hardness results), because the agent has the opportunity to gather more data online and adjust its policy.
We discuss a stralegy to use oflline data for selecting hyperparameters for learning from scratch in deploy- ment. The idea simple: we use the data to learn a calibration model, and evaluate hyperparameters in the calibration model. For example, consider designing learning system for controlling a water treatment plant, given only a set of data logs, visualized in Figure 1. We want an agent to control pump speeds and chemical treatments to clean the water with minimal energy usage-but but how do we set the learning rate and other hyperparamnterss of this agent? We can learn a calibration model offline from data logs previously collected while human operators controlled the plant. The agent can be tested with different hyperparameter settings in the calibration model, gauge which resulted in best online performance (e.g., reward accumulation or stable learning). The agent with these best hyperparamcters then deployed on the real plant.
Calibration model
Plant
A
A
Data logs
Deployment
III
Hyper evaluation
Figure 1: Data2Online: each hyperparameter setting is denoted by A. The plant imagined by the agent (via calibration model) is intentionally pixilated reflect approximation of the true plant.
This idea is natural, and versions of it are likely being used in industry already. For example, one could first construct construct a high-fidelity simulator and then use RL. This strategy was taken for impressive applications like for flying balloons (Bellemare et al., 2020) and for controlling plasma (Degrave et al., 2022). Still other work has learned slightly less high-fidelity simulators, using (recurrent) neural networks, prototype and compare algorithms, such as for gas turbine control (Schaefer et al., 2007). These are natural steps, to understand which algorithms are effective, before deploying on real system!
2

---

Published in Transactions on Machine Learning Research (07/2022)
The primary novelty in work is to investigate learning and using calibration models black-box algorithm for Data2Online, particularly in settings where we do not have access to physics simulator of the deployment scenario. Instead of considering the use of a simulator as part of the engineer's prototyping, we consider the learning and use of a calibration model as part of an automated process. The input to the automation pipeline is logged data from interaction under the current (human) operator and the outpul is an agent that learns in deployment (not static policy). It may seem like a hopeless endeavour, considering the evidence in Sim2Real and in model-based RL suggesting small errors in models can result in poor policies (Talvitie, 2017; Höfer et al., 2021). However, the key distinction for our setting is that we identify hyperparameters, rather than learning a complete policy. The calibration model need not be a perfect simulator to be useful for identifying reasonable hyperparameters, whereas learning a transferable policy likely requires accurate models.
This automated process is what we call Data2Online, and there has as yet been very little work understanding theoretical limits, algorithms and potential failure modes. To the best of our knowledge, there is only one other work that considers how to use fixed offline dataset to evaluate an agent that learns online in deployment (Mandel et al., 2016). Their approach uses novel strategy to reason about how the policy changes during lcarning, but is only effective for short horizon problems.
In this paper, we first introduce our Data2Online strategy, and outline conditions on the calibration model and learning agents for this strategy to be effective. We bound the difference in the value in the real environment of the hyperparameters chosen in the calibration model, to the true best hyperparameters, in terms of the calibration model error and length of interaction. We then develop a ponaaaamttiic calibration model, based k-nearest neighbors and carefully chosen distance metric, that satisfies the conceptual criteria needed for the calibration model. We investigate the approach in three problems with different types of learning dynamics, two different learning agents, under different offline data collection policies, and with ablations on the key components of our proposed calibration model. We conclude by highlighting that grid search can be replaced with any hyperparamcter optimization algorithm, and that this further improves performance in the real environment.
2 Related Problem Settings
Offline RL involves learning from dataset, but with a different goal: to deploy fixed (near-optimal) policy. As result, the hyperparameter selection approaches for offline RL are quite different. One strategy that has been used is to evaluate different policies corresponding to different hyperparameter settings, under what has been introduced as the Offline Policy Selection problem (Yang et al., 2020). This setting differs from our sctting in that they evaluate the utility of the learned policy, rather than the utility of hyperparamoters for learning online, in deployment. Some work in offline RL examines learning from data, and then adapting the policy online (Ajay et al., 2021; Yang & Nachum, 2021; Tee et al., 2021), including work that alternates between data collection and high confidence policy evaluation (Chandak et al., 2020a;b). Our problem is complementary to these, as strategy is needed to select their hyperparameters.
In Sim2Real the objective is to construct high fidelity simulator of the deployment setting, and then transfer the policy and, in some cases, continue to fine tune in deployment. We focus on learning the calibration model from collected data, whereas in Sim2Real the main activity is designing and iterating on the simulator itself (Peng et al., 2018). Again, however, approaches for Data2Online are complementary, and even provide another avenue to benefit from the simulator developed in Sim2Rcal to pick hyperparameters for fine tuning.
Domain adaptation in RL involves learning on set of source tasks, to transfer to target task. The most common goal has been to enable zero-shot transfer, where the learned policy is fixed and deployed in the target task (Higgins et al., 2017; Xing et al., 2021). Our problem has some similarity to domain adaptation, in that we can think of the calibration model as the source task and the real environment as the largel (ask. Domain adaptation, however, is importantly different than our Data2Online problem: (a) in our setting we train in a learned calibration model not a real environment, and need a mochanism to learn that model (b) the relationship between our source and target is different than the typical relationship in domain adaptation and (c) our goal is to select and transfer hyperparameters, not learn and transfer policies.
3

---

Published in Transactions on Machine Learning Research (07/2022)
Learning from demonstration (LfD) and imitation learning involve attempting to mimic extract policy at least as good as demonstrator. If the agent is learning to imitate online, then it is unrealistic to assume the demonstrator would generate enough training data required to facilitate hyperparameter sweeps. If the learner's objective is to imitate from a dataset, then this is exactly the problem study of this paper. Unfor- tunately, hyperparameter luning in L.D usually not addressed; instead it is common to use explicil sweeps (Merel et al., 2017; Barde et al., 2020; Ghascmipour et al., 2019; Behbahani et al., 2019) or manual, task- specific tuning (Finn et al., 2017). Hyperparameter tuning, however, is a major obstacle to the deployment of LfD methods (Ravichandar et al., 2020).
Finally, there is large literature on hyperparameter selection in RL. Most introduce meta algorithms that learn hyperparameters, including work on meta-descent. for stepsizes (Sutton, 1992; Xu et al., 2018; Jacobsen et al., 2019) and selecting the trace parameter (Downey & Sanner, 2010; Mann et al., 2016; White & White, 2016). These algorithms could be beneficial for offline hyperparameter selection, because they help reduce sensitivity to hyperparameters; but they are not a complete solution as they still have hyperparamoters to lune. Other work has provided parameter-free methods that have theoretically defined formulas for hyperparameters (Papini et al., 2019). Deriving such algorithms is important, but is typically algorithm specific and requires time to extend to broader classes of algorithms. Finally, recent work has examined online hyperparameter selection, using off off-policy learning to assess the utility of different hyperparameters in parallel (Paul et al., 2019; Tang & Choromanski, 2020). Otherwise, much of the work has been focused on settings where it is feasible to obtain multiple runs under different hyperparameters such as in simulation with the goal to improve on simple grid search (Srinivas et al., 2010; Bergstra & Bengio, 2012; Snock et al., 2012; Li et al., 2018; Jaderberg et al., 2017; Falkner et al., 2018; Parker-Holder et al., 2020).
3 Problem Formulation
In RL, an agent learns to make decisions through interaction with an environment. We formulate the problem a Markov Decision Process (MDP), described by the tuple (S,A, P). S is the state space and A the action space. : 5 A :
S + R is the reward, a scalar returned by the environment. The transition probability S x A - [0, 1] describes the probability of transitioning to a state, for a given state and action. On each discrete timestep the agent selects an action A, in state S,, the environment transitions to new state S,+1 and emits a scalar reward Re+1-
The agent's objective is to find a policy that maximizes future reward. A policy : S - [0, 1] defines how the agent chooses actions in each state. The objective is to maximize future discounted reward or the return, G. =/2 Re+1 + Pe+1G++ for discount 7++ E [0, 1] that depends on the transition (S,, Ar, S2+1) (While, 2017). For continuing problems, the discount may simply be a constant less than 1. For episodic problems the discount might be 1 during the episode, and bocome zero when St, A₁ leads to termination. Common approaches to learn such a policy are Q-learning and Expected Sarsa, which approximate the action-values the expected return from a given state and action and Actor-Critic methods that learn a parameterized policy (see (Sutton& & Barto, 2018)).
We assume that in the DataeOnline setting the agent has access to an offline log of data that it can use to initialize hyperparameters before learning online. This log consists of Adata tuples of experience D = (SS, Ar, Re+1. Se+1, [)] generated by interaction in the environment by previous controller or controllers.' For example, an agent that will use Expected Sarsa might want to use this data to decide on a suitable stepsize a, the number of layers in the neural network (NN) architecture for the action-values and even an initialization Gg for the NN sarameters-namlyy a policy initialization. There are several options for each hyperparameter combination, À= = (a, 0o), resulting in a set of possible hyperparameters A to consider. This set can be discrete or continuuus, depending on the underlying ranges. For example, the agent might want to consider any a E [0, 1] and a 6, only from a set of three possible choices.
Procedurally, the Data2Online algorithu is given the datasel D and the set of hyperparameters A, and outputs a selected hyperparameter setting X. A good choice is one that is within s-optimal of the best
I 'Coing back our water treatment example, this data might have been collected over the last two years using actions from human operator. We expect the data is likely from reasonable or nearly optimal policy. Before letting our agent learn and control the water treatment plant, it is sensible to use this already available data improve performance online.
4

---

Published in Transactions on Machine Learning Research (07 (0222)
hyperparameters
Perl(A) 2 max Perl(A) - AEA
where (e) is the online performance of the agent, when deployed with the given hyperparameters. Typ- ically, this will be the cumulative reward for continuing problems and average return for episodic problems, for fixed number of steps T. Many hyperparameters may allow the agent perform well, so focus on nearly-optimal performance under À rather than identifying the best hyperparameters.
The central question for this Data2Online problem is: how can the agent use this log of data to select hyperparameters before learning in deployment? This is no easy task. The agent cannot query Perf(A). It is not only evaluating a fized policy, for which we could use the large literature on Off-policy Policy Evaluation. It is evaluating a learning policy. In the remainder this paper, we introduce and test a new algorithm for this Data2Online problem.
4 Data2Online using Calibration Models
This section introduces the idea of calibration models and how they can be used for hyperparameter selection. We first discuss how to use the calibration model to select hyperparameters, before providing a way to learn the calibration model. We then discuss certain criteria on the calibration model and agent algorithm that make this strategy more appropriate. We conclude with some thcorctical characterization of the error in hyperparameter selection, based on inaccuracies in the calibration model.
4.1 Using Calibration Models to Select Hyperparameters
A calibration model a simulator of the environment leaanneett--erreed from an offiine batch of data used to specify (or calibrate) hyperparameters the agent. With the calibration model, the agent can test different hyperparameter settings. It evaluates the online performance of differrnt hyperparameter choices in the calibration model and selects the best one. It can then be deployed into the true environment without any remaining hyperparameters to tune.
The basic strategy is simple: we train a calibration model, then do grid search in the calibration model and pick the top hyperparameter, as summarized in Algorithm 1. For each hyperparameter, we obtain a measure of the online performance of the agent across Aruns in the calibration model, assuming it gets to learn for Nateps of interaction. The pscudocode for AgentPerfInEnv is Algorithm 2, for the episodic sctting where we report average returns during learning. Note that we add a cutoll in the evaluation scheme to guaranlee that at least 30 episodes are seen by the agent during training in the calibration model. We cut off episodes that run longer than Mateps/30 steps, teleporting the agent back to start state.
A Algorithm Hyperparameler Selection with Calibration Models using Grid Search Input: A: hyperparameter set for learner Agent D: the offline data log
nsteps: number of interactions or steps
Pruns: number of runs
Train calibration model C. with 'D
for in do
Perf[A] = AgentPeriInnwv(C Ag)nt(A), Nsteps: Nruns)
(see Algorithm 2)
end for
Return: arg maxxEa Perf[A]
Many components in this approach are modular, and can be swapped with other choices. For example, instead of expected return during learning (online performance). optimizing the hyperparameters might be more desirable to find the best policy after a budget steps. This would make sense if cunulative reward during lcarning deployment was not important. We might also want more robust agent, and instcad of expected return, we may want median return. Finally, the grid search can be replaced with more efficient hyperparameter selection method; we discuss this further in Section 7.
5

---

Published in Transactions on Machine Learning Research (07/2022)
Algorithm 2 Agenl
Input: C: calibration model, Agent: learner, Pateps: # of steps, Pruna: # of runs
Reutolf <- nsteps/ /0 > Ensure there are at least 30 episodes
Returns Across Runs= =
for 1.... Tryns do
t O, 0, Returns |] 8 random start state from C
for = Msteps do
Obtain from Agent(s), obtain s',r= C(s,u), give 8rr to Agent
G - G+++7'
t + 1++1
if s' is terminal or t > Reutoff then
Append G to Relurns
random start state from C, t 0, end end for Across < averagel (eeerage(Returns) Returns)
end for Return: dergge(lcturms
We can also make this hyperparameter search more robust to error in the calibration model by oblaining performance across an ensemble of calibration models. This involves using Rensembles random subsets of the log data, say by dropping at random 10% of samples, and training Pensembles calibration modcls. The hyperparameter performance can either be averaged across these models, or a more risk-averse criterion could be used like worst-case performance. Using an average across models like using set of source environments to select hyperparameters rather than a single source and could improve transfer to the real environment.
These are all additions to this new Data2Online strategy. The central idea is to use this calibration model to evaluate hyperparameters, as if we had a simulator. We, therefore, investigate this idea first in its simplest form with expected returns, grid search and only one calibration model.
4.2 When is this Approach Effective?
This section highlights three conceptual criteria for designing the calibration model and selecting agents for which Data2Online should be effective. This includes 1) stability under model iteration, 2) handling actions with low data coverage and 3) selecting agent algorithms that only have initialization hyperparameters, namely those that affect early learning but diminish in importance over time.
Producing reasonable transitions under many steps of model iteration is key for the calibration model. The calibration model is iterated for many steps, because the agent interacts with the calibration model as if it were the true environment for an entire learning trajectory. An agent often relies on random exploration to discover good rewards before learning can adapt the policy. Thus the first few episodes may be extremely long before a goal state is found. In Mountain Car, for example, a linear function approximation agent takes over 65,000 steps to wander to the goal, and a nonlinear function approximator may take even longer.² 2
It key, therefore, that the calibration model be stable and self-correcting. A stable model one where, starting from any initial state in region, the model remains in that region. A self-correcting model is one that, even if it produces a few non-real states, comes back to the space of real states. Otherwise,
"It might like solution is to have short episode cutoffs. However, some deployment scenarios may allow short cutoffs, and want calibration model reflect the real-world best Further, in continuing problems there terminations all. Finally, cutting off episodes typically makes the exploration problem much easier and thus may indeed impact the hyperparameters selected.

---

Published in Transactions on Machine Learning Research (07/2022)
model iteration can produce increasingly non-sensical states, as has been repeatedly shown in model-based RL (Talvitie, 2017; Jafferjee et al., 2020; Abbas et al., 2020; Chelu et al., 2020).
The model also needs to handle actions with no coverage, or low coverage. For unvisited or unknown states, the model simply does not include such states. The actions, however, can be queried from each state. If an action has not been taken a state, nor a similar state, the model cannot produce a reasonable transition. Any choice will have limitations, because inherently we are filling in this data gap with an inductive bias. A common choice in offiine RL is to assume pessimistic transitions: if an action is not observed, it is assumed to have a bad outcome. This ensures the learned, fixed policy avoids these unknown areas.
The choice is even more nuanced in Data2Online. Just like offline RL, it can be sensible to avoid these unknown actions, to answer: in the space known to the agent, what hyperparameters allow the agent to learn quickly? But, another plausible alternative is that we want to select hyperparameters to encourage the agent to explore unknown areas, since once deployed the agent can update its policy these unknown areas. In other words, a principle of optimism could also be sensible. Selecting the right inductive bias will depend on the environment and the types of hyperparameters we are selecting. This question will likely be one of the largest questions in Data2Online, similarly to how it remains open question in offline RL.
The third criterion is a condition on the agent, rather than the model. Practically, we can only test each hyperparameter setting for a limited number of steps in the calibration model. So, the calibration model is only simulating early learning. This suggests that this approach will be most effective if we tune initialization hyperparameters: those that provide an initial value for constant but wash away over time. Examples include an initial learning rate which is then adapted; policy initialization; and an initial architecture that is grown and pruned over time.
These criteria are conceptual, based on reasoning through when we expect success or failure. We use these conceptual criteria to propose an appropriate approach to learn a calibration model in the next section. In addition to conceptual reasoning, theoretical understanding of the Data2Online problem setting is also critical. We provide a first step in that direction in the next section.
4.3 Theoretical Insights
This problem has aspects that both make it harder and potentially casier than standard offline RL problem. One aspect that makes this problem harder is that we have to evaluate a learning policy offline, rather than fixed learned one. A fixed policy can be assessed using policy evaluation, and there exists variety of theoretical results on the quality of those estimates, many based on the foundational simulation lemma (Kearns & Singh, 2002). No such results exist for evaluating a policy that will be changing online.
At the same time, intuitively, the problem could be easier than the offline RL problem, because the policy adapts online. Instead, we only have to select from potentially small number of hyperparameters, rather than from potentially large policy space. For example, it may be relatively easy to identify the best stepsize out of a set of three stepsizes. Further, if policy learning algorithm is chosen that is robust to its hyperparameter settings, then the problem may be even simpler. For example, it may be simple to select the initial stepsize for an adaptive stepsize algorithm, where the initial stepsize only influences the magnitude of updates for the first few steps.
We first extend the foundational simulation lemma to the Data2Online setting, in Theorem 1. Then, in Theorem 2, we show how to use this result, to bound how far the value of the hyperparannttess chosen in the learned calibration model are from the best hyperparameters. Finally, we discuss how it might be possible to formalize this second intuition, for future theoretical investigation.
We start by defining some needed terms. An online learner can be characterized by a history dependent policy (see Chapter 38 of (Lattimore & Szepesvári, 2020)). A history dependent policy is म = (no, R1, .... where n, H4 + A(A) and H, = (S x A R)' x S the history time stop For simplicity, we assume the rewards are deterministic in [0, rmax] and the MDP has one initial state sg. The online learning agent interacts with the environment for 7 steps in total, in either a continuing or hored-n setting.
7

---

Published in Transactions on Machine Learning Research (07 (07/2022)
The value function for this online learner π is the sum of rewards from time t to the end of learning at T
T-1
(h+) E
Σ A()
He he
where the expectation is under A ~ me(- Hi), S441 ~ P(: St, At). Note that (x0) is the step objective that we use select hyperparameters. For the fixed-horizon setting where episodes are of length horizon, T K horizon where K is the number of episodes. In this setting, the expectation under Si+1 P(. St, Ac) if t is not the last step of an episode and S411 So if is the last step of an episode. Dividing VO (8o) by K gives the average episodic return over K episodes.
Theorem 1 (Simulation Lemma for Online Learners). Assume the rewards r(8, a) are deterministic in [0, mmazl and the MDP has one initial state so Suppose we have a learned model (P, f) such that
|P s,a) - P(. s, Epp and [r(s, a) f(s,a)) Er for all (s.a) E SXA
and that P(s,a) a) [maz). Let (.) denote the value function under the learned model. Then for onyy
history dependent policy n, we have that
IV (so) (80)) S S EpTmaaz? T2
Proof. We follow the proof of the simulation lemma. Since the rewards are deterministic, the history does not need to contain reward, that is, H4 (8 A)t x S. For any h4 = (Su. (su.a,....,88) H4 with t < T 1,
(h.) Σ te(a| he)r(s:, a) + Σ
Tr(a ht) Σ P(s,a, 8')
aEA
acd
d'ES
(h,) Σ
xe he)f(sr, + Σ
(e(a h+)
Σ
P(s, a 8') s')
BEA
acA
s'ES
and for the last step we have
(hr-)1 = Σ
-r- hr ())s))(()-,,a)) and ( = Σ
Tr- (())))))
aEA
aEA
We prove the simulation lemma from the last step. For t = -
( - Di= (
Σ पт I (1)
गT TT- /T (( Se. Σ
aEA
aEA
For all 1 TT
(h+) - 177 (p()) Σ Te(a be)r(st, + Σ me(a hx)
Σ
P(x als,,,8')VV (1)
aEA
aEA
s'ES
Σ Te(a ht)P(ss, - Σ Te(a h4)
Σ (0,
aEA
aEA
s'ES
Σ ne(a| hi)) Σ P(s,a,8')Vi (VH - Σ (l,
< E, +
aed
S'E
s'ES
Σ me(a ht))
Σ
EP(s,,
P(s,a, 8' (H,,,, - Σ s'),a,
E,
aEA
YES
s'ES
P(s, 8'V., 8')) Σ s'
S'ES
S'E
S +
Σ
me(a ha)l Σ (P(s,a,,') - 8)) V Vit ((h,,,,)) 8))
aEA
FES
5(T-1-1)mm
+ Σ
Telal h4)) Σ
P(s, a, s 5V) (( - ((he,, s')))
aEA
s'ES
8

---

Published in Transactions on Machine Learning Research (07/2022)
,, Tela (T) - 1 max
Σ
- a(s,,,8') -)
max V +-maxiVH (1b,, (
a.d
e'ES
S + Epp max (T - -t-1] 1) + max V ((,,, s')) 1/7 (1,,,
Therefore, (so) - (so) I + EpTmax - 1) + te + Eprmax + E + EpTman'
2
Theorem 1 says that if we have a good model of the environment, we can evaluate the T step objective for any online learner with bounded error. In particular, we can control this evaluation error by controlling the error of the learned model. Note that Umax = Trmax is the maximum value, so the last term 2 be interpreted as SpDmaxx mcaning the bound scales with T: (e, + Expmmaxy )T.
Sprmas T
can
2
Back to our problem setting. Let A be the set of hyperparameters and Tλ be a learner's policy with À E A. In our algorithm, we choose the best hyperparameters based on VoT (so), which is an estimator for ViJ (so) by rumning n runs with P. Let Î = arg maNλea V/T*A (su) be the hyperparameters returned by our algorithm and X arg maxxea V0a (sp) be the true best hyperparameters in the set. The following theorem shows that our hyperparameters will not be too far from the best hyperparameters in terms of the T step objective.
Theorem 2. Under the same conditions Theorem 1, with probability
6, we have
2 In (4/6)
VOTA (so) - *0"((s)) < 22,T + Eprmes T2 + Tmool
2 2n (4/6) (2er + Epemar) T + Umar
n
approzimation
estimation error
Proof. By Hoeffding's inequality, for a given n, we have with probability 1- 6/2 that
In (4/8)
(s0) Vo5 (so)) Tr max
2n
because the return in each run, to give the sample average (so). is in 10. Trmax]. Using the union bound, we can say this inequality holds for both πX- and T. with probability 1 1-8. 8. The source of this difference is from using limited number of runs to approximate V0T (so). As we increase the number of runs n, then the difference between our estimator Vi5 (8o) and V0 (s0) goes to zero.
Now we can reason about the hyperparameters chosen using X = arg maxxEA (so).
V - (50) (so) = Vita (so) V (80) + (so) - 10 (s0)
S ViTaa (so) ViTTA (so) + ()TA(s)) (/0((s)) (5o)
Vo (oo) - Vy (so) + 1T (sp) PTTT (80)
+ (s0) - (s0) <s00 (sa)
< 1VTA (so) 1/- (so) + 18 (so) V(-Aa (so)
+ 1i/TT' (sp) 1/TT* (s0) + 11, (5o) - V.7 *so))
In (4/8)
< 2 max VOTA (so) - Vot (so) + 2T max
xEλ
2n
2 In (4/0)
$ 20,T + Ept max T + Fmax TV
The last inequality follows from Theorem
This result is a sanity check that we can reason about error in identifying hyperparameters based on model error. However, it has several limitations. One limitation is that the result. is for continuing problems
9

---

Published in Transactions on Machine Learning Research (07 (07/202) /022)
Find k nearest neighbors in dataset i(s', at, r"), using d(((), a'), ap))
Calibration Model
Sample start state CB
Pick (s), a,, r") proportionally to distance
Let suz =s", ,,, =r'
%
30
Serr fer
Sample action
Update with Suy
Sample action
Agent
8 ~ x(1ss,
0, ~ nl =
t=
t=t+1
Figure 2: The interaction between the calibration model and the agent in cach episode.
and fixed-horizon episodic problems, but not variable length episodic problems. The analysis does not address variable length episodic problems, because it would impact both the histories which policies are conditioned as well as the definition of the value function for this learning policy. The more important limitation, however, is that the bound depends on the length of interaction T, and worse on the squared length of interaction 22 if we assume Unaa = Trmax- Even if episodes are short, we expect the agent to learn for thousands of steps and so T can be quite large. It may be difficult obtain sufficienlly low s (model error) control for this accumulating error over many steps.
We could potentially obtain better result by considering smoothness in performance with respect to hy- perparameters. Empirical studies suggest performance changes smoothly as function of hyperparameters, when hyperparameter sensitivity plots are shown. We hypothesize there exists a subset of hyperparameters such that V (so) is smooth w.r.t. the hyperparameters in the subset, and hyperparameters outside the subset have very low * Therefore, the error bound from Theorem just needs to be smaller than the performance gap between hyperparameters in the good subset and hyperparameters outside the good subset guarantee finding a nearly optimal hyperparameter setting. This direction an important next step.
5 Stable Calibration Models with KNNs
We develop non-paramone canpparamctric k-nearest neighbor (KNN) calibration model that (a) ensures the agent always produces real states and (b) remains in the space states observed in the data, and so is stable under many iterations. The idea simple: the entire offline data log constitutes the model, with trajectories obtained by chaining together transitions. Figure 2 shows the interaction between the calibration model and the agent in each episode. The KNN model is easy to use, which is relevant for those trying to get things working in the real world. In addition, the KNN calibration model is extremely lightweight and allows fast simulation, which is essential for our application: training possibly hundreds of RL agent's 's with different hyperparameter con configurations from scratch.
There are, however, several nuances using this strategy to obtain calibration models. In particular, the method relies heavily on the distance metric used to find nearest neighbors. Further, the dataset is limitod, and may have poor coverage for actions in certain states. We start by introducing the basic approach, and then discuss these two muances in the following two subsections. We conclude by contrasting the approach to other ways to learn the calibration model, particularly with neural networks.
5.1 The KNN Calibration Model
The calibration model needs to produce a (stochastic) outcome next state and reward r, given state and action We produce novel trajectories from dataset, by noting that if state-action pair (st, (s))a; is
10

---

Published in Transactions on Machine Learning Research (07/2022)
Algorithm 3 Learn KNN Calibration Model
Input: dataset DD with tuples of the form A,,, R), number of nearest neighbors (default 3) Constructs: Representation &e for distances, Tree Trees for fast nearest neighbors search, Rdefoult default return
1 Sesaplaaeeeeprrannnng in Algorithm 11
Trees (eoDttrrCoouttuucttonntt eb, D) in Algorithm 5
Extract starting states B _ D
Set Raefault to minimum return in dataset (pessimistic default return)
Algorithm 4 Sample KNN Calibration Model
Input: State St, Action o; if action given, procodure returns a start state if No action is given then
return Sample 8 E B uniform randomly end
1/ Find k nearest neighbors to 8t, 4r, to get potential next states and rewards (5 Tá, di) + KDTreeSearch((j(s)
Trees, k) in Algorithm 7 // Tf closest neighbor is far, then return a default return and terminate if min; > threshold then
r + Raefault, s' t terminal return (r,8') end
Sample i E [1, according to probabilities p: Eyen, return (ri,s))
de
similar to (Ss. 0z) for stored tuple (st, at Ti, 8(), then it is plausible that ra, s could also have been observed from (st, de). To allow for stochastic transitions, the k most similar pairs to (8),4)) can be found, and the next state and reward selected amongst these by sampling proportionally to similarity.
More formally, given the current state-action pair (s,, a,), the model searches through all tuples (s, a, rr and selects the & nearest neighbors, according to similarity between (st, 0e) and (s,a). (We discuss how to compute similarity Scction 5.2.) Let {(s;; Gi,
Ti, Ti, s correspond to these tuples, and de to the distance
between (st, ar) and (si,a;) a.). Then these (ra, s() are all possible outcome rewards and next states, where the likelihood corresponds to similarity to a.). If az) is very similar to (st, a4), then (r,,,() is likely outcome. Otherwise, the more dissimilar, the more unlikely it is that (ra, is a plausible outcome. The tuple is sampled proportionally to P;- = ",e()aadd
d; where a smaller distance indicates higher similarity.
This procedure is summarized in Figure At the start of each episode, a start state so is sampled randomly from the set of start states in the dataset. The agent takes its first action ag, to get the first pair (su, ag) and the k nearest neighbors are found. This process continues until the agent reaches terminal state, or the episode is cutoff and the agent teleported back to a start state. An overview of learning the KNN calibration model is given in Algorithm 3 and sampling the model in Algorithm 4.
There are several details worth mentioning in the algorithms. First, a KNN model relies heavily on an appropriate distance. For example, for input states that correspond to (a, y) position, Euclidcan distance wall in the environment, two states might be similar in
can be poor measure of similarity. If there Euclidean distance, but far in terms of reachability with notably different dynamics. We ameliorate this by learning a new representation-callle a Laplace representation-vtt and using Euclidean distance in this new space that better reflects similarity in transition dynamics, as described in Section 5.2.
Second, there may be no similar pairs in the data for a given (s, a). The state is one that is observed in the dataset, but the action may not be since is selected by the agent running in the calibration model. When the next outcome state 8t+1 is chosen from st, the agent selects action ă+t1- The dataset might contain multiple transitions from states like 8111 including of course the transition that includes 8t+1 but
11

---

Published in Transactions on Machine Learning Research (07/2022)
these may be for only a subset of the actions. If none of these transitions uses u,+1, then the dataset has insufficient coverage to infer what might occur when taking that action in the environment. When this occurs in Algorithm/ 4 when the closest point (minimum distance) is too far away (above a threshold) we set the return to a default return and terminate the episode. We discuss an appropriate choice for this default return in Section 5.3.
Finally, we want ensure that the model is efficient to query, even if we have a large dataset. For the discrete action setting, it is possible to get an ((1) look-up by caching the nearest neighbors upfront. For datapoints, for each action we construct a table with n rows and k columns for the nearest neighbors, where cach neighbor is stored as its index from 1 to Each transition consists of jumping between rows, using these indices. The detailed pseudocode is provided in Appendix A.1. More generally, for continuous actions, we can use a k-d tree (Bentley, 1975) to search for the k-nearest neighors. The k-d tree takes the transformed state-action pair; (1b(s), a) as the key for the search. For a datasct of size n, it costs On log n) to construct the k-d tree and O(log n) to query for nearest neighbor. This low computational complexity is key to allow to use all of the data to create our calibration model.
5.2 Improving the Distance Metric for the KNN
It is not hard to imagine examples where Fuclidean distance on states or observations does not appropriately reflect similarity of states. For example, in maze environment, if inputs correspond to (z, y), two nearby points in Euclidean distance may actually be on opposite sides of a wall, thus far apart in terms of transition dynamics. Similarly, Euclidean distance does not apply to images, since pixel-wise difference can make every image look very diffcrent from all the others in the datasct.
Instead, we exploit a standard approach in metric learning: we first map the inputs to a new space where Euclidean (E2) distance is meaningful. In particular, we would like a new representation (ss) where states at and 8, that have similar outcomes in terms of states and rewards are mapped to similar vectors (s;)) ss (s;)), and ones with dissimilar outcomes are mapped to different representations.
Such representations that reflect similarity in terms of transition dynamics have been explored under what are called Laplace representations (Wu al., 2019). The approach relies on having a stored trajectory that maintains the order of interaction. The objective includes two components: an attractive term that encourages two states that are nearby in the trajectory to have similar reprosentations, and repulsive term that encourages randomly sampled states to have different representations. For neural network the with parameters 0, the last layer of the NN (8) has loss
Σ (1)0()))) 20o (00 + Σ REND
((o(ss)) (1)0 (1
The inclusion of representation norms -llvo ensures that the representation is not simply decreased to zero to satisfy the first attractive term. Minimizing this objective encourages ([)) (be()))) to be small for states right beside each other in the trajectory-tempnne the repulsive term that encourage random pairs to have ort hogongll representations. It is possible for St, 8141 to be randomly selected for the second term, but this is not that likely under the possible n² pairs; the first term dominates, ensuring these nearby points have similar representations. More details on learning the Laplace representation are given in Appendix A.1.1.
close. The second term (woo(s:) (s,))) is
The distance for state-action pair is defined differently for discrete and continuous actions. For discrete actions, two actions are considered similar only when they are exactly the same. The resulting distance is
da(si, 83) if a; = aj d((si, ac)s (8j,a;))
for d,(s,, 85) = [1(((s)) (
8
clse
In practice, we simply keep separate data structures for each action, find nearest neighbors. For continuous action problems, the Laplace representation can actually be learned on (s,a) directly, to obtain a))
12

---

Published in Transactions on Machine Learning Research (07/ (222)
5.3 Insufficient Data Coverage
We do not require the datasct to have perfect state and action space coverage. We only query the KNN calibration model from states that are in the dataset, by construction. But, for a given action a, there may be no state-action pair that is similar to (8, a) and so there is insufficient information about the outcome for that pair. What then should the model return?
A natural choice is to truncate the episode, provide a default return-as as if the agent had managed visit future states in the environment-and transition back to the start state. This synthetic interaction in the calibration model is inaccurate, so encourage the agent to learn within the parts of the calibration model that meaningfully reflect the true environment and avoid these unknown areas. This suggests using a pessimistic default return. The default return can be set to the minimal return experienced in the dataset. When the agent reaches these unknown state-action pairs, it receives a low return and on the next episode is less likely to come back to this unknown state-action pair.
Pessimism has also been used in offline RL, but for a subtly different purpose than here. The goal of pessimism in offiine RL is to avoid unknown arcas, as it is unlikely for the fixed policy to be good in region that it has never observed and further that unknown region may be dangerous. It is much safer to stay within the data distribution, and simply improve performance within that region.
For us, the policy can adapt online if it reaches unknown arcas, so is not necessary to ensure the agent avoids them in the environment. But, we avoid encouraging the agent to visit these unknown areas in the calibration model because they are not reflective of the true environment, potentially skewing hyperparameter selection. For example, if the agent was instead encouraged to visit these state-action pairs (using optimism), then it might find an unknown state-action pair and spend most of its time visiting it. The hyperparameters would be tuned to work well for this interaction, with short episodes and (default) Monte-carlo return from this state-action that do not require any bootstrapping. Our primary purpose with this choice, therefore, is to make interaction in the calibration model more similar to interaction in the environment, under the unavoidable limitations of insufficient data coverage.
5.4 Alternative Calibration Models
We can contrast this KNN calibration model to two natural alternatives: kernel density estimator (KDF) model and neural network model. A KDE model is a non-parametric estimator, that has even been investigated for model-based RL (Pan et al., 2018). Like our KNN calibration model, it should also stably remain within the region defined by the dataset. However, unlike the KNN calibration model, KDE calibration model could produce non-existent states. It effectively interpolates between obscrved datapoints, and so results in siguificant generalization. If we consider again the example with (a, v) position in a gridworld with walls, then the KDE calibration model could produce transitions within the wall.
Another alternative is to use neural network (NN) to learn calibration model. The dataset can be used to learn the expected next state and reward, for given state and action, using regression inputs (s.a) and targets sr, Or, to obtain a distribution over the next states and rewards, conditional distribution can be learned using mixture density networks or stochastic networks. Simulators have been learned in RL on real data, particularly with recurrent NNs (RNNs) to handle partial observability, such as for gas turbine control (Schaefer et al., 2007; Schäfer, 2008), drawing on the larger literature using RNNs for system identification (Barabanov Prokhorov, 2002; Yu, 2004; Mohajerin, 2012).
At the same time, learning transition dynamics with NNs in RL can be challenging, and can cause issues when used as simulators. Such NN models can produce non-existent states, just like the KDE model. With an extremely long rollout, the prediction error accumulates, and the model may generate states that are out of distribution or become unstable. Several works in model-based RL have illustrated that ilerating such models can produce less and less plausible outcomes states (Talvitie, 2017: Jafferjee et al., 2020; Abbas et al., 2020; Chelu et al., 2020). This is particularly problematic in the Data2Online sctting, where the number of steps of iteration is much larger than what typically used in model-based RL. The calibration model must mimic the real deployment environment. Initially the agent relies on random exploration to discover good
13

---

Published in Transactions on Machine Learning Research (07/2022)
rewards before learning can adapt the policy, and thus the first few episodes simulated by the model may be extremely long, even thousands of steps.
Avoiding these issues with iterating NN models is an active area of research. One direction for model-based RL has been to train models to be correct over multiple steps of iteration (Talvitie, 2014; Venkatraman et al., 2015; Talvitie, 2017; Williams et al., 2017; Ke et al.; 2019). Other work has looked at constraining the architecture to ensure stability (Manek & Kolter, 2019; Lawrence et al., 2020; Takeishi & Kawahara, 2021: Drgona et al., 2022). Such advances are likely to accelerate with the growth in sequence modeling and generation.
In this work, we leverage the power of neural networks to improve the distance metric within our KNN model. Arguably, this distance metric does much of the heavy lifting, with mostly simplistic rules layered on top to transition between samples. The combination allows us to leverage the ability of NNs to scale to high-dimensional inputs, and the simplicity and interpretability of KNNs. It provides an casy-to-use alternative to learning NNs RNNs from scratch, which can often require significant expertise. And, by design, it is guaranteed to remain stable and only produce states that have been observed. In some cases end-to-end RNN model may be more effective, nonetheless, this KNN approach with NN metric expands the types of models users can consider for their application.
6 Experiments
We conducted battery of experiments to provide a rounded assessment of when an approach can cannot be expected to reliably select good hyperparameters for online learning. We investigate varying the data collection policy and size of the data logs to mimic a variely of deployment scenarios ranging from a near-optimal operator to random data. We explore selecting hyperparameters of different types for several different agents, and investigate a non-stationary setting where the environment changes from data collection to deployment. We begin with the simplest first question: how does our approach compare to simple basclines and with different. choices of calibration model type.
To extensively test the reliability of our approach, we deploy the algorithm on variants of Acrobot, Puddle World, and Cartpole (Sutton & Barto, 2018). All three environments are episodic and have a low-dimensional continuous state and discrete actions. Small environments allow extensive experimentation; critical for hyperparameter analysis and achieving statistically significant results. In addition, recent studies have shown that conclusions from small classic control environments match those generated in larger scale environments like Atari (Ceron & Castro, 2021). Experiments were conducted on a cluster and a powerful workstation using 8327 CPU hours and no GPUs. Full lists of all the hyperparameters can be found in the appendix.
6.1 Experiment 1: Comparing Calibration Models
In this experiment we investigate the benefits of our approach with different choices of model in two classic control environments. We compare our KNN calibration model with learned Laplace similarity metric to an NN model trained to predict the next state and reward given input state and action observed in the calibration data. In addition, we also test an NN calibration model that takes the Laplacian encoding (see Section 5) of the current state as input and predicts the next state and reward to provide the network with a better transition-aware input representation. We used two continuous state, discrete action, episodic deployment environments, Acrobot and Puddle World, as described in the appendix and in introductory texts (Sutton & Barto, 2018).
In this first experiment select the hyperparameters for linear softmax-policy Expected Sarsa agent (from here on, lincar Sarsa) from data generated by simple policy with good coverage. The agent uses tile coding to map the iontis state variables to binary feature vectors (see Sutton & Barto (Sutton & Barto, 2018) for a detailed discussion of tile coding). This on-policy, Sarsa agent learns quickly but is sensitive to several important hyperparameters. We investigate several dimensions of hyperparameters including the step-size and momentum parameters of the Adam optimizer, the temperature parameter of the policy, and the value function weight. initialivation. We choose these hyperparameters because their impact on performance is
14

---

Published in Transactions on Machine Learning Research (07/202)
Return per episode. Higher (return closer to zero) is better
-100.0
-24.0
473.9
-112.7
Iruc pect.
-11.77
-29.0
perf
pert
Random
-29.0
Calib (KNN)
Calib (KNN)
FQI
-150.0
-386.7
-49.3
-2049.6
Calib (NN)
FQI
-200.0
Random
-753.6
-74.7
-4453.11
Calib (NN)
-250.0
-1120.5
-100.0
-7096.6
Acrobot
Puddle World
Figure 3: Hyperparameter transfer with different calibration models. Each subplot shows the performance of two calibration models compared against two basclinos: FQI and random (described in text). The dotted horizontal line indicales the average performance of the best hyperparameter setting in the sweep in the deployment enviromment. Each box shows the distribution summarizing the true performance in deployment of the best hyperparameters selected in each run of the experiment. We plot the return per episode, thus higher the better. The LHS of each subplot uses the LHS y-axis and the RHS (separated by the dotted vertical orange line) uses the RHS red y-axis. In each subplot the bold line represents the median, the boxes represent the 25th and 75th percentiles, and outliers are represented as circles. Low variance indicates that the system reliably chooses the same hyperparameters cach run. The performance of the random baseline characterizes the maximum possible variation. Recall the performance of each hyper is precomputed off-line for each hyperparamcter combination and is thus not a source of variation in our setup.
somewhat transient and can be overcome by continued learning; this reflects our desire for the agent to continually learn and adapt in deployment.
We used a near-optimal policy for each environment to collect data for building the calibration models. The near-optimal data collection policy for Acrobot can solve the task in 100 steps, and the near-optimal data collection policy in Puddle World achieves an average return of -25. In both cases the policy will provide the system with many examples of successful trajectories the goal states in the 5000 transition data log.
Our evaluation pipeline involves several steps. First we evaluate the true performance (steps per episode for Acrobot and return per episode in Puddle World) of each hyperparameter combination in the deployment environment: running for 15,000 steps in Acrobot and 30,000 steps in Puddle World, averaging over 30 runs. We then use the data collection policy to generate the calibration data log and learn each model. We record the true performance of the selected hyperparamotors to summarize the performance. This whole process running the data collection policy to generale a data log, learning the calibration model, and evaluating the hyperparameters- is repeated 30 times (giving 30 datasets with 30 corresponding hyperparameter selections). The statistic of interest is the median and distribution of the true performance for the syper-parameterd selected across runs. In the ideal case, if there is one truly best set of hyperparameters, the system will choose those every time and the variance in true performance would be zero.
We also included two baselines. The first is randomly selecting hyperparameters, called Random, to get a sense of the spread in performance; all methods should outperform Random. We also include an Offline RL algorithm, called Fitted-Q Iteration (FQT) (Frnst et al., 2005), that learns a policy from the calibration data and then deploys the learned policy fixed in the deployment environment. For the FQI bascline we simply plot the distribution of performance of each of the 30 extracted policies on the deployment environment. For each policy, we average the performance over 30 random seeds. We tested FQI with tile coded representation and a NN representation; the tile coded version performed best and we report that result.
Figure 3 summarizes the results. In both environments the KNN calibration model performed well, selecting the same hyperparameters as would a sweep directly in the deployment environment. The NN calibration models perform poorly overall. The NN calibration model using raw inputs (no Laplacian encoding) was not as effective, and so we only include results for the NN with the Laplacian encoding in Figure and relegate the other to Appendix D.I. Their performance can be unstable, choosing hyperparameters with
15

---

Published in Transactions on Machine Learning Research (07/2022)
Return per episode (Median). Higher (closer to zero) the better.
500 1000 5000
-*
Near-optimal
Naive
500 1000 5000
The 261
Near-optimal
Naive
Medium
Medium
-4-
41248
-104
4224
-30
1209
-154
(a) Acrobot: log-size
(b) PW: log-size
(c) Acrobot: policy
(d) PW: policy
Figure 4: The role of data logs. Plots (a) and (b) show the median deployment performance of hy- perparmaters selected from calibration models constructed from different sized data logs (with 25% and 75% quartiles). Plots (c) and (d) show the median deployment performance under different policies used to collect data logs for constructing the calibration model. In both subplots, higher is better. In these bar plots the median is shown by the top of the colored bar, and the quartiles are shown by the black whiskers. Overall, the utility of the calibration model appears largely insensitive to the data log size and policy in these environemnts; hyperparameters that perform well in deployment can be selected.
good performance in some runs, but often choosing poor hyperparameters. FQI generally performs worse than even Random. Note that spent quite a bit of time improving FQI, well as optimizing over several stepsize and regularization hyperparameters. This suggests the calibration data log is too limited to extract a good policy and deploy it without additional learning, but the data appears useful for selecting hyperparamnterrs with the KNN calibration model.
We also used our approach to tune both step-size parameters of a linear Actor-critic agent with tile coding, The KNN calibration model was able to select top performing hyperparameters for Actor-critic in both Acrobot and Puddle World though the agent performed worse than linear Sarsa (results in Appendix D.2).
6.2 Experiment 2: Varying Data Collection Policies
The objective of this experiment was to evaluate the robustness of our approach to changing both the amount of offline data available and the quality of the policy used to collect the data. We experimented with three different policies corresponding to near-optimal, medium, and naive performance to collect 5000 transitions for training our KNN Laplacian calibration model. The near-optimal policy was identical to the one used in the previous experiment. The medium policy was designed to achieve roughly half the visits to goal states after 5000 steps (approximately 90 for Puddle World & 25 for Acrobot) compared to the near optimal policy. The naive policy was designed such that it achieved significantly fewer visits (approximately 35 for Puddle World & 12 for Acrobot). We also tried different data log sizes of 500, 1000, and 5000 samples using the medium policy, all shown in Figure 4.
The results in Figure 4 show that our approach is largely insensitive to data log size and policy in these classic environments. Even 500 transitions contains enough coverage of the state-space and successful terminalions to produce a useful calibration model. This is in stark contrast to the FQI results in Experiment 1, where policy trained offline from the same size data log failed to solve either task. Exploration in both these environments is not challenging; therefore, the success of the calibration model not surprising. This positive outcome, however, reflects that it may be simpler to pick hyperparameters in some environeents.. In Experiment 4, we investigate failure case in Cartpole.
6.3 Experiment 3: When the Environment Changes
Learning online is critical when we expect the environment to change. This can happen duc to wear and tear on physical hardware, un-modelled seasonal changes, or the environment may appear non-stationary to the agent because the agent's 's state representation does not model all aspects of the true state of the MDP. In this latter case it is often best to track rather than converge; to never stop learning (see (Sutton et al., 2007)). In our problem setting, the deployment environment could change significantly between (a) calibration data collection and (b) the deployment phase. Intuitively we would expect batch approaches that simply deploy
16

---

Published in Transactions on Machine Learning Research (07/2022)
fixed policy learned from data to do poorly in such settings. The following experiment simulates such scenario with the Acrobot environment.
The experiment involves two variants of the environment. As before, we collected 5000 transitions using the near-optimal policy in Acrobot and then applied our approach to select good hyperparameters for the linear Sarsa agent. Unlike before, we evaluate the hyperparameters selected on a second, changed Acrobot environment. In the changed Acrobot environment we doubled the length and mass of the first link length. Our two phase setup changes the dynamics of Acrobot but does not prevent learning reasonably good policies you will see in the results below. This whole process repeated 30 times (generating 30 datasets with corresponding 30 calibration models) to aggregate the results presented in Figure 5.
We included three baselines to help contextualize the results: (1) transferring the policy from the first environment, (2) transferring the policy learned the calibration model, and (3) FQL. The first baseline, called Sarsa (True), simply transfers the policy Icarned in the first Acrobot environment to the changed Acrobot environment (no calibration model was used, hence the label Irue). The second baseline, called Sarsa (Calibration) simply uses the best performing policy learned by Sarsa in our calibration model, where the calibration model is created with data from the first Acrobot environment. Finally, we also included a FQI baseline. We trained policy using FQI with tile coding on the data generated from the first environment (the same data used to build the calibration model). Then we evaluated the policy learned by FQI on the changed Acrobot environment. These baselines are meant to illustrate how performance might be effected if the environment dynamics changed but a prelearned policy was applied without taking the changes into account, perhaps because no one noticed the environment had changed. In all three baselines the policy evaluated in the second environment is fixed (no learning in deployment).
Return per episode (logscale) Higher (return closer better
True pert.
.797.6
Figure 5: Selecting hyperparameters in the face of non- stationarity. The plot above summarizes the performance of our approach compared with three fixed-poliey transfer approaches (described in text). Selecting hyperparameters for deployment works well even when the environment changes between calibra- tion data collection and deployment. Deploying fixed policies, on the other hand, performs poorly by comparison.
Calib (KNN)
-1000.0
FQI
Sarsa (True)
Sarsa (Calibration)
-10000.0
The results in Figure 5 highlight that transferring fixed policies can be problematic when the environment changes. Our calibration- approach performs best and appears robust under the abrupt non-stationaril tested here. Clearly, the difference between the two environments is significant enough such that transferring a policy directly learned on the first environment (the Sarsa-True baseline) performs worse than using our approach to select hyperparameters in the calibration model and then learning a new policy from scratch. Interestingly, learning and transferring a policy from the calibration model was worse than using than training on the first environment or training from the calibration data (as in FQI). It is not surprising that transferring hyperparameters and learning in meptoyment, is more robust. than transferring fixed policies, in these non-stationary settings.
6.4 Experiment 4: A Failure Case
Our approach is not robust to all environments and data collection schemes. In this section we investigate when it can fail. One obvious way our approach can fail is if the agent's performance in the calibration model is always the same: no matler what hyperparameter we try, the system thinks they all perform similarly. To illustrate this phenomenon we use the Cartpole environment. In Cartpole, the agent must balance a pole in an unstable equilibrium as long as it can. If the cart reaches the end of the track or the pole reaches a critical angle, failure is inevitable. Near-optimal policies can balance the pole for hundreds of steps rarely experiencing failures and, thus, visit only small fraction of the state-action space. A data log collected
17

---

Published in Transactions on Machine Learning Research (07/2022)
Return per run. Higher (closer to zero) is better
Random
True Perf.
645.2
-1524.0
Random
2000-
Near Optimal
-346.00
-4000
Near
Perf in & deployment
Medium
Optimal
Medium
5448.0
600D
Perf deployment
Perf according to model
Figure 6: Success and failure in Cartpolc. This plot shows performance of three different calibration models constructed from random, near-optimal and medium policies. Left: the performance of the hy- perparameters deployment as picked by different calibration models. Right: each model's evaluation of all hyperparanters across all 30 runs. Ideally the distribution of performance would match that of the hyperparameter performance in the deployment environment blrrc dots far right.
from the near-optimal policy would likely produce a calibration model where failures are impossible and all hyperparameters appear excellent.
We test this hypothosis by looking at three data collection policics. We used a random policy, near- optimal policy with random initial pole angles, medium policy that was half as good as the near-optimal policy (twice as many failures). We expect the random policy to provide useful data for the calibration model, whereas the near-optimal policy will cause failure due to the above reason. The interim policy helps understand sensitivity to this issue: we might expect the issue to be resolved with less optimal policy.
Figure 6 indeed shows that the dynamics of Cartpole combined with particular data collection policies can render the calibration model ineffective We see in the left-hand plot that the hyperparameter chosen with the calibration model using random data performs somewhat reasonably, but fails for both the medium and near-optimal policies. Even with random starting states the calibration model for near-optimal policy: the calibration model never simulated dropping the pole. The random policy produced the best calibration model. Unsurprisingly, the random policy drops the pole every few sleps and thus the log contained many failures and higher state coverage. Nevertheless, the performance was still poor because there were no examples of balancing the pole for many consccutive steps: the model constructed from random data was still a poor model of the true deployment enviroument.
We can see this further, by looking at the performance estimates under the three calibration models. The right-hand plot in Figure shows the performance of all the hyperparameters according to the calibration model (before deployment). The blue dots show that most hyperparameters appear good in calibration when the model is constructed with data from a near-optimal policy. At the other extreme the grey dots show a large spread in performance of hyperparameters when the model is constructed with data from a the random policy. Note that none of the grey dots appear as low on the y-axis compared with the blue and orange dots corresponding to the other two policies. This indicates that calibration with the medium and near-optimal policy models incorrectly inflate the performance of many hyperparameter combinations, whereas calibration with the random-policy model potentially undervalues some perpaarraeeeer combinations.
One could certainly argue that many applications might not exhibit this behavior-especiialyy especially since it is largely caused by a task with two modes of operation (failing or balancing). Extracting a policy initial- ization from the calibration data (perhaps via behavior cloning) and then using this initial policy in both hyperparameter selection and deployment could avoid these problems in Cartpole, but we leave these ex- periments to future work. Regardless, this experiment provides an important reminder that we will not anticipate all situations in the deployment of RL in the real world; there is no general black-box strategy for deployment and failures will happen.
18

---

Published in Transactions on Machine Learning Research (07/2022)
7 Moving Beyond Grid Search
The calibration model is an offiine artifact that we can use as we like without impacting the deployment enviromment. We can use the model in smarter ways discover and evaluate good hyperparameteess for deployment. In fact, we can leverage the large literature on algorithm configuration, which explicitly deals with efficient methods to search over hyperparameters. In this section, we explain how to incorporate these approaches and test two strategies, as a replacement for grid search.
7.1 Improving the Hyperparameter Search
A variety of hyperparameter search approaches were introduced under sequential model-based optimization (SMBO) (Hutter et al., 2011), ) but methods built on Bayesian optimization (BO) (Snoek et al., 2012) have become more popular. Complementary to these approaches are those that direct computation to promising hyperparameterss and stop performance evaluations of poor hyperparameters early, as in Hyperband (Li et al., 2018), or that design the algorithm to do both (Klein et al., 2017; Falkner et al., 2018). All these approaches attempt to find the maximum of the performance function, assuming that function expensive to query.
BO algorithms approximate the performance function f(A), and use this approximation to decide what hyperparameter setting À to test next. The general approach is to (1) maintain posterior distribution over the performance function f, (2) find a candidate set of optimal hyperparameters de according to a criteria like expected improvement under the current posterior over J, (3) evaluate Ae. obtaining = ((Ae) and (4) update the posterior with sample (Ae.y). Once the algorithm terminates typically by reaching a time limit-the best de out of all the candidates tested is returned, according to the maximal y. The primary purpose of learning the posterior ʃ is to direct which da should be lested, though some algorithms do solve optimization the very end of this procedure to find À with the maximal posterior mean (see (Frazier, 2018) for nice overview).
Due to the importance of hyperparameter optimization for machine learning in a growing field called Auto Mh-the development of BO methods has been focused on large numbers of hyperparameters, for training large models on large datasets. For this highly expensive setting, it is worth carefully crafting advanced algorithms that minimize the need to train and evaluate large models. These complex methods can then be released within packages, to facilitate their use, as they may be difficult to implement from scratch.
BO in our experiments: We use an open-source package (Nogueira, 2014 ), which uses gaussian processes for optimizing the hyperparameter setting. We chose to use upper confidence bounds, with a confidence level of 2.576 the default in the package as the acquisition method. The queue is initialized with 5 random samples and the algorithm is run for 200 itcrations.
For our setting, each evaluation not as complex and we need not use such advanced approaches. Instead, our primary goal is simply to answer: if we allow hyperparameters to be optimized over a continuous set, can improve on basic grid search? For this question, we also test two simple approaches: random search and the cross-entropy method (CEM). Random search involves simply testing a fixed number of hyperparameter settings, and selecting the one with maximal performance. Though simple, it is a common baseline because it has been shown to be competitive with more complex search strategies (Bergstra & Bengio, 2012).
CEM (Rubinstein, 1999) is an approach to global optimization of functions, like BO, but is based on a simpler strategy of (1) maintaining a distribution over the inputs (hyperparamoters) (2) incrcasing the likelihood of the top percentile of sampled values under this distribution, according to the performance function. The distribution is simple to sample, and the percentile easy-to-compute, making this approach simpler to use than BO.
CEM has not been used for hyperparameter optimization, to the best of our knowledge. Likely the reason is that BO strategies provide a more powerful way to carefully reason about what candidate points to sample. CEM instcad slowly narrows a distribution over hyperparameters, and does not reason about confidence intervals nor about a criterion (acquisition function) to identify ideal candidates. Nonetheless, we include it simpler strategy, investigate performance of using calibration models with a hyperparameter search
19

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Return per episode. Higher (return closer to zero) is better
-10.22
-24.3
perf.
-112.7
29.0
oerf
CEM
Bayesian Optimization
-129.0
Bayesian Optimization
-35.0
Grid Search
Random Search
Random Search
Grid
-153.9
Search
-178.7
-56.6
Acrobot
Puddle World
Finding good hyperparamcters via black-box optimization in calibration. Here we compare four different strategies for optimizing hyperparametets: (1) grid search (which we have used in all previous experiments), (2) random search, (3) our CEM procedure, and (4) the Bayesian optimization (BO) approach. The y-axis is same as the one in Figure 3. Generally, all four approaches perform well highlighting the robustness of using the calibration model and transferring hyperparamoters to deployment. Random search is better or comparable to grid search, but this is not a surprising as random search is typically found to be strong bascline in hyperparamcter optimization. Note the True performance the plots above represent the deployment performance of the best hyperparameter combination from discrete set; the same set used by grid search. CEM, BO and random search can obtain higher deployment performance because they search continuous range and thus find better hyperparameter settings. This is one of the major benefits of using better optimization methods in calibration than grid search.
Figure 7:
in-between naive random search and the more advanced BO search. We emphasize that it is not critical which hyperparamcter optimization approach is used within our framework; any method can be swapped in.
CEM in our experiments: Our setting has two nuances compared to the typical setting where CEM is used: our function is expensive to query and we only get a stochastic sample. We provide modified CEM algorithm, that still refiects the general framework for CEM, but using incremental update-similar to stochastic gradient descent- to account for the stochasticity in our function query. The algorithm is summarized in Algorithm 13 in Appendix C.1.
7.2 Experiment 5: Hyperparameter Tuning with Alternative Optimization Approaches
In this section we compare grid search, random search, Bayesian optimization, and our simple CEM approach for hyperparameter selection. Are these approaches interchangeable in setup? Does searching contin- wous hyperparameter space result in performance gains and at what cost? The goal of the experiment is to highlight that alternative hyperparameter optimization approaches beyond a basic grid search are possible and to investigate if they are beneficial.
In this experiment, we use the same settings as above, but now optimize the temperature and stepsize a as continuous values in the ranges [0.0001, 5.0] and (0.0, 0.1] respectively for Acrobot, and [0.0001, 10.0) and [0.0, 1.0] respectively for Puddle World. The random search approach simply generates k possible hyperparameters from the continuous ranges above and evaluates each in parallel in the calibration model. The best performing hyperparameters according to the calibration phase are used in deployment. Both random search and CEM use 100 iterations, to make computation used comparable to grid search, while Bayesian optimization uses 200 iterations.
Random search, Bayesian optimization and CEM outperform grid scarch, as we can see in Figure 7. The performance improvements are especially stark in Puddle World. Even when tuning only on the calibration model, the agent can outperform the best hyperparameters found by grid search the true environ-
20

---

Published in Transactions on Machine Learning Research (07/2022)
ment. This is why the return for CEM is higher than the dotted line showing the performance of the best hyperparameters within the set used for the grid search. These results are promising, in that they show more carefully optimizing hyperparameters on the calibration model helps rather than hurts. A possible hypothesis, apriori, could have been that optimizing more carefully to the calibration model could cause odd or very poor hyperparameterss to be chosen and that the restricted set in the grid search actually helped avoid this failure. These results suggest otherwise, and in fact highlight that our previous results could have been produced even more consistent performance with smarter hyperparameter algorithm.
This experiments in this paper highlight the generality and fiexibility of our approach. The calibration model can take different forms. Data collection can be done in number of different ways. Hyperparameters can be systematically searched or optimized. In the end, numerous other specializations and algorithmic innovations could be included to further optimize performance in real deployment scenarios.
8 Conclusion
In this work, we introduced the Data2Online problem: selecting hyperparameters from log of data, for deployment in real environment. The basic idea is to learn a calibration model from the data log, and then allow the agent to interact in the calibration model to identify good hyperparameters. Essentially, the calibration model is treated just like the real environment. We provide a simple approach, using k-nearest neighbors, to obtain a calibration model that is stable under many iterations and only produces real states. We then conduct a battery of tests, under different data regimes.
Naturally, as the first work explicitly tackling this problem, we have only scratched the surface of options. There is much more to understand about when this strategy will be effective, and when it might fail. As we highlight throughout, this problem should be more feasible than offline RL, which requires the entire policy to be ifidd from log rather than just suitable hyperparameters for learning. Our own experiments highlight that offline methods that attempted to learn and deploy fixed policy performed poorly, whereas identifying reasonable hyperparamoters was a much casier problem with consistently good performance across many policies and even small datasets. Nonetheless, we did identify one failure case, where the data resulted in model that made the environment appear too easy and so most hyperparameters looked similar. Much more work can be done, theoretically and empirically, to understand the Data2Online problem.
References
Zaheer Abbas, Samuel Sokota, Erin Talvitie, and Martha White. Selective dyna-style planning under limited model capacity. In International Conference on Machine Learning, 2020.
Amurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofr Nachum. Opal: Offline primitive
discovery for accelerating offline reinforcement learning. In International Conference on Learning Repre- sentations, 2021.
N.E. Barabanov and D.V. Prokhorov. Stability analysis of discrete time recurrent neural networks. 1EEE Transactions on Neural Networks, 13(2), 2002.
Paul Barde, Julien Roy, Wonseok Jeon, Joelle Pineau, Derek Nowrouzezahrai, and Christopher Pal. Ad- versarial soft advantage fitting: Imitation learning without policy optimization. In Advances in Neural Information Processing Systems, 2020.
Feryal Behbahani, Kyriacos Shiarlis, Xi Chen, Vitaly Kurin, Sudhanshu Kasewa, Ciprian Stirbu, Joao Gomes, Supratik Paul, Frans A Olichock, Joao Messias, et al. Learning from demonstration in the wild. In International Conference on Robolics and Aulomation, 2019.
Marc G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Subhodecp Moitra, Sameera S. Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning- Nature, 588(7836), 2020.
Jon Louis Bentley. Multidimensional binary search trees used for associative searching. Communications of the ACM, 18(9):509-51) 1975.
21

---

Published in Transactions on Machine Learning Research (07/2022)
James Bergstra and Yoshua Bengio. Random search for prper-parameter optimization. Journal of Machine Learning Research, 13:281-300 -305, 2012.
Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting Rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning, 2021.
Yash Chandak, Scott M Jordan, Georgios Theocharous, Martha White, and Philip Thomas. Towards safe policy improvement for non-stationary mdps. In Advances in Neurol Information Processing Systems, 2020a.
Yash Chandak, Georgios Theocharous, Shiv Shanka, Martha White, Sridhar Mahadevan, and Philip S Thomas. Optimizing for the future in non-stationary mdps. In International Conference on Machine Learning, 2020b.
Veronica Chelu, Doina Precup, and Hado P van Hasselt. Forethought and hindsight in credit assignment. In Advances in Neural Information Processing Systems, 2020.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackie Kay, Antoine Merle, Jean-Mare Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis, and Martin Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897), 2022.
C. Downey and Sanner. Temporal difference Bayesian model averaging: Bayesian perspective on adapting Lambda. In International Conference on Machine Learning, 2010.
Jan Drgona, Aaron Tuor, and Draguna Vrabie. Learning Constrained Adaptive Differentiable Predictive Control Policies With Guarantees. arXie preprint orx::22004.1118, 2022.
Damien Ernst, Pierre Geurls, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556 2005.
Stefan Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. International Conference on Machine Learning, 2018
Chelsca Finn, Tianhe Yu, Tianhao Zhang, Picter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Conference on Robot Learning, 2017.
Peter I Frazier. A tutorial bayesian optimization. arXiv preprint arXio:1807.02841 2018.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, 2019.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pincau, Doina Precup, and David Meger. Decp reinforcement learning that matiers. In the AAAI Conference on Artificial Intelligence, 2018.
Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. DARLA: Improving zero-shot transfer in reinforce- ment learning. In International Conference on Machine Learning, 2017.
Sebastian Höfer, Kostas Bekris, Ankur Handa, Juan Camilo Gamboa, Mclissa Mozifian, Florian Golemo, Chris Atkeson, Dieter Fox, Ken Goldberg, John Leonard, C. Karen Liu, Jan Peters, Shuran Song, Peter Welinder, and Martha. White. Sim2Rcal in Robotics and Automation: Applications and Challenges. IEEE Transactions on Audomation Science and Engineering, 18(2), 2021.
Frank Hutter, Holger Hoos, and Kevin Leyton-Browe Brown. Sequential model-based optimization for general algorithm configuration. In International Conference Learning and Intelligent Optimization, 2011.
22

---

Published in Transactions on Machine Learning Research (07/2022)
Andrew Jacobsen, Matthew Schlegel, Cam Linke, Thomas Degris, Adam White, and Martha White. Meta- descent for online, continual prediction. In the AAAT Conference Artificial Intelligence, 2019.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jefs Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukeuoglı Population based training of neural networks. arXiv preprint arx::: 1rXii:1711. 1711.09944. 2017.
Taher Jafferjee, Ehsan Imani, Erin Talvitie, Martha White, and Micheal Bowling. Hallucinating value: A pitfall of dyna-style planning with imperfect environment modcls. arXiv preprint Xrx:2:0006. 04363, 2020.
Nan Rosemary Ke, Amanprect Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, Devi Parikh, and Dhruv Batra. Learning dynamics model in reinforcement learning by incorporating the long term future. In International Conference on Learning Representations, 2019.
Michael Kearns and Satinder Singh. Near-optimal reinforcement learning polynomial time. Machine
Learning, (9(2):209-253) 202, 2002.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:14 12.. 6980, 2014.
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hemnig, and Frank Hutter. Fast Bayesian optimization of machine learning hyperparameters on large datasets. In International Conference on Artificial Intelligence and Statistics, 2017.
Tor Lattimore and Csaba Szepesvári. Bandit algorithmms. Cambridge University Press, 2020.
Nathan Lawrence, Philip Loewen, Michael Forbes, Johan Backstrom, and Bhushan Gopaluni. Almost Surely Stable Deep Dynamics. In Adeances in Neural Information Processing Systems, volume 33, 2020.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic Q-ensemble. In Conference Robot Learning, 2021.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18 (1):6765-6816, 2018.
Rupamn Mahmood, Dmytro Koreukevych, Gautham Vasan, W. Ma, and J. Bergstra. Benchmarking reinforcement learning algorithms on real-world robots. In Conference on Robot Learning, 2018
Travis Mandel, Yun-En Liu, Emma Brunskill, and Zoran Popović. Offline evaluation of online reinforcement learning algorithms. In the AAAI Conference on Artificial Intelligence, 2016.
Gaurav Manek and J Zico Kolter. Learning stable deep dynamics models. In Advances in Neural Information Processing Systems, 2019.
Timothy A. Mann, Hugo Penedones, Shie Mannor, and T. Hester. Adaptive Lambda least-squares temporal difference learning, arXiv preprint 1xXi::6612.0466. 2016.
Amir massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesvári, and Shie Mannor. Regularized fitted q-iteration for planning in contimnous-ppace markovian decision problems. In American Control Conference, 2009.
Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXie preprint orXiv:1707.022 2017.
N. Mohajerin. Identification and Predictive Control Using RecurrentNeural Networks. PhD thesis, Orebro University, 2012.
Fernando Nogueira. Bayesian Optimization: Open source constrained global optimization tool for Python, 2014-. URL https://tttte http:://gggaaaacoooo////a//aaaas
23

---

Published in Transactions on Machine Learning Research (07 (07/2022)
Yangchen Pan, Muhammad Hamad Zaheer, Adam White, Andrew Patterson, and Martha White. Organizing experience: A deeper look at replay mechanisms for sample-based planning in continuous state domains. In International Joint Conference on Artificial Intelligence, 2018.
Matteo Papini, Matteo Pirotta, and Marcello Restelli. Smoothing policies and safe policy gradients. arXiv preprint arXi::1905.00 03231, May 2019.
Jack Parkor-Holder, Vu Nguyen, and Stephen J. Roberts. Provably efficient online hyperparameter opti- mization with population- based bandits. In Conference and Workshop on Neural Information Processing Systems, 2020.
Supratik Paul, Vitaly Kurin, and Shimon Whiteson. Fast efficient hyperparameter tuning for policy gradient methods. In Advances in Neural Information Processing Systems, 2019.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In International Conference on Robotics and Automation, 2018.
Harish Ravichandar, Athanasios $ Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Avaeual Review of Control, Robolics, and Aulonomous Systems, 3:297 330, 2020.
Martin Riedmiller. Neural Fitted Q Iteration - First experiences with data efficient neural reinforcement learning method. In European Conference on Machine Learning, 2005.
Reuven Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability, 1(2):127-190, 1999.
Anton Maximilian Schaefer, Daniel Schneegass, Volkmar Sterzing, and Steffen Udluft. A Neural Reinforce- ment Learning Approach to Gas Turbine Control. In 2007 International Joint Conference on Neural Networks, 2007.
Anton Maximilian Schäfer. Reinforcement Learning with Recurrent Neural Networks. PhD thesis, Universilat
Osnabruck, 2008.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Newral Information Processing Systems, 2012.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: no regret and experimental design. In International Conference on Machine Learning, 2010.
Richard S Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-dee bar-dela.. In the AAAI Conference on Artificial Intelligence, 1992.
Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances in Neural Information Processing Systems, 1995.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, Anna Koop, and David Silver. On the role of tracking in stationary environments. Tn International Conference on Machine learning, 2007.
Naoya Takeishi and Yoshinobu Kawahara. Learning Dynamics Models with Stable Invariant Sets. In AAAI Conference on Artificial Intelligence, 2021.
Erik Talvitie. Self-correcting models for model-based reinforcement lcarning. In the AAAI Conference on Artificial Intelligence, 2017.
Erin Talvitie. Model regularization for stable sample rollouts. In International Conference on Uncertainty in Arlificial Intelligence, 2014.
24

---

Published in Transactions on Machine Learning Research (07/2022)
Yunhao Tang and Krzysztof Choromanski. Online hyper-parameter tuning in off-policy learning via evolu- tionary strategies. arXiv preprinl ar Xr::22000.0070 07554, 2020.
Arun Venkatraman, Martial Hebert, and Andrew Bagnell. Improving multi-step prediction of learned time series models. In the AAAT Conference Artificial Intelligence, 2015.
Ruosong Wang, Dean P. Foster, and Sham M. Kakade. What are the statistical limits of offline RL with linear function approximation? In International Conference on Learning Representations, 2021.
Martha White. Unifying task specification in reinforcement learning. In International Conference on Machine Learning, 2017.
Martha White and Adamn White. A greedy approach adapting the trace parameter for temporal difference learning. In International Conference Autonomous Agents and Multiagent Systems, 2016.
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evange- los A Theodorou. Information theoretic MPC for model-based reinforcement learning. In International Conference on Robotics and Automation, 2017.
Yifan Wu, George Tucker, and Ofrr Nachum. The laplacian in rl: Learning representations with efficient approximations. International Conference on Leurning Representations, 2019.
Jinwei Xing, Takashi Nagata, Kexin Chen, Xinyun Zou, Emre Neftci, and Jeffrey L. Krichmar. Domain adaptation in reinforcement learning via latent unified state represontation. In the AAAI Conference on Artificial Intelligence, 2021.
Zhongwen Xu, Hado Hasselt, and David Silver. Meta-gradient reinforcement learning. In Advances in Newral Information Processing Systems, 2018.
Mengjiao Yang and Ofir Nachum. Representation mallers: Oflline pretraining for sequential decision making. In International Conference on Machine Learning, 2021.
Mengjiao Yang, Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline policy selection under uncertainty. arXiv preprint Xrx:2201 069199 2020.
Wen Yu. Nonlinear system identification using discrete-time recurrent neural networks with stable learning algorithms. Information Sciences, 158, 2004.
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh. A self-tuning actor-critic algorithm. In Advances in Newral Information Processing Systems, 2020.
25


Reviews:
[{"review_id": "rzlcWrxVqO", "reviewer": "Reviewer_7MV8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- the presented method is simple and elegant\n- overall the paper is very well executed\n- the paper is well-motivated (i.e. addresses an important problem)\n- experiments cover various scenarios\n\nWeaknesses:\n\n- it is not clear if the suggested solution solves the stated problem\n- it is unclear how the solution would scale (to more difficult problems)\n\nIn more details, the paper proposed *Data2Online*\n method, which, in a nutshell, builds a calibration model used for tuning hyperparameters. The standing assumption on which the method hinges is that the quality of such a model might be relatively weak. I enjoyed reading the paper; the text's quality is high, the structure is well thought out, and, importantly, the experiments cover various situations (e.g., failure case).\n\nNevertheless, I have doubts about the practical significance. I assume that the ultimate goal is sample efficiency understood in a broad sense, meaning that one takes into account all samples used, including hyperparameter search.\n\nThe cost of collecting the offline dataset also needs to be considered (or argued to be irrelevant). Let us consider two extremal cases: near-optimal data, and random exploration data. In the first one, the problem has been solved before (and the prize had been paid). Random exploration data are cheap but have typically poor coverage, therefore making it possible only to find hyperparameters for the initial stages of training. However, perhaps, tuning these initial stages would be cheap to make standard online tuning?\n\nI would imagine that it is (might be) possible to benefit from the proposed method; however, at the moment, I feel the evidence is poor. The environments are somewhat simple (even a few steps suffice to provide good coverage).\n\nA few more details (most of the comments regarding text are nit-picks):\n\n- I find Section 1 and Section 2 slightly too long; for example I’d consider removing some parts of formalization\n- I find Section 4.2 a little bit detached from the rest (perhaps some forward references would help?)\n- I find Section 4.3 mostly ‘decorative’. Theoretical analysis is able to provide only very pessimistic estimates, which in my view, do not help much in understanding the method. I’d suggest moving it to the appendix\n- I like a lot the idea of KNN models.\n- I like the solid execution of the experimental section. This softens a little bit my criticism of the choice of environments, i.e. having relatively small and fast ones made it possible to provide a careful statistical evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Data2Online, a method of selecting hyperparameters based on a log of offline data. The method is tested empirically on three environments: Acrobot, Puddle world and  Cartpole, providing some evidence of its usefulness. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I wonder if the authors can provide stronger arguments supporting that their method is likely to be practically useful.\n\nI could imagine various ways. The first one, perhaps the most standard, would be to run more experiments, including more complex environments. There might also be some cheaper ways. Experiments in Section 6.3 suggest that there might be interesting multi-task/CL transfer. Perhaps, one could argue that the calibration data can be reused multiple times (effectively decreasing the cost of their collection). Last but not least, it is possible to pinpoint sample efficiency benefits even with the current data.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1uQYkCbUA2", "reviewer": "Reviewer_3uTX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the idea of the method is intuitive and easy to follow. It is very natural to consider a model-based offline hyperparameter selection rule. The method is well-motivated and the empirical evidence shows that the proposed method is sensible and can outperform random hyperparameters, offline RL with offline hyperparameter tuning based on Q-values and offline RL with NN-parameterized dynamics models.\n\nHowever, I do have several concerns, which are listed as follows. First, I think it is a bit unclear why the authors consider the offline-to-online setting as the whole experiment section. The method seems to focus on hyperparameter selection for offline RL without using any online interactions and I don't see why this method is particularly fitted to the offline-to-online scenario. It may be able to apply to the setting where we train RL policies without different hyperparameters either online or offline and then perform offline hyperparameter selections, but I'm not sure if the case where we train the policy online and then select hyperparameters offline is necessary.\n\nMoreover, I think the experiments need to include more comparisons such as [1] and challenging domains such as DM control, DM locomotion, Manipulation playground and tasks from the D4RL benchmark, which will be pivotal to see the effectiveness of the approach.\n\nFinally, I believe that it is important to test the effectiveness of the method on different (offline) RL algorithms. It would be particularly interesting to test how the calibration models help pick hyperparameters for standard offline model-free RL algorithms (e.g. CQL, IQL, etc.) and model-based methods (e.g. MOReL, MOPO, COMBO, etc.) \n\n[1] Paine, Tom Le, et al. \"Hyperparameter selection for offline reinforcement learning.\" arXiv preprint arXiv:2007.09055 (2020).\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an offline hyperparameter selection scheme based on a non-parametric calibration model for offline RL with online adaptation. The authors present a calibration model trained with KNN, simulate learned policies under the learned calibration model and then pick the hyperparameter that yields the best simulated performance. The authors compare the method to a calibration model trained with neural networks, prior offline RL method with offline tuning scheme (FQI) and offline RL with random hyperparameter selections and show that the proposed method achieves better performance in two control tasks with varying data quality.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See the above section.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FanidEvZ8R", "reviewer": "Reviewer_MhrF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\\\nThe topic is important.\\\nThere is a lack of published studies on experience with this technique.\\\nThe use of KNN models improved by metric learning using Laplace representation as calibration models is, as far as I know, new.\\\nThe term calibration model, which as far as I know is introduced in this paper is very fitting.\n\n**Weaknesses**\\\nThe biggest problem in my opinion is that the paper presents the discussed technique as generally new although it is state of the art. \nThe selection of appropriate reinforcement learning algorithms (and this includes the hyperparameters of the algorithms) based on a model learned on real data is known from patent specification US8099181 [1], for example.\nFurthermore, I know this approach as common practice in the application of RL to real applications. \n\n[1] https://patentimages.storage.googleapis.com/c5/5d/d6/b03f78fa17ce13/US8099181.pdf\n\nSeveral statements are made for which no sufficient evidence is given in their generality:\n\n•\tThe statement „We propose a new approach to tune hyperparameters from offline logs of data, to fully specify the hyperparameters for an RL agent that learns online in the real world“ is correct, if one thinks of the specific design of the KNN with Laplace representation. However, since the statement continues with the sentence „The approach is conceptually simple: we first learn a model of the environment from the offline data, which we call a calibration model, and then simulate learning in the calibration model to identify promising hyperparameters“, it is claimed that the whole approach of selecting hyperparameters for reinforcement learning based on a model of the environment, where this model is created using pre-recorded data of the environment, is novel, which is not the case.\n\n•\tThis is repeated with „We propose a novel strategy to use offline data for selecting hyperparameters“. Again this sentence is correct, if one thinks of the specific design of the KNN with Laplace representation. By the subsequent sentence  „The idea is simple: we use the data to learn a calibration model, and evaluate hyperparameters in the calibration model“ it is again claimed that the approach is new in general.\n\n•\tLikewise „Though this is a simple and natural idea, to the best of our knowledge, it is the first general approach for Data2Online“. This approach already existed before. New is the special design of the calibration model as KNN with Laplace representation. Then, in the subsequent sentence, it is also admitted that the use of models is known: „It is common in reinforcement learning to learn models for offline policy evaluation or for planning“. The justification why the own approach is new „These approaches, however, do not need to tackle a key problem we consider in this work: iterating the model for thousands of learning steps.“ is not convincing. 1. why should thousands of time steps be necessary in general? After all, the data on which the calibration model was learned is available, so one can always start from a randomly selected real state from that data set during learning and then iterate the model, say, only 100 time steps. 2. it is not shown that the proposed model is better suited for thousands of iterations than established methods of system identification, like RNN, e.g. [2], [3], and references therein. My conviction is that RNN are much better able to approximate the environment than KNN with Laplace representation. The performed comparison with NN is not sufficient. First, the NN used are not state of the art, in particular not RNN, and second, the environments studied are extremely low dimensional, with very simple dynamics. In my experience, a KNN based approach will be structurally inferior to an RNN, especially in high dimensionality.\n\n•\tThe statement „There is only one other work considering how to use offline data to evaluate an online agent (Mandel et al., 2016)“ is not true in this form. See [4], [5], [6]\n\n•\tAlso irritating is the sentence „We are hopeful that, with more research, NN models will become a viable choice for learning calibration models“. NN are an established technique of system identification [2], [3] and references therein, which has also been used in high dimensional environments. In the present work, no sufficient evidence was brought that the proposed KNN based models have advantages over state of the art RNN.\n\nOne more note to avoid misunderstandings: NN and RNN are not generally limited to identifying deterministic systems: ensembles and NN with stochastic components (e.g., Bayesian Neural Networks [7]) have been used successfully in stochastic environments.\n\n[2] J. Sjöberg, H. Hjalmarsson, L. Ljung, Neural Networks in System Identification, 1994\\\n[3] D Ha, J Schmidhuber, Recurrent world models facilitate policy evolution, 2018\\\n[4] T. Gabel, M. Riedmiller, Reducing policy degradation in neurodynamic programming, 2006\\\n[5] M. Migliavacca et al., Fitted policy search: Direct policy search using a batch reinforcement learning approach, 2010\\\n[6] A. Hans, S. Duell, S. Udluft, Agent self-assessment: Determining policy quality without execution, 2011\\\n[7] S Depeweg, Modeling Epistemic and Aleatoric Uncertainty with Bayesian Neural Networks and Latent Variables, 2018\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper deals with the possibility to choose hyperparameters for reinforcement learning based on a model of the environment. This model is created from pre-recorded data of the environment. This model is called \"calibration model\" in this paper, which I think is a very appropriate name.\n\nOne problem is that the paper presents this technique as being new, although it is state of the art. \nEspecially the insight \"The calibration model need not be a perfect simulator to be useful for identifying reasonable hyperparameters, whereas learning a transferable policy typically requires accurate models.\", though completely correct and very important in practice, is not a new insight.\n\nHowever, since there is little published systematic research on this technique, this paper can help to fill this gap.\n\nIn this paper, experiments are performed on two very simple benchmarks.\nAs a data-based model, a k-nearest neighbors (KNN) approach improved by metric learning is used and compared to a NN that also uses the variables of the learned metric.\nFurthermore, some limitations are discussed and theoretical considerations are provided.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The paper can contribute to scientific progress 1) by naming and discussing the known approach for which published studies are scarce, and 2) by presenting KNN with with Laplace representation. Where I consider 1) to be the more important contribution.\n\nHowever, to be valuable, all inaccurate or insufficiently substantiated claims (see weaknesses) must be reformulated so that they are true and sufficiently substantiated. In particular, because this means not making the claim that the approach is generally novel, it is necessary to completely reword the Abstract, Introduction, and other parts of the text. Possibly, the title should also be chosen more modest and appropriate to the contribution actually made.\n\nExtensive experiments in more difficult and high dimensional environments and comparisons with state of the art system identifiers would drastically increase the usefulness of the paper. However, I think it is understandable if this is seen as future work.\n", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a procedure for tuning the hyperparameters of online reinforcement learning algorithms, using a model estimated using offline interaction data. After describing the proposed approach, the authors show empirical results on three traditional reinforcement-learning testbeds.

The three expert reviewers who assessed this work agree that the proposed approach is simple and intuitive, the topic is relevant, and the paper is clearly written and well executed. On the other hand, they pointed out a series of weaknesses related to the novelty of the proposed approach and its scalability to more complex/real-world domains. Most of these issues have been effectively addressed by the authors in their rebuttal and they have to update their paper accordingly. In particular, since some misunderstandings were common among multiple reviewers, this is a symptom that the writing was not clear enough about those points and the authors need to take particular care in adjusting them.
For what concerns the experimental part, the authors mention that this work was inspired by a real application on a water treatment use case. It would be wonderful if the authors could show results in this domain even by providing them in a qualitative way in order not to violate any non-disclosure agreement.

In conclusion, according to the TML guidelines, we are glad to accept this manuscript, while strongly encouraging the authors to make a final effort to address the reviewers' concerns.
Decision: Accept (poster)