User-Interactive Offline Reinforcement Learning
Authors: Phillip Swazinna
Steffen Udluft
Thomas Runkler
Venue: ICLR 2023 poster (2023)
Keywords: Offline RL
Reinforcement Learning
User
Model-based
Adaptive

Abstract: Offline reinforcement learning algorithms still lack trust in practice due to the risk that the learned policy performs worse than the original policy that generated the dataset or behaves in an unexpected way that is unfamiliar to the user. At the same time, offline RL algorithms are not able to tune their most important hyperparameter - the proximity of the learned policy to the original policy. We propose an algorithm that allows the user to tune this hyperparameter at runtime, thereby addressing both of the above mentioned issues simultaneously. This allows users to start with the original behavior and grant successively greater deviation, as well as stopping at any time when the policy deteriorates or the behavior is too far from the familiar one.

Published as a conference paper at ICLR 2023
USER-INTERACTIVE OFFLINE REINFORCEMENT LEARNING
Phillip Swazinna Siemens & TU Munich Munich, Germany swazinna @ de
Steffen Udluft
Thomas Runkler
Siemens Technology
Siemens & TU Munich
Munich, Germany
Munich, Germany
thomas.runkeed @ siemens.com
ABSTRACT
Offline reinforcement learning algorithms still lack trust practice due the risk that the learned policy performs worse than the original policy that generated the dataset or behaves in an unexpected way that is unfamiliar to the user. At the same time, offline RL algorithms are not able to tune their most important hyperparameter the proximity of the learned policy to the original policy. We propose an algorithm that allows the user to tune this hyperparameter at runtime, thereby addressing both of the above mentioned issues simultaneously. This allows users start with the original behavior and grant successively greater deviation, well as stopping at any time when the policy deteriorates or the behavior too far from the familiar one.
INTRODUCTION
Recently, offline reinforcement learning (RL) methods have shown that it is possible to learn effective policies from a static pre-collected dataset instead of directly interacting with the environment (Laroche et al. 2019 Fujimoto et al., 2019; Yu et al. 2020; Swazinna et al. 2021b). Since direct interaction is in practice usually very costly, these techniques have alleviated large obstacle on the path of applying reinforcement learning techniques in real world problems.
major issue that these algorithms still face is tuning their most important hyperparameter: The proximity to the original policy. Virtually all algorithms tackling the offline setting have such hyperparameter, and it is obviously hard tune, since no interaction with the real environment permitted until final deployment. Practitioners thus risk being overly conservative (resulting in no improvement) or overly progressive (risking worse performing policies) in their choice.
Additionally, one of the arguably largest obstacles the path to deployment of RL trained policies most industrial control problems is that (offline) RL algorithms ignore the presence of domain experts, who be seen users of the final product the policy. Instead, most algorithms today can be seen as trying to make human practitioners obsolete. We argue that is important to provide these users with utility- something that makes them want to use RL solutions. Other research fields, such as machine learning for medical diagnoses, have already established the idea that domain experts crucially important to solve the task and complement human users in various ways Babbar et al.) (2022); Cai et al. (2019); De-Arteaga et al. (202)) >): Fard & Pineau (2011): Tang et al. (2020). We see our work line with these and other researchers (Shneiderman 2020 Schmidt [l.[2021 who suggest that the next generation of Al systems needs to adopt usser-centere approach and develop systems that behave more like an intelligent tool, combining both high levels of human control and high levels of automation. We seek to develop offline RI. method that does just that. Furthermore, we see giving control to the user as a requirement that may in the future much more enforced when regulations regarding AI systems become more strict: The EU's high level expert group AI has already recognized "human autonomy and oversight" as a key requirement for trustworthy AI in their Ethics Guidelines for Trustworthy AI (Smuha 2019). In the future, solutions found with RL might thus be required by law to exhibit features that enable more human control.
In this paper, we thus propose simple method to provide users with more control over how offline RL policy will behave after deployment. The algorithm that we develop trains a conditional policy, that after training adapt the trade-off between proximity to the data generating policy on

---

Published as a conference paper at ICLR 2023
the one hand and estimated performance the other. Close proximity to known solution naturally facilitates trust, enabling conservative users to choose behavior they are more inclined to confidently deploy. That way, users may benefit from the automation provided by offline RL (users don't 't need to handcraft controllers, possibly even interactively choose actions) yet still remain in control as they can e.g. make the policy move such an algorithm can be designed, as well as compare its performance with a variety of offline RL baselines and show that a user can achieve state of the art performance with it. Furthermore, we show that our method has advantages over simpler approaches like training many policies with diverse hyperparameters. Finally, since we train a policy conditional on one of the most important hyperparameters in offline RL, we show how a user could potentially use it to tune this hyperparameter. In many cases of our evaluations, this works almost regret-free, since we observe that the performance as function of the hyperparameter is mostly a smooth function.
more conservative more liberal trade-off. We show how
2 RELATED WORK
Offline RL Recently, a plethora of methods has been published that learn policies from static datasets. Early works, such as FQI and NFQ (Ernst et al. 2005 Riedmiller, 2005), were termed batch instead of offline since they didn't explicitly address issue that the data collection cannot be influenced. Instead, similarly to other batch methods (Depeweg et al. 2016,] (Hein et al. 2018 Kaiser et al., 2020), they assumed a uniform random data collection that made generalization to the real environment simpler.
Among the first to explicitly address the limitations in the offline setting under unknown data collection were SPIBB(-DQN) (Laroche et al. 2019) in the discrete and BCQ (Fujimoto et al. 2019) in the continuous actions case. Many works with different focuses followed: Some treat discrete MDPs and come with provable bounds on the performance at least with a certain probability Thomas ct al.
et al. (2019), however many more focused the continuous setting: EMaQ
BEAR, BRAC, ABM, various DICE based methods, REM, PEBL., ESEC-TD-O, CQL, IQL. BAIL.
CRR, COIL, O-RAAC, OPAL, TD3+BC, and RvS (Ghasemipour al. [02TT]
1 Kumar al.
2019
Wu et al., 2019; Siegel et al. 2020; Nachum et al.
2019: Zhang et al. 2020 Agarwal et al. 2020 Kostrikov et al.) [021] Chen al.
2020
Smit et al. 2022 Payse al. [2020] Kumar et al.
2019
Wang et al. 2020 Liu et al.) |021 et al. 2021 AAjay et al. 2020: Brandfonbrener et al. 2021
Emmons et al.j (021) are just few of the proposed model-free methods over the last few years.
Additionally, many model-based as well as hybrid approaches have been proposed, such as MOPO,
MOReL, MOOSE, COMBO. RAMBO, and WSBC (Yu al. 2220 Kidambi et al. 2020 [wazinna Yu et al. 202T |Rigeer et al., 2022 Swazinna et al. (021a)). Even approaches that train
et al.
2b2
policies purely supervised, by conditioning on performance, have been proposed (Peng ct al. 2019
Emmons et al.
2021 Chen et al. 2021.. Model based algorithms more often use model uncertainty.
while model-free methods use a more direct behavior regularization approach.
Offline policy evaluation or offline hyperparameter selection is concerned with evaluating (or at least ranking) policies that have been found by an offline RL algorithm, in order to either pick the best performing one or tune hyperparameters. Often, dynamics models used evaluate policies found in model-free algorithms, however also model-free evaluation methods exist (Hans et al.) 2011 Paine al. 2020 Konyushova et al. 2021 Zhang et al.) 2021b: Fu et al. (021). Unfortunately, but also intuitively, this problem is rather hard since if any method is found that can more accurately assess the policy performance than the mechanism in the offline algorithm used for training, should be used instead of the previously employed method for training. Also, the general dilemma of not knowing in which parts of the state-action space we know enough to optimize behavior seems to always remain. Works such as Zhang et al. (202 Lu et al. (2021) become applicable if limited online evaluations are allowed, making hyperparameter tuning much more viable.
Offline RL with online adaptation Other works propose an online learning phase that follows after offline learning has conceded. In the most basic form, Kurenkov & Kolesnikov (2021) introduce an online evaluation budget that lets them find the best set of hyperparameters for an offline RL algorithm given limited online evaluation resources. In an effort to minimize such a budget, Yang et al. (2021) train set of policies spanning diverse set of
trade-offs. Ma et al. (202I) propose a conservative adaptive penalty, that penalizes unknown behavior more during the beginning and less during the end of training, leading to safer policies during training. In Pong et al.
2

---

Published as a conference paper at ICLR 2023
(2021); Nair et al. (222)) Zhao al. (2021) methods for effective online learning phases that follow the offline learning phase are proposed. In contrast to these methods, we are not aiming for a fully automated solution. Instead, we want to provide the user with valuable tool after training, not propose an actual online phase, also since practitioners may find any performance deterioration inacceptable. To the best of our knowledge, no prior offine RL method produces policies that remain adaptable after deployment without any further training.
do
3 LION: LEARNING IN INTERACTIVE OFFLINE ENVIRONMENTS
In this work, we address two dilemmas of the offline RL setting: First and foremost, we would like to provide the user with high level control option in order to influence the behavior of the policy, since we argue that the user is crucially important for solving the task and not to be made obsolete by an algorithm. Further we address the issue that in offline RL, the correct hyperparameter controlling the trade-off between conservatism and performance is unknown and can hardly be tuned. By training a policy conditioned in the proximity hyperparameter, we aim to enable the user to find a good trade-off hyperparameter. Code will be made available at http:/gghhhbcooomswwwiianne
As mentioned, behavior cloning, will most likely yield the most trustworthy solution due to its familiarity, however the solution is of very limited use since it does not outperform the previous one.
Offline RL on the other hand is problematic since we cannot simply evaluate policy candidates on real system and offline policy evaluation is still an open problem (Hans et al., 201T Paine et al. 2020
Konyushova et al., 2021| Zhang et al. 2021b] Fu et al. 2021). In the following, we thus propose a solution that moves the hyperparameter choice from training to deployment time, enabling the user to interactively find the desired trade-off between BC and offiin optimization. user may then slowly move from conservative towards better solutions.
3.1 TRAINING
During training time, we optimize three components: A model of the original policy (s)s), an ensemble of transition dynamics models {ff W. (s, a)li E ., - 1}. as well as the user adaptive policy Ta (s, A). The dynamics models {f well the original policy B are trained in isolation before the actual policy training starts. Both æ and /3 are always simple feedforward neural networks which map states directly actions in deterministic fashion (practitioners likely favor deterministic policies over stochastic ones due to trust issues). , is trained to simply imitate the behavior present in the dataset by minimizing the mean squared distance the observed actions:
((ø) =
Σ [a, - -s,
N
BL.OLND
Depending on the environment, the transition models are either also feedforward networks or simple recurrent networks with a single recurrent layer. The recurrent networks build their hidden state over G steps and are then trained to predict window of size F into the future (similarly to (Hein) (et al, (017b)), while the feedforward dynamics simply predict single step transitions. Both use mean squared error loss:
I(bb)) N
Σ
[8141 -8, (8e,
(2)
es De
I(b))
Σ IND f=1
|84G+/+1
fhx (se, at.. S14G, ..... "u..
where 8 f are the model predictions that are fed back to be used as input again. For simplicity. in this notation assume the reward to be part of the state. Also we do not explicitly show the recurrence and carrying over of the hidden states.
After having trained the two components (s(s) and ( a)). we can then move on to policy training. Similarly to MOOSE and WSBC, optimize the policy тт by sampling start states from D and performing virtual rollouts throughout the dynamics ensemble using the current policy candidate. In every step, the ensemble predicts the reward the minimum among its members and the next state
3

---

Published as a conference paper at ICLR 2023
that goes with it. At the same time collect the mean squared differences between the actions that Ty took in the rollout and the one that 3g would have taken. The loss is then computed as a weighted sample the weighting factor À randomly and pass to the
sum of the two components. Crucially, policy as an additional input- the policy thus needs to learn all behaviors ranging from pure hehavior cloning entirely free optimization:
L(0) -- Σ Σ (A((s((s (1-1)a((a))) S0~D
at To(sE, A)
where we sample À between 0 & 1, c(8e; a4) = min (st, output of the ensemble prediction for reward (we omit explicit notation of recurrence for simplicity) and p(a,) [B,b (sx) azJ² denotes the penalty based on the mean squared distance between the original policy and the actions proposed by no- See Fig. for visualization of our proposed training procedure.
az) E O,.., denotes the
Return H
Penalty P
L(
Dataset Scart State
Predicted
A Distrbbtion
Sampled A
Original Policy
Policy n
Dynamics
Model
Behavior
Action
Policy
Distance
Action
Figure I: Schematic of LION policy training. During policy training (Eq, 3 only ma (in green) is adapted, while the original policy model Bg (orange) and the dynamics ensemble {.fes) } (blue) are already trained and remain unchanged. From left to right, we first sample a start state (black) from the dataset and À value from its distribution. Then, let the original policy (orange) well the currently trained policy (green) predict actions note that the newly trained policy is conditioned on A. Both actions are then compared calculate the penalty for that timestep (red). The action from the currently trained policy is then also fed into the trained transition model (blue) together with the current state (black/ / blue), get the reward for that timestep (yellow) This procedure is repeated until the horizon of the episode is reached. The rewards and penalties are then summed up and weighted by be used loss function for policy training.
well as the next state (blue).
We motivate our purely model-based approach (no value function involved) with the fact that we have fewer moving parts: Our ensemble can be kept fixed once is trained, while a value function has to be learned jointly with ng. which is in our case more complex than usual. See experimental results in Fig. 10 brief attempt at making our approach work in the model-free domain.
In addition to Eq. 3 we need to penalize divergence not only from the learned model of the original policy during virtual roll-outs, but also from the actual actions in the dataset at À this is not done, the trained policy π sticks to the (also trained) original policy / during the rollouts, but during those rollouts, there are states that did not appear in the original dataset, enabling ा to actually diverge from the true trajectory distribution. We thus penalize both rollout as well as data divergence at À 0:
seems that
I(0) ΣΣ
x (e((s, - -))((a))) 1 La la-((s A= 0)}
SOND
8.0D
where 77 controls the penalty weight for not following dataset actions at = 0, see Appendix(A]too more details. Furthermore, normalize states to have zero mean and unit variance during every
4

---

Published as a conference paper at ICLR 2023
forward pass through dynamics model or policy, using the mean and standard deviation observed in the dataset. We also normalize the rewards provided by the ensemble r. = c((s. at), so that they live in the same magnitude the action penalties (we assume actions to be in [-1,1] that the penalty can be in [0, 4]P where D is the action dimensionality).
Intuitively, one might choose sample À uniformly between zero and one, however instead choose a beta distribution with parameters (0.1, 0.1), which could be called bathtub-shaped. Similarly to (Seo al. 2021), we find that it is important to put emphasis on the edge cases, so that the extreme behavior is properly learned, rather than putting equal probability mass on each value in the [0. 1] range. The interpolation between the edges seems to be easier and thus require less samples. Fig. shows policy results for different lambda distributions during training.
3.2 DEPLOYMENT
At inference time, the trained policy can at any point be influenced by the user that would otherwise be in con- trol of the system, by choosing the À that passed to the policy together with the current system state to obtain an action:
Algorithm LION (Training)
Require Dataset D = (T,), randomly initialized param- eters 9, o, sb, lambda distribution parameters Beta(a, b), horizon H, number of policy updates U
2: // dynamics and original policy models can be trained supervised and independently of other components
02 = ()) X E User(st) (5)
3: train original policy model 3& using D and Equation
He or she may choose to be conserva-
4: train dynamics models f with D and Equation 2
tive or adventurous, observe the feed- back and always adjust the proximity parameter of the policy accordingly. At this point, any disliked behavior can immediately be corrected without any time loss due re-training and deploying new policy, even if the user's specific preferences were not known at training time.
5: for j in 1.. do
sample start states So D
7: sample lambda values À ~ Beta(a, b)
8: initialize policy loss L()) =0
9:
for t in O..H do
10:
calculate policy actions a₂ no (8,, A)
11:
calculate behavioral actions b. = (8(
12:
calculate penalty term p(ac) = 18 - asl
13:
Tr, 8t+1 = fe. (sc. ar)
We propose to initially start with A = 0 during deployment, in order to check whether the policy is actu- ally able to reproduce the original pol- icy and to gain the user's trust in the found solution. Then, depending on how critical failures are and how much time is at hand, À may be increased in small steps for as long as the user is still comfortable with the observed behavior. Figure3showe an example of how the policy behavior changes over the course of A. Once the performance stops to increase or the user is otherwise not satisfied, we can immediately return the last satisfying À value.
s.t. i= arg min; (r( (St, ac))}
14:
= 10)+=->'00 - (1 - -))p(a;)]
15:
update xa using gradient V (() and Adam
16:
return me:
4 EXPERIMENTS
At first, we intuitively showcase LION in a simple 2D-world in order to get an understanding of how the policy changes its behavior based on A. Afterwards, we move to a more serious test, evaluating our algorithm on the 16 industrial benchmark (IB) datasets (Hein et al. 2017a; Swazinna et al. 2021b). We aim to answer the following questions:
Do LION policies behave as expected, i.e. do they reproduce the original policy at À = 0 and deviate more and more from it with increased freedom to optimize for return?
Do LION policies at least in parts of the spanned X space perform better or similarly well to state of the art offline RL algorithms?
Is easy to find the À values that maximize return for practitioners? That is, are the performance courses smooth or do they have multiple local mini- & maxima?
Is it possible for users to exploit the À regularization at runtime to restrict the policy to only exhibit behavior he or she is comfortable with?
5

---

Published as a conference paper at ICLR 2023
4.1 2D- WO-LD
Behavicral Policy wi naie
[b] Ein reward distr bulion 10
We evaluate the LION approach on a simplistic 2D benchmark. The states are & y coordi- nates in the environment and rewards are given based on the position of the agent, following Gaussian distribution around a fixed point in the
state space, i.e. r(s) dv22m
1
-.3((===-)/o)²
In this example we set μ (3. 6) and a (1.5, 1.5)". A visualization of the reward dis- tribution can be seen in Fig. (b). We collect data from the environment using a simple policy that moves either to position (2.5, 2.5)² or to (7.5, 7.5)". depending on which is closer to the randomly drawn start state (shown in Fig. 2 a)), adding 10% random actions exploration. Then we follow the outlined training procedure, by training a transition model, original policy model and finally a new policy that can at runtime change its behavior based on the desired proximity the original policy. Fig. [3] shows policy maps for E {0.0, 0.6, 0.65, 0.7, 0.85, 1.0), moving from simply imitating the original policy, over different mixtures, pure return optimization. Since the task easy and accurately modeled by the dynamics ensemble, one may give absolute freedom to the policy and optimize for return only. As it can be seen, the policy moves quickly to the center of the reward distribution for 1.
Figure 2: (a) Original policy for data collection
and color represents action direction (b) reward distribution in the 2D environment color repre- sents reward value
Pollcymao 0.000
Policymap 0.6
Policymap 0.65
Pollcymap 0
Policymap 0.85
Pollcymao A1110
1.5m
0.5m
Figure 3: Policy maps for increasing values of X in the 2D environment colors represent action direction. Initially, the policy simply imitates the original policy (see Fig. (a)). With increased freedom, the policy moves less to the upper right and more the bottom left goal state of the original policy, since that one closer to the high rewards. Then, the policy moves its goal slowly upwards on the y-axis until it is approximately at the center of the reward distribution. Since enough data was available (1,000 interactions) and the environment so simple, the models capture the true dynamics well and the optimal solution is found at À 1. This is however not necessarily the case if not enough or not the right data was collected (e.g. due to a suboptimal original policy see Fig. 4
4.2 INDUSTRIAL. BENCHMARK
Datasets We evaluate LION on the industrial benchmark datasets initially proposed in (Swazinna et al. 2021b). The 16 datasets are created with three different baseline original policies (optimized, mediocre, bad) mixed with varying degrees of exploration. The optimized baseline is an RL trained policy and simulates forth around fixed point that rather well behaved, while the bud baseline steers to a point on the edge of the state space in which rewards are deliberately bad. Each baseline is combined with E E {0.0, 0.2, 0.4, 0.6, 0.8, 1.0]-ge 1.0--gredyy exploration to collect a dataset (making the s = 0.0 datasets extreme cases of the narrow distribution problem). Together, they constitute diverse set of offline RL settings. The exact baseline policies are given by:
expert practitioner. The mediocre baseline moves the system back and
100 Uz.
25 Ut
- 5 0.91 2 p 5 1.43
Tbad
100 gt.
Tied
25
gt
Topt
100 ht
25
h
-3.48h
h4-1 1 + 2p + 0.81
The datasets contain 100,000 interactions collected by the respective baseline policy combined with the E-greedy exploration. The IB is a high dimensional and partially observable environment if access to the full Markov state were provided, it would contain 20 state variables. Since only six of those are observable, and the relationship to the other variables and their subdynamics complex

---

Published as a conference paper at ICLR 2023
and feature heavily delayed components, prior work Hein et al. 2017b) has stated that up to 30 past time steps are needed to form a state that can hope to recover the true dynamics, so the state can be considered 180 dimensional. In case thus set the number of history steps G = 30. The action space is 3 dimensional. The benchmark is not supposed to mimic a single industrial application, but rather exhibit common issues observable in many different applications (partial observability, delayed rewards, multimodal and heteroskedastic noise, ...). The reward is weighted combination of the observable variables f'ttigee and consumption, which are conflicting (usually move in opposite directions and need trade-off) and are influenced by various unobservable variables. As in prior work Hein et al. (2018); Depeweg et al. (2016); Swazinna et al. (202Tb) we optimize for a horizon of 100. The datasets are available at httpt: thw..b...huuu.ggggggggggg..... com/ s emmen/ / industria a benchharrk
tree/ ottee/ffffieeeetetttteteeeeeeeee datasets datasets under the Apache License 2.0.
had
had
bad
bed 0.8
-150
-122
0.90
2.50
0.33
0.32
0.25
120
230
033
0/1
medierre 9.0
2 medisrr 0.2
mediecre 0.4
medioce 2.5
mediccre 0.8
150
-100
-100
112
155
-150
-70
0.5
dppptin sator PadLus Faratete
1-Opt BE0 Matzre Parametar
4. 1-Gptmooston Pistue Pa amoee
7-Opteenstr Mature hasmattr
optimized 0.0
IB optimized 0.2
3 oztimized 0.4
optimized 0.8
B apciniedd 0.8
-57.5
-00.3
-50.5
-65
0.25
Oatim balee
Pidtrr Trango
bo0m)
SCD TD3-0C
HORD
Figure 4: Evaluation performance (top portion of each graph) and distance to the original policy (lower portion of each graph) of the LION approach over the chosen À hyperparameter. Various state of the art baselines are added as dashed lines with their standard set of hyperparameters (results from (Swazinna et al. 2022). Even though the basclines all exhibit some hyperparameter that controls the distance to the original policy, all are implemented differently and we can neither map them to a corresponding lambda value of our algorithm, nor change the behavior at runtime, which is why we display them as dashed lines over the entire X-spectrum. See Fig. 12 for the 100% exploration dataset.
Baselines We compare performances of LION with various state of the art offline RL baselines:
BEAR, BRAC, BCQ. CQL and TD3+BC (Kumar al. 2019 Wu et al. 2019 Fujimoto
et al. 2019 Kumar et al., 2020 Fujimoto & Guj 2021) l) are model-free algorithms. They mostly regularize the policy by minimizing divergence the original policy. BCQ samples only likely actions and CQL searches for a O-function that lower bounds the true one.
MOOSE and WSBC (Swazinna al. 2021a l b) are purely model based algorithms that optimize the policy via virtual trajectories through the learned transition model. MOOSE penalizes reconstruction loss of actions under the original policy (learned by an autoencoder), while WSBC constrains the policy directly in weight space. MOOSE is from the policy training perspective the closest to our LION approach.
MOPO and MOReL (Yu et al. 2020 Kidambi et al. 2020) are hybrid methods that learn a transition model as well as a value function. Both use the models to collect additional data and regularize the policy by means of model uncertainty. MOPO penalizes uncertainty directly, while MOReL simply stops episodes in which future states become too unreliable. MOReL uses model disee--iisgrreement and MOPO Gaussian outputs to quantify uncertainty.
7

---

Published as a conference paper at ICLR 2023
Evaluation In order to test whether the trained LION policies able to provide state of the art performance anywhere in the À range, we evaluate them for from 0 to I in many small steps. Figs. and [2 show results for the 16 IB datasets. We find that the performance curves do not exhibit many local optima. Rather, there is usually a single maximum before which the performance is rising and after which the performance is strietly dropping. This is a very desirable characteristic for usage in the user interactive setting, as it enables users to easily find the best performing À value for the policy. In 13 of 16 datasets, users can thus match or outperform the current state of the art method on that dataset, and achieve close to on-par performance on the remaining three. The co-oonceoto-ooiaala-------
curves are even monotonously increasing from start to finish, making possible for the practitioner to find the best solution he or she is still comfortable with in terms of distance to the familiar behavior.
Discrete baseline A simpler approach might be to train an existing offline RL algorithm for many trade-offs advance, to provide least discrete options. Two downsides are obvious: (a) we wouldn't be able to handle the continuous case, i.c. when user wants a trade-off that lies between two discrete policies, and (b) the computational cost increases linearly with the number of policies trained. We show that a potentially collection policies with different hyperparameters. completely independently of each other, they often exhibit wildly different behaviors even when the change in hyperparameter instead expresses the collection as a single policy network, training them jointly and thus forcing them to smoothly interpolate among each other. This helps make the performance smooth function of the hyperparameter (although this must not always be the case) and results in a performance landscape that much easier to navigate for a user searching for a good trade-off.
bigger issue exists in Figure 5 When train discrete
small. LION
$e0-04 -Nul Hyporporemmttrrreo Bustiro
Mudaopa-03 -MLL Hyporperreutte Buseiro
Opiriico Malli wpp.raaapooaocooo Dareing
W
-31
-000
2.5
3.5
2.5
Figure 5: Prior offline RL algorithms like MOPO do not behave consistently when trained across a range of penalizing hyperparameters.
Return conditioning baseline Another interesting line of work trains policies conditional on the return to go. such as RvS (Emmons al.) (021) (Reinforcement Learning via Supervised Learning) or DT (Chen et al. 2021) (Decision Transformer). A key advantage of these methods is their simplicity they require neither transition model nor value function, just a policy suffices, and the learning can be performed in entirely supervised fashion. The resulting policies could be interpreted in a similar way as LION policies: Conditioning on returns close to the original performance would result the original behavior, while choosing to condition on higher returns may lead to improved performance if the extrapolation works well. Fig. 6 we report results of the RvS algorithm on the same datasets as the discrete bascline. The returns in the datasets do not exhibit lot of variance, it is unsurprising that the approach did not succeed in learning a lot of different behaviors.
Bad-0.4
Mediorre-o..
2e5-0.
-40
-55
.00
-340
-3.0
-150
-230
-30
-27
-250
-22
-200
-175
-150
-325
-287
-18
-70
-100
-2.5
-70.0
-013
-06
-0.5
-00.
-013
-850
Condiaaned Aaum
Condiionad Pestur
Conditioed Raturn
Figure 6: Return conditioned policies did not learn many different behaviors on the datasets.
8

---

Published as a conference paper at ICLR 2023
Finding suitable À We would like to emphasize that we neither want to optimize all offline hyperparameters with our solution, nor are we interested in a fully automated solution. Users may thus adopt arbitrary strategies to finding their personal trade-off of preference. We will however provide conservative example strategy: The operator starts with the most conservative value available and then moves in small, but constant steps towards more freedom. Whenever the performance drops below the previous best or the baseline performance, he immediately stops and uses the last A before that. Table I summarizes how this strategy would perform.
LION
MOPO
RvS
LION
MOPO
RvS
Dataset
Final X
Final A
Final ft
Return
Return
Return
Bad-0.4
1.0
2.4
-287
-102.4
-17722
-158.2
Mediocre-O.0
0.2
2.4
-81
-70.8
-87.8
-86.6
Optimized-0.6
0.35
2.5
-71
-58.9
-72.4
-61.9
Table I: If a user adopts the simple strategy of moving in small steps (0.05 for LION, 0.1 for MOPO since its range is larger, / 1.0 for RvS) from conservative towards better solutions, immediately stopping when a performance drop is observed, LION finds much better solutions due to the consistent interpolation between trade-offs. Note that in MOPO, start with large À = 2.5 (1.0 is the default) since there it controls the penalty, while we start with À = O in LION, where it controls the return.
DISCUSSION & CONCLUSION
In this work we presented a novel offline RL approach that, to the best of our knowledge, is the first to let the user adapt the policy behavior after training is finished. We let the user tune the behavior by allowing him to choose the desired proximity to the original policy, in an attempt to solve two issues: (1) The problem that practitioners cannot tune the hyperparameter in offline RI. & (2) the general issue that users have no high level control option when using RL policies (they might even have individual preferences with regards the behavior of a policy that go beyond just performance).
We find that effectively, LION provides a high level control option to the user, while still profiting from high level of automation. It furthermore takes much of the risk that users normally assume in offline RL away since deployments can always start with BC policy when they start at À = 0, before moving to better options. While behavior cloning does not have to work in general, we did not experience any issues with it in our experiments, and it should be easier than performing RL since it can be done entirely start with minimal risk. In prior offline algorithms, users experienced the risk that the algorithm did not produce a satisfying policy on the particular dataset they chose. E.g.: WSBC produces state of the art results for many of the IB datasets, however for mediocre-0.6 it produces catastrophic -243 (original performance is -75). Similarly, CQL is the prior best method on optimized-0.8, however the same method produces a performance of -292 on bad-0.2 (MOOSE, MOPO, and WSBC get between -110 & -130). Due to the smoothness of the interpolation of behaviors in LION, practitioners should be able to use it to find better trade-offs with lower risk than prior methods. Adaptable policies thus likely step towards more deployments in industrial applications.
supervised fashion. Given that BC works, deployments can thus
Future Work As outlined at the end of Section[ooff the appendix, we were unable to incorporate value functions into our approach. This can be seen as a limiting factor, since there exist environments with sparse very delayed rewards or that for other reasons exhibit long planning horizons. The industrial benchmark features delayed rewards and evaluation trajectories are 100 steps long, however other environments can be more extreme in their characteristics. At some point, even the best dynamics models suffer from compounding errors and cannot accurately predict the far away future. We do not believe that it is in principle not possible to combine the LION approach with value functions, however future work will likely need to find methods to stabilize the learning process.
Other potential limitations of our approach include difficulties with the behavior cloning, e.g. when the original policy is stochastic or was not defined in the same state space we use (e.g. different human operators controlled the system at different times in the dataset), as well as difficulties when interpolating between vastly different behaviors on the Pareto front spanned by proximity and performance. We mention these potential limitations only for the sake of completeness since we were unable to observe them in our practical experiments.

---

Published as a conference paper at ICLR 2023
6 ETHICS STATEMENT
Generally, (offline) reinforcement learning methods can be used to optimize control policies for many different systems beyond the control of the researchers. Malicious individuals may thus use findings in manners that negatively impact others or society as whole (e.g. by using it in weapons). Faulty policies could also potentially result in other safety concerns (e.g. policies for heavy machinery). In ML research, there is also always the concern that increased automation may lead to rising unemployment. While some institutions like the UN and the WEF are convinced Al will lead to long term job growth, Bordot (002)) found that a slightly increased unemployment risk can be found empirically.
Beyond these usual concerns that mostly apply to ML/RL research whole, rather than individual paper, we believe that no additional considerations are necessary for our method. If at all, the societal impact should be positive, since solution fosters trust by practitioners and should thus increase the usage and human-machine collaborative deployment of offline RL policies in practice, enabling more efficient control strategies in many industrial domains.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pp. 104-114. PMLR, 2020.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. 2020.
Varun Babbar, Umang Bhatt, and Adrian Weller. On the utility of prediction sets in human-Al teams. arXiv preprint arXiv:2205.01411, 2022.
Florent Bordot. Artificial intelligence, robots and unemployment: Evidence from OECD countries.
Journal of Innovation Economics Management, 37(1):117-138, 2022.
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without off- off-policy evaluation. Advances in Neural Information Processing Systems, 34:4933-4944 2021.
Carrie J Cai, Emily Reif, Narayan Hegde, Jason Hipp. Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg Corrado, Martin C Stumpe, et al. Human-centered tools for coping with imperfect algorithms during medical decision-making In Proceedings of the 2019 chi conference on human factors in compuling systems, pp. 1-14, 2019.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 04415084-15097, 2021.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. BAIL: Best-action imitation learning for batch deep reinforcement learning. arXiv preprint arXiv: 2019.
Maria De-Arteaga, Artur Dubrawski, and Alexandra Chouldechova. Leveraging expert consistency to improve algorithmic decision support. arXiv preprint arXiv:2101.00 099648 2021.
Stefan Depeweg, José Miguel Hernández--Lbato, Finale Doshi-Velez, and Steffen Udluft. Learning and policy search in stochastic dynamical systems with Bayesian neural networks. arXiv preprint 1rXiv:1605.07122 2016.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. RvS: What is essential for offline RL via supervised learning? arXiv preprint arXiv:2112.10755 2021.
Damien Emst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. J. Mach. Learn. Res., 6:503-556, December 2005. ISSN 1532-4435.
M Milani Fard and Joelle Pineau. Non-deteriinistic policies in Markovian decision processes. Journal of Artificial Intelligence Research, 40:1-24, 2011.
10

---

Published as a conference paper at ICLR 2023
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang. Michael R Zhang, Yutian Chen, Aviral Kumar, et al. Benchmarks for deep off- evaluation. arXiv preprint arXhv:2003.1659, 2021.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Scott Fujimoto. David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2002-202222222 2022. PMLR, 2019.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-man Q-learnnn operator for simple yet effective offline and online RL. In International Conference on Machine Learning, pp. 3682-3691, PMLR, 2021.
Alexander Hans, Siegmund Duell, and Steffen Udluft. Agent self-assessment: Determining policy quality without execution. In 201/ IEEE Symposium Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pp. 84-90. IEEE, 2011.
Daniel Hein, Stefan Depeweg, Michel Tokic, Steffen Udluft, Alexander Hentschel, Thomas A Runkler, and Volkmar Sterzing. benchmark environment motivated by industrial control problems. In 2017 IEEE Symposium Series Computational Intelligence (SSCI), pp. 1-8. IEEE, 2017a.
Daniel Hein, Steffen Udluft, Michel Tokie, Alexander Hentschel, Thomas A Runkler, and Volkmar Sterzing. Batch reinforcement learning on the industrial benchmark: First experiences. In 2017 International Joint Conference on Neural Networks (IJCNN), pp. 4214 4221. IEEE, 2017b.
Daniel Hein, Steffen Udluft, and Thomas A Runkler. Interpretable policies for reinforcement learning by genetic programming. Engineering Applications of Artificial Intelligence, 76:158-169 2018.
Markus Kaiser, Clemens Otte, Thomas Runkler, and Carl Henrik Ek. Bayesian decomposition of multi-modal dynamical systems for reinforcement learning. Neurocomputing, 2020.
Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 11810-21823, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988) 2014.
Ksenia Konyushova, Yutian Chen, Thomas Paine, Caglar Gulcehre, Cosmin Paduraru, Daniel Mankowitz, Misha Denil, and Nando de Freitas. Active offline policy selection. Advances in Neural Information Processing Systems, 34, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning arXiv preprint arXiv:2 10.0v:2110.06166) 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pp. 11761-11771,2011
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1 179-119,,, 2020.
Vladislav Kurenkov and Sergey Kolesnikov. Showing your offline reinforcement learning work: Online evaluation budget matters. arXiv preprint arXiv:21 10.04 041004 156, 2021.
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International Conference Machine Learning, pp. 3652-3661. PMLR, 2019.
Minghuan Liu, Hanye Zhao, Zhengyu Yang, Jian Shen, Weinan Zhang, Li Zhao, and Tie- Yan Liu. Curriculum offline imitating learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
11

---

Published as a conference paper at ICLR 2023
Cong Lu, Philip Ball, Jack Parker-Holder, Michael Osborne, and Stephen Roberts. Revisiting design choices in offline model based reinforcement learning. In International Conference on Learning Representations, 2021.
Yecheng Jason Ma, Andrew Shen, Osbert Bastani, and Dinesh Jayaraman. Conservative and adaptive penalty for model-based safe reinforcement learning. arXiv preprint arXiv:2112.07701, 2021.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnootic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019.
Kimia Nadjahi, Romain Laroche, and Rémi Tachet des Combes. Safe policy improvement with soft baseline bootstrapping. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 53-68. Springer, 2019.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang. and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09053 2020.
Brahma Pavse, Ishan Durugkar, Josiah Hanna, and Peter Stone. Reducing sampling error in batch temporal difference learning. In International Conference on Machine Learning, pp. 7543-7552. PMLR, 2020.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintage-weiggtted regression. Simple and scalable off- soficpllcy reinforcement learning. arXiv preprint arXiv:1910.00017,, 2019.
Vitchyr Pong, Ashvin Nair, Laura Smith, Catherine Huang, and Sergey Levine Offline meta- reinforcement learning with online self-supervision. arXiv preprint wrXi::2107.03974,0 2021.
Martin Riedmiller. Neural fitted Q iteration experiences with a data efficient neural reinforcement learning method. In European Conference on Machine Learning, pp. 317-328. Springer, 2005.
Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-RL: Robust adversarial model-basee offline reinforcement learning. arXiv preprint arXiv:2204. 12581, 2022.
Albrecht Schmidt, Fosca Giannotti, Wendy Mackay, Ben Shneiderman, and Kaisa Väänänen. Artifi- cial intelligence for humankind: panel on how to create truly interactive and human-centeree AT for the benefit of individuals and society. In IFIP Conference Human- Interaction, pp. 335-339. Springer, 2021.
Sungyong Seo, Sercan Arik, Jinsung Yoon, Xiang Zhang, Kihyuk Sohn, and Tomas Pfister. Con- trolling neural networks with rule representations. Advances Neural Information Processing Systems, 34, 2021.
Ben Shneiderman. Comman-Computer Interaction, 12(3):199-124, 2020.
Human-centered artificial intelligence: three fresh ideas. AIS Transactions on
Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe. Roland Hafner, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint 2000:2002.00836 2020.
Jordi Smit, Canmanie Ponnambalam, Mathhijs TJ Spaan, and Frans Oliehoek. PEBL: Pessimistic ensembles for offline deep reinforcement learning. In Robust and Reliable Autonomy in the Wild Workshop at the 30th International Joint Conference of Artificial Intelligence, 2021.
Nathalic Smuha. Ethics guidelines for trustworthy AI. European Commission, Chectorate-Generat for Communications Networks, Content and Technology Publications Office, 2019.
Phillip Swazinna, Steffen Udluft, Daniel Hein, and Thomas Runkler. Behavior constraining in weight space for offline reinforcement learning. arXiv preprint arXiv: 107.05479 2021a.
12

---

Published as a conference paper at ICLR 2023
Phillip Swazinna, Steffen Udluft, and Thomas Runkler. Overcoming model bias for robust offline deep reinforcement learning. Engineering Applications of Artificial Intelligence, 104:104366, 2021b.
Phillip Swazinna, Steffen Udluft, Daniel Hein, and Thomas Runkler. Comparing model-free and model-based algorithms for offline reinforcement learning. arXiv preprint arXiv:2201.04333, 2022.
Shengpu Tang. Aditya Modi, Michael Sjoding, and Jenna Wiens. Sinician-inn-te-lloy decision making: Reinforcement learning with near-optimal set-valued policies. In International Conference on Machine Learning, pp. 9387-9396. PMLR, 2020.
Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement. In International Conference on Machine Learning, pp. 2380-2383 PMLR, 2015.
Núria Armengol Urpi, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. arXiv preprint arXiv:2102.05371, 2021.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. arXiv preprint wrXiv:2006.55133 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1361, 2019.
Yijun Yang, Jing Jiang, Tianyi Zhou, Jie Ma, and Yuhui Shi. Pareto policy pool for model-based offline reinforcement learning. International Conference Learning Representations, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, volume 33, pp. 14129-14142, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34, 2021.
Baohe Zhang, Raghu Rajan, Luis Pineda, Nathan Lambert, André Biedenkapp. Kurtland Chua, Frank Hutter, and Roberto Calandra. On the importance hyperparameter optimization for model-based reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 4015-4023. PMLR, 2021a.
Michael Zhang, Tom Le Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Ziyu Wang, and Mohammad Norouzi. Autoregressive dynamics models for offline policy evaluation and optimization. XrXiv preprint
2021b.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. International Conference on Machine Learning, pp. 11194- 11203. PMLR, 2020.
Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behavior cloning regularization for stable offline-to-olline reinforcement learning. 2021.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
A OTHER HYPERPARAMETERS
Similarly to other methods, we introduce a of hyperparameters in our method either explicitly or implicitly. Any hyperparameter that is involved in training the original policy model B& or the transition dynamics models {far} can obviously be tuned offline, by holding out validation set of transitions from the original dataset and evaluating the trained models them to test how well they predict dataset actions or future states and rewards. This is true for example for the architecture of the transition models, their learning rate, or the number of epochs to train.
When we train the policy in offline reinforcement learning however, the equivalent of an evaluation dataset does not exist, since cannot evaluate the policy the true environment dynamics. While efforts are being made to make hyperparameter tuning possible via offline policy evaluation, the problem is hard for obvious reasons and we cannot really rely on them yet. Especially the arguably most important hyperparameter, how closely the newly trained policy needs to follow the actions proposed by the original policy, cannot realistically be tuned since involves knowing when the evaluation method itself cannot be trusted any more. This is precisely why we propose the selection of this hyperparameter at runtime via user interaction.
Our method however introduces some hyperparameters that are used during policy training and that could thus be seen as critical hyperparameters that cannot be tuned: 17, controlling the penalty magnitude for not following the dataset actions at À O, and the parameters of the Beta distribution from which is sampled during training. We did not tune them, since we simply copied the parameters from (Seo et al. 2022 that À Beta 0.1) and also set 7) = 0.1 without any experiments. We would however like to point out, that these parameters are at least partially tunable in the offline setting, since they do not depend on the accuracy of the transition models: n controls how strongly one adheres to the actions of the dataset, and we can easily asses whether this working properly, by evaluating whether the final trained policy still adheres to the actions of the dataset at À = 0. We evaluated policies for different in Figure 7 (a) and show that there is not great difference between different 77, as long as not zero, so it seems reasonable to choose something small that is still doing the trick. Similarly, we show an evaluation for different parameters of the beta distribution in Fig. 7 (b). The parameters control how often which parts of the X-spectrum are seen during training (see Fig. [T). As can be seen, the more the distribution tends towards uniform distribution, the more the policy deviates from the original policy model at À = O, which is that
undesired. When more probability mass is focused on the edges of the distribution, the original policy model is reproduced more accurately. Both 77 and B thus be reasonably well tuned entirely offline, without using the transition models.
Impact Å
Beta Distribution Impact @ A=
0.40
Distance to Onaginal Policy Model
0.080
0.075
0.38
Data Distance
0.070
0.36
0.065
0.060
0.34
0.055
0.050
0.045
0.30
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
in Beta(x.x)
Figure 7: Some hyperparameters like n and the parameters of the Beta distribution can be adjusted entirely offline, since they do not require evaluation on the true transition dynamics.
14

---

Published as a conference paper at ICLR 2023
B COMPLETE POLICY VISUALIZATION IN 2D-ENV
n Fig. we chose only a set of interesting À values for visualization for page limit reasons. See Fig for more fine- fineggrined evaluation.
Figure 8: Visualization for more À values of the trained policy in the simple 2D Env than Fig.
C MODEL-FREE EXPERIMENTS
Model-free We motivate our purely model-based implementation with the idea that it should be simpler to implement compared to model-free policy training. When learning policy with value function, we have always two models that are trained jointly the policy and the Q-function interact with each other and both are essentially a moving target. While it should be possible derive similar algorithm LION in the model-free world, we were not quite able to do so: Figs. and TO show experiments in which we augment TD3+BC with our approach by sampling À from the same distribution in LION and providing it to the policy, only that now it controls the influence the value function has on the optimization of the policy and (1-X) controls the distance to the behavior actions. Interestingly however, the trained policy is unable to alter its behavior anywhere in the A-rang.. Instead it seems that the policy training is rather unstable and the policy collapses to a solution that behaves always the same, regardless of X. Similarly, the distance the original policy does not change with varying X. We chose TD3+BC due to its simplicity it exhibits fewer trained models and implementational pitfalls than others. The original Q-function and policy optimization objectives in TD3+BC are:
IB global 1.0
52
(sum) 4-m+-nc
-64
0.00
0.25
0.50
D.75
1.50
A-
Optpmiaattionn Parameter
Figure 9: Model-free periment on the 100% ex- ploration dataset.
arg min E' D [[Q(s, a) (r + (
Q
where Qt denotes the slow moving target Q-function used for computing the Q-targets. And:
arg max E. EDDD
a N Σ.
Q(s, (()) (T(s) a)²
our extension, denoted by X-TD3+BC, we provide the sampled À to the policy and use it to balance the two optimization terms, similarly to our LION approach. The new objectives are:
Q arg min E Es [iQ(s,,
-r + (s', (T,
113
15

---

Published as a conference paper at ICLR 2023
and:
arg max E BR
Q(s, (s,
(1
8,
a)²
E (Q (s,
We show complete experiments for all of the IB datasets in Figs. TO & 9 As already mentioned in Section C we are unable to observe that the policy is able to alter its behavior anywhere in the X-range We hypothesize that methods for stabilizing the Q-function are needed in order for the
policy to not collapse.
bed 0.0
bod 0.2
bod 0.4
bod 0.6
bod 0.8
-300
/. Marver
T-peeeentee Marcer Rovedee
2-1peee-dee Man Roveeee
-peeeeaaee Marcer
-103
-129
-221
.213
935
239 1.00
626 933
231
130
606 123
231
130
EG6 925
233 130
R00 033
235
1.0
aptmaad
optiniaed
optimized
optimbed
100
-20
-99
0.2
Figure 10: Model-free experiments incorporating the LION approach. The trained policies unable to alter their behavior for different inputs of A.
D SAMPLING A
As mentioned, we do not sample À uniformly since doing leads to less accurate learning at the edges of the X-spectrum. In Fig 11 we show visualizations of different symmetric Beta distributions from which we draw A together with the policy results on the bad-0.2 dataset. The Beta(1, 1) case corresponds to the uniform distribution and has a significant discrepancy at the À = 0 end of the range: The original policy (its performance) is not accurately reproduced. Generally, it seems the flatter the distribution becomes, the more are the two extreme cases moved together. We observe this phenomenon even though the policy has plenty of capacity and even in the simple 2D environment.
PDFs
perfomances
-202
other performena
-125
230
0013 12.5, 2.61
01.B (400)
Figure 11: Beta distributions and resulting per- formances on bad-0.2, Edge cases become more accurate with increased probability mass.
16

---

Published as a conference paper at ICLR 2023
E EXPERIMENTAL DETAILS
We conducted experiments on system with a Xeon Gold 5122 CPU (4 x 3.6 GHz, no GPU support used). A single policy training took about 45 minutes, a single dynamics model about 2 hours. We train dynamics models using 90/10 random data split and select the best models according to their validation performance. The final ensemble contains four models. The recurrent models for the industrial benchmark have an RNN cell with size 30 and an output layer mapping from the cell state the state space. We use G 30 history steps to build up the hidden state of the RNN and then predict F = 50 steps into the future. The feedforward models of the 2D env have two layers of size 20 & 10. We use adam (Kingma & Ba 2014) with standard parameters and an exponentially decaying learning rate with factor 0.99 and train for up to 3000 epochs. The models of the original policy have a single hidden layer of size 30 and the final policies have two layers of size 1024. We use ReLU nonlinearities throughout all experiments. During policy and original policy model training we use the entire dataset instead of just the 90% split from the dynamics training. We use discount factor of y 0.97 and perform rollouts of length H = 100 to train and evaluate the policy.
IB globall 1.0
50.0
-5.0
0.4
0.7
0.00
0.25
0.50
0.75
1.00
A-pptintzation Parameter
Figure 12: LION results as presented in Fig. for the 100% exploration dataset.
F HYPERPARAMETERS OF THE TRANSITION ENSEMBLE
Instead of using the minimum over the predicted rewards during rollouts, one might as well use other forms of aggregation, like the mean. In Fig. 13 we show results for such mean reward experiments the future state was then randomly selected among the predictions of the transition ensemble. The resulting performance curves look similar as those in 4 except that in the mediocre and the optimized setting, the performance decrease on the = end is even steeper than with the minimum reward prediction. This makes intuitively sense, since the mean is much easier exploited than the minimum.
b.
IB mediscre
optimized
-72
-130
-52.5
-150
-303
0.50
9.25
4- -pplmmaalonn Miriure Tunumcial
3. -p4imuuaiion Matiat Purmmii
2- Opplmmatoonn Maslare amsiil
COL
Figure 13: Results as in Fig. 4 for the mean reward rollout experiments.
Similar effects can be observed when the number of ensemble members is reduced. In Fig. 14 we show results on the same datasets when only a single model instead of an ensemble with four members is used (thus mean and min become the same). Here, the performance drops even earlier as well as stronger in the mediocre and optimized settings.
IB 0.5
18 medie 0.2
1B optimieed 0.4
-302
133-
-122
-202
-92.5
-130
-95.
02
0.5
0.2
0.5
0.5
04
3.6 02
04
0.6
3.
1,0
5.0
0.
0.5
26
04
1.0
Optimmattoonn Mirture Turumciar
Op.immaaaiionn Metuue Paramiiirr
A-
Opt0mmattonn Minture Feramiierr
Li0k(eenge)
B3ACh
SCAA
5C3
TD3+92
COL
NC00
Figure 14: Results as in Fig. 4 for the single model rollout experiments.
17

---

Published as a conference paper at ICLR 2023
G PRELIMINARIES ON OFFLINE RL
In offline reinforcement learning, we are trying to optimize the sum of discounted rewards, called return, Ex>rr, obtained by acting in the MDP M =< S, A, R, T, 7, 80 >, where S is the set of states, 80 C S the set of starting states, A the set of actions, R : S x A + I the reward function, T:
A + the transition dynamies function, and 7 E [O, 1) the discount factor. We train a policy To S + A to obtain an action when in state 8. in order to produce trajectories of length H: T ( dg; ro; 81, d1; . H aH, TH). While online RL methods are able to continuously explore their environment, collect new data and test new behaviors, offline RI. methods need to optimize the return by learning only from a fixed dataset of interactions D {Ti).
i = 1, ..., K. Often, this is done by solving the approximate MDP 'M =< S, A, R, TD. So >, where TD is directly estimated with transition model using the transitions from D. However, also model-free offline approaches exist, that do not estimate TD and directly try to optimize a policy with value function that is only trained D. Since neither model-free nor model-based variants can be sure to accurately assess the policy's 's performance in unknown parts of the state-action space, offline RL algorithms regularize their policy stay way 'close' to the policy that generated the data. The crucial question how close should To be to the original policy? is an open problem. We address the dilemma that the user has no meaningful way of choosing the correct proximity since offline policy selection / evaluation remain generally unsolved problems.
18


Reviews:
[{"review_id": "vzAZJkJcOI", "reviewer": "Reviewer_f6wy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The article describes a training strategy where one wants to obtain (from a dataset of episodes) a policy that is controllable by the user. The underlying assumption is to have the user in the loop at inference time (for some industrial reasons) instead of just outputting one single policy. Said otherwise, the authors want to put more weight on the human when the policy is deployed. To achieve this, they propose to proceed in two phases: in the first phase, they learn a behavioral cloning policy over the dataset, but also a world model to extract a simulator of the world. In a second step, they train a policy to maximize the reward on simulated trajectories but also add a regularization to enforce the new policy to potentially behave like the previously learned imitation policy (kind of distillation). What is interesting is that the weight of the regularization is considered as an output of the trained policy such that, as an output, one will obtain a continuum of policies, from pure (distilled) imitation policy to reward maximization ones. Experiments are made on a toy 2D problem, but also on some industrial benchmarks. ", "summary_of_the_review": "On my side, I don't really catch the interest of the proposed method and don't see it as a strong enough contribution. ", "strength_and_weaknesses": "First of all, I must say that I don't really capture the path from the idea to the execution of the idea. If the objective is to learn a continuum of policies between pure imitation and reward maximization ones, I would suggest the authors to study for instance a loss function that would weight the behavioral cloning loss together with a reward driven loss (like IQL for instance, or CQL). Right now, the authors are doing this through a simulator of the world (deterministic in their case) which is far from being able to model complex problems. If the weighted loss is defined over the dataset, then the method becomes very close to the TD3+BC approach, and the originality would be to take the hyper parameter as an input to define the continuum of policies. \n\nThe second critical point is about what we can do with the resulting model. So, as an output, we have a policy that we can control with lambda. How do we concretely control the value of lambda at inference time? Maybe, for some values of lambda, the resulting policy is very good, but we don't know it. It seems to me that the value of lambda will then be chosen by using some kind of A/B testing (am I right ?). But in that case, it would mean that we have collected more data, and we could thususe these new data to update our policy...  So I don't understand exactly how the model is concretely used at last. \n\nLast point, the experiments are made on not classical benchmarks. It is critical to achieve experiments on D4RL for instance to clearly compare with state of the art, particularly the fact that the authors are learning a world model in a very naive way which may have strong consequences on D4RL. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C15AfhUfH", "reviewer": "Reviewer_edim", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary: This paper proposes the problem of user-interactive offline RL. Specifically, a setting where the user can control the proximity of the learned policy to the behavior policy during deployment of the learned policy. The paper proposes a simple algorithm that provides this control, and compares it to standard offline RL and behavior cloning algorithms on a benchmark of industrial datasets. \n\n\nContributions:\n - Proposes the problem statement of offline RL that can be interactively tuned by a human user during deployment. While human-in-the-loop RL, online fine-tuning of offline RL policies, and meta-offline RL involve adapting the policy during deployment, I believe the specific aim to allow a human to control the interpolation between the behavior policy and a reward-optimized policy is novel. \nProposes a straightforward algorithm that builds on top of existing model-based offline RL algorithms.\nPerforms a thorough experimental evaluation on the industrial datasets benchmarks, comparing to a number of offline RL and BC algorithms.\n", "summary_of_the_review": "Overall, the problem statement is well-motivated and practical. I think there are many applications in which we would like to do better than the expert, but have varying tolerance for doing worse (per state and per user) - this is a practical algorithms that makes it easy to tune this trade-off during deployment, without re-training. The method is a straightfoward combination of existing components, and experimental results show it is effective (caveat: I'm not familiar with the industrial benchmark used for experiments). \n\n\n\n----- Update 12/9 ------\n\nI have read the author response and the other reviews, and I maintain my positive assessment of this paper. \n\n", "strength_and_weaknesses": "Questions and Comments:\n1) Why do you use the min of the ensemble instead of the expectation? This seems like a form of pessimism to avoid model exploitation, but if the environment is stochastic, the resulting policy would be biased. How important is this decision?\n2) What is the effect of the ensemble experimentally? Is this necessary for the method to work?\n3) In the middle row of Figure 4 (“mediocre” setting), why does performance decrease dramatically with increasing \\lambda? Is this a case of the learned model being exploited? I’m surprised this doesn’t happen as dramatically in the bottom row (“optimized” setting).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AVoNgtlVoN3", "reviewer": "Reviewer_Kz6Z", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new offline RL learning procedure (named LION) that, at inference time, is responsive to a trade-off hyperparameter that controls how much the new policy can deviate from observed behavior; past work typically treats this as a hyperparameter at training time. \n\nExperiments all in the offline setting, are conducted on a goal-finding task in 2D grid-world and an industrial control benchmark. Results show that the proposed approach can smoothly interpolate between imitating observed behavior and purely optimizing rewards, and that simple tuning strategies can help users find the optimal trade-off hyperparameter and also outperform existing offline RL methods. ", "summary_of_the_review": "This is a well written paper with simple yet novel proposed method, strong empirical results, and clear writing. ", "strength_and_weaknesses": "**Strengths:**\n- Problem is well motivated and well situated in the existing literature. \n- The proposed approach requires some change in perspective (on how the trade-off parameter is used) but is intuitive to understand and simple to implement. \n- Experiments on simulated problems and benchmarks are thoroughly executed with strong empirical results that support the main claims. \n\n**Weaknesses:**\n- Experiments are conducted on a industrial benchmark suite, could you further comment on the characteristics of the MDPs considered? e.g. deterministic vs stochastic transition, discrete or continuous states/actions? How much do these benchmark-specific characteristics affect the extent we can generalize the conclusions to other problems, for example medical domains and education (intelligent tutoring systems). \n\n**minor comment on a parenthetical note:** on page 3 the authors state \"practitioners likely favor deterministic policies over stochastic ones due to trust issues\". Some past work have discussed the benefit of using non-deterministic (not the same as stochastic) policies and how they can be incorporated into a human-in-the-loop decision support system, which can be seen as another form of online user-interactive adaptation. It would be interesting to include a discussion in related works. \n> - Fard & Pineau. \"Non-deterministic policies in MDPs\". JAIR 2011. https://doi.org/10.1613/jair.3175\n> - Tang et al. \"Clinician-in-the-Loop Decision Making: RL with Near-Optimal Set-Valued Policies\". ICML 2020. https://proceedings.mlr.press/v119/tang20c.html\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "1Xx1yBIfoV", "reviewer": "Reviewer_u1gw", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper takes a user-centered approach to the problem of the behavior of off-line reinforcement learning models when put into use. It takes an idea that is simple in concept and practical to apply that addresses some of the risks of using RL models that have been trained off-line. The idea is that users be given a choice of how aggressive or conservative a policy they follow by parameterizing the choice as a single parameter learnt at training time.  Learning this parameter has a clear computation solution, and the method can be demonstrated to generate a smooth transition from a given behavioral policy to a more aggressive one. ", "summary_of_the_review": "By considering a user-centric approach, this paper addresses an outstanding dilemma about implementing off-line RL policies in a way that is easily representable to the user and builds on current offline methods. ", "strength_and_weaknesses": "Given that the use of offline reinforcement learning, as is, remains problematic, this paper offers a both sound and practical way to put it to use. It's refreshing to see an idea with apparent merit demonstrated to the point where it can be implemented, And bringing the user back, to propose a user centric approach that touches on the \"representation\" theme of the conference is notable. \n\nI wish the authors had mentioned the computational effort needed for training.  Another thought is to be more specific about how choices are presented to the user -- do they chose only \\lambda, or do they also get to see the policy that their choice implies?  One might imagine that exposing the risk / reward tradeoff implicit in the choice, in terms of the reward distribution would be  useful. None of these wishes are flaws, but rather extensions should the authors wish to address them. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster