Challenges of Real-World Reinforcement Learning

Gabriel Dulac-Arnold 1 Daniel Mankowitz 2 Todd Hester 2

Abstract

Reinforcement learning (RL) has proven its worth
in a series of artificial domains, and is beginningto show some successes in real-world scenarios.
However, much of the research advances in RL
are often hard to leverage in real-world systems
due to a series of assumptions that are rarely satisfied in practice. We present a set of nine uniquechallenges that must be addressed to productionize RL to real world problems. For each of thesechallenges, we specify the exact meaning of thechallenge, present some approaches from the literature, and specify some metrics for evaluating thatchallenge. An approach that addresses all ninechallenges would be applicable to a large numberof real world problems. We also present an example domain that has been modified to present thesechallenges as a testbed for practical RL research.

1. Introduction
Reinforcement learning (RL) (Sutton & Barto, 2018) is a
powerful algorithmic paradigm encompassing a wide arrayof contemporary algorithmic approaches (Mnih et al., 2015;
Silver et al., 2016; Hafner et al., 2018). RL methods havebeen shown to be effective on a large set of simulated environments (Mnih et al., 2015; Silver et al., 2016; Lillicrap
et al., 2015; OpenAI), but uptake in real-world problems
has been much slower. We posit that this is due in large partto a too-wide divide between the casting of current experimental RL setups and the generally poorly defined realities
of real-world systems.
In this paper, we present the main challenges that make RL
in the real-world more difficult than RL in research. At ahigh-level these challenges are:1

Google Research, Brain Team 2 DeepMind. Correspondence
to: Gabriel Dulac-Arnold <dulacarnold@google.com>.
Real-world Sequential Decision Making workshop at ICML 2019.
Copyright 2019 by the author(s).

1. Training off-line from the fixed logs of an external
behavior policy.
2. Learning on the real system from limited samples.
3. High-dimensional continuous state and action spaces.
4. Safety constraints that should never or at least rarely
be violated.
5. Tasks that may be partially observable, alternatively
viewed as non-stationary or stochastic.
6. Reward functions that are unspecified, multi-objective,
or risk-sensitive.
7. System operators who desire explainable policies andactions.
8. Inference that must happen in real-time at the control
frequency of the system.
9. Large and/or unknown delays in the system actuators,
sensors, or rewards.
While there has been research focusing on these challenges
individually, there has been little research on algorithms
that address all of these challenges together. We hope thatthese challenges can guide researchers towards developing more applicable RL algorithms. For each challenge
above, we motivate its importance, specify it, present approaches for addressing the challenge, and provide methods
for evaluating that particular challenge. Finally, we illustratethese challenges on a toy-problem, a control suite (Tassaet al., 2018) task modified to present all of the challenges
described above.
We consider control systems grounded in the physical world,
optimization of software systems, and systems that interact
with users such as recommender systems and smart phones.
These systems can range in size from a small drone to a datacenter, in complexity from a one-dimensional thermostat
to a self-driving car, and in cost from a calculator to a spaceship. In all these scenarios there are recurring themes: thereis rarely a good simulator, the systems are stochastic & nonstationary, have strong safety constraints, and running themis expensive and/or slow. This is very different from training on a simulated environment where data is effectively
unlimited, consequences for poor actions are non-existentand system dynamics are clean and often deterministic.
Although current RL algorithms can learn superhuman policies for systems we can properly simulate (Silver et al.,2016; Mnih et al., 2015; OpenAI), for many real-world sys-

Challenges of Real-World Reinforcement Learningtems, not only is there no existing simulator, but building
one can be extremely difficult. Many interesting systems areeither too complex to model properly (datacenter coolingplants or deformable object manipulation tasks), or sufficiently varied (arbitrary object assembly with the samerobot arm) that modelling each instance would be impractical. This lack of available simulators means learning must
be done using data from the real system, and all acting andexploring must be done on the real system. Thus, we cannot
simply collect massive datasets to solve these challenges,
nor can we ignore safety during training.
To deploy RL to a real production system, robust evaluationis required. Many research papers in RL look at average
episodic return to evaluate the quality of their agent (although they train with discounted return). This makes sensewhen the only optimization criteria is the return itself, however in real systems there are other aspects of agent behavior
which are equivalently important. In many cases, it may be
important to evaluate performance for the worst case user,or the worst case object for manipulation, rather than the
average reward. For many real world applications, respecting safety can be much more important than maximizing
returns. Additionally, because the global reward function
is generally a balance of multiple sub-goals (e.g. reducing
both time-to-target and energy use), a proper evaluation
should explicitly separate the individual components of the
reward function to better understand the policy’s tradeoffs.

2. Practical Challenges
In this section we present a series of practical challenges
that appear when using RL on real world systems. Not all ofthese challenges will be present in every real system, but inmany cases all of the challenges are present to some degree.
To guide practitioners working on applications of RL, we
present current research directions from the literature foreach challenge. To guide researchers who wish to research
only a subset of challenges at a time, we present evaluation
criteria for that particular challenge. We believe that an RL
algorithm that addresses all of these challenges would beapplicable to a vast number of real world problems.
For this work, we assume the standard Markov Decision
Process (MDP) formulation. An MDP is represented by atuple hS, A, P, r, γi where S is the state space, A the action
space, P is the stochastic transition function p(s0 | s, a),and r(s, a) is the reward function which returns a reward
for given state-action. A tuple of experience is of theform (st , at , rt , st+1 ). The policy π produces an action
as at ∼ π(·|st ). Our notation describes a discrete state
and action space but generalizes to continuous ones. Later,
we consider modifications of this formalism for safety constraints, robustness, non-stationarity and partial observability. As there will likely be multiple iterations of the policy
through time we index them as πi to indicate the learning it-

eration. An existing system is often controlled by an existing
policy, either in the form of human operators or black-box
controllers; we call this a behavior policy and denote it πB .
2.1. Batch Off-line and Off-Policy Training
As mentioned above, many systems cannot be trained on directly and need to be learned from fixed logs of the system’s
behavior. In many cases, we are deploying an RL approachto replace a previous control system, and logs from thatpolicy are available. In future training iterations, batches ofdata will be available from the most recent iteration of thecontrol algorithm. This setup is an off-line and off-policy
training regime where the policy needs to be trained frombatches of data. We begin by proposing a basic frameworkand then discuss possible design choices.
We consider an ordered set of experience tuples producedby policy πB , DπB = [{s0 , a0 , r0 , . . . , sT , aT , rT }1≤i≤n ].
The initial policy π0 is trained from data from the previous
controller of the system, DπB . π0 generates data for Dπ0
which is used to train π1 and so-on 1 . The details of policy
training are left up to the implementer. The general framework is that of batched reinforcement learning (Scherrer
et al., 2012), which we recall in Algorithm 1 for clarity. We
note that the RL algorithm used to train the policy after eachbatch is not restricted; it could be policy iteration, valuefunction based, policy gradient, etc. (Sutton & Barto, 2018).
Algorithm 1 Batch RL Training
1: procedure BRT(DπB , Train, N, L)
2:
π0 ← Train(DπB )3:for i = 0, · · · , N do
4:
Dπi ← {}
5:for t = 0, · · · , L do
6:at ← πi (st )7:
Dπi ← Dπi ∪ (st , at , rt (st , at ))8:
st+1 ∼ P (· | st , at )9:
end for10:πi+1 ← Train(Dπi )11:
end for
12: end procedure

For a production system where drops in performance could
be very costly, we want to ensure that the new policy improves upon the previous policy. Estimating the policy’s
performance without running it on the real system is termed
off-policy evaluation (Precup et al., 2000). Off-policy evaluation becomes more challenging as the difference between
the policies and the resulting state distributions grows.
The simplest approach to off-policy evaluation is importance sampling (Precup et al., 2000), which accounts forthe difference between the behavior and target policies. Alternatively, the direct method learns a transition model and1
We consider that any mix of previous experience is acceptable
for training the policy, one does not have to limit training to Dπt−1 .

Challenges of Real-World Reinforcement Learning

uses that for evaluation. Doubly-robust estimators (Dudı́k
et al., 2011; Jiang & Li, 2015) combine both and get thebest evaluations from both worlds. There are many more
approaches such as MAGIC (Thomas & Brunskill, 2016) ormore robust doubly robust (Farajtabar et al., 2018) that canbe considered as well.
The performance of the initial policy π0 often dictates
whether access to a system will be granted by system owners as there is usually a minimum performance thresholdthe system must respect. Therefore, an important quantity
to evaluate for a new learning algorithm is the warm-start
performance given the behavior policy’s data:
J start = R(Train(DπB )),(1)

where R is the cumulative return from the policy
Train(DπB ). Evaluating the training algorithm’s performance on different sizes of DπB , as well as for different quality behavior policies can help understand the algorithm’s ability at finding a starting policy even in the faceof sub-optimal, over-fit, or insufficient data.
2.2. Learning On the Real System from Limited
Samples
Unlike much of the research performed in deep reinforcement learning (Mnih et al., 2015; Hester et al., 2018), realsystems do not have separate training and evaluation environments. All training data comes from the real system, andthe agent cannot have a separate exploration policy during
training as its exploratory actions do not come for free. Instead, the agent must perform reasonably well and act safely
throughout learning. For many systems, this means that
exploration must be limited, and the resulting data is lowvariance – very little of the state space may be covered in
the logs. In addition, since there is often only one instanceof the system, approaches that instantiate hundreds or thousands of environments to collect more data for distributed
training are usually not compatible with this setup (Horgan
et al., 2018; Espeholt et al., 2018; Adamski et al., 2018).
Almost all of these real-world systems are either slowmoving, fragile, or expensive enough that the data they
produce is costly, and policy learning must be data-efficient.
In the case where there are off-line logs of the system, thesemight not contain anywhere near the amount of data or data
coverage that current RL algorithms expect. Learning iterations on a real system can take a long time, as slower
control frequencies might range from 1-hour to multi-monthtimesteps, and reward horizons could be on the order ofmonths (e.g. online advertisement, drug therapies). Evenin the case of higher-frequency control tasks, the learningalgorithm needs to learn quickly from potential mistakes
without needing to repeat them multiple times before fixing
them. Thus, learning on a real system requires an algorithm
to be both sample-efficient and performant.

There are a number of related works that deal with RL
on real systems and, in particular, focus on sample efficiency. One such work is Model Agnostic Meta-Learning(MAML) (Finn et al., 2017) that focuses on learning abouttasks within a distribution and then, with few shot learning, quickly adapting to solving a new in-distribution taskthat it has not seen previously. Bootstrap DQN (Osband
et al., 2016) learns an ensemble of Q-networks and uses
Thompson Sampling to drive exploration and improve sample efficiency, and PILCO (Deisenroth & Rasmussen, 2011)
uses Gaussian processes to efficiently model a system andtrain a policy.
Another approach to improving sample efficiency is touse expert demonstrations to bootstrap the agent, rather
than learning from scratch. This approach has beencombined with DQN (Mnih et al., 2015) and demonstrated on Atari (Hester et al., 2018), as well as combinedwith DDPG (Lillicrap et al., 2015) for insertion tasks on
robots (Vecerı́k et al.). Recent Model-based deep RL approaches (Hafner et al., 2018; Chua et al., 2018), where thealgorithm plans against a learned transition model of theenvironment, show a lot of promise for improving sample
efficiency. A common approach is to learn ensembles oftransition models and use various sampling strategies fromthose models to drive exploration and improve sample efficiency (Hester & Stone, 2013; Chua et al., 2018; Buckman
et al., 2018).
To evaluate the data efficiency of a particular method, asimple yet useful measure is to look at the amount of datanecessary to achieve a certain performance threshold:J ef f. = min |Di | s.t. R(Train(Di )) > Rmin ,(2)

where Rmin is the desired performance threshold. Similar
approaches have been used in model-based RL literature
to demonstrate efficiency (Hafner et al., 2018; Chua et al.,2018).
2.3. High-Dimensional Continuous State and Action
SpacesMany practical real world problems have large and continuous state and action spaces. For example, consider the hugeaction spaces in recommender systems (Covington et al.,2016), or the number of sensors and actuators to control
cooling in a Google data center (Evans & Gao; Evans et al.,2018). These large state and action spaces can present serious issues for traditional RL algorithms, as identified in(Dulac-Arnold et al., 2015).
There are a number of recent works focused on addressing
this challenge. Dulac-Arnold et al. present an approachbased on generating a vector for a candidate action andthen doing nearest neighbor search to find the closest realaction available. Zahavy et al. propose an Action Elimination Deep Q Network (AE-DQN) that uses a contextual

Challenges of Real-World Reinforcement Learning

bandit to eliminate irrelevant actions. He et al. present
the Deep Reinforcement Relevance Network (DRRN) for
evaluating continuous action spaces in text-based games.
Finally, Chandak et al. proposes a method to learn action
embedding according to their effects on state transitions.
Evaluating a policy in the face of large action spaces should
consider both the number of actions and the quality of themetric over the action space: well-organized action spaces
are easier to reason with than smaller but poorly ordered
spaces. Even if embeddings are learned, the inherent relationship between actions should be varied when evaluating.
2.4. Satisfying Safety Constraints
Almost all physical systems can destroy or degrade themselves and their environment. As such, considering thesesystems’ safety is fundamentally necessary to controllingthem. Safety is important during system operation, but also
during exploratory learning phases as well. These could besafety considerations either of the system itself (limiting system temperatures, contact forces or maintaining minimum
battery levels) or of its environment (avoiding dynamic obstacles, limiting end effector velocities). There may exist
a fallback watchdog controller, which would take over if
the learned policy violates the safety constraints, but weconsider that it should not be explicitly relied upon.
Recent work in RL safety (Dalal et al., 2018; Achiam
et al., 2017) has cast safety in the context of Constrained MDPs (CMDPs) (Altman, 1999), and we willconcentrate on pre-defined constraints on the environment in this context. Constrained MDPs define a constrained optimization problem and can be expressed as:maxπ∈Π R(π) subject to C k (π) ≤ Vk , k = 1, . . . , K.
Here, R is the cumulative reward of a policy π for a given
MDP, and C k (π) describes the incurred cumulative costof a certain policy π relative to constraint k. The CMDP
framework describes multiple ways to consider cumulative
cost of a policy π: the total cost until task completion, thediscounted cost, or the average cost. Specific constraints aredefined as ck (s, a).
The CMDP setup allows for arbitrary constraints on state
and action to be expressed. In the context of a physicalsystem these can be as simple as box constraints on a specificstate variable, or more complex such as dynamic collision
avoidance constraints. One major challenge with addressingthese safety concerns in real systems is that safety violations
will likely be very rare in logs of the system. In many cases,
safety constraints are assumed and are not even specified by
the system operator or product manager.
An alternative to CMDPs is budgeted MDPs (Boutilier &Lu, 2016; Carrara et al., 2018). While for a CMDP, the constraint level Vk is given, for budgeted MDPs, it is unknown.
Instead, the policy is learned as a function of constraint level.

Then the user can examine the trade-offs between expected
return and constraint level and choose the constraint levelthat best works for the data. This more closely matches thecommon real-world scenario where the constraints may not
be absolute, but small violations may be allowed for a large
improvement in expected returns.
Recently, there has a been a lot of work focused on the
problem of safety in reinforcement learning. One focus has
been the addition of a safety layer to the network (Dalalet al., 2018; Pham et al., 2017). This type of approach hasenabled an agent to learn a task with zero safety violations
as well as transfer some problems to the real world. These
approaches focus on safety during training. There are otherapproaches (Achiam et al., 2017; Tessler et al., 2018; Bohezet al., 2019) that learn a policy that violates constraints
during training but produce a trained policy that respects
the safety constraints. Thomas (2015) presents the notion of‘safe RL’ where the algorithm searches for a new, improved
policy and ensures that the probability of a ‘bad policy’
being proposed is low. In addition, this work computes
a high confidence lower bound on the performance of anevaluation policy.
Other RL approaches include using Lyapunov functions
to learn safe policies (Chow et al., 2018) and safe exploration strategies that predict the safety of neighboringstates (Turchetta et al., 2016; Wachi et al., 2018). A Probabilistic Goal MDP (Mankowitz et al., 2016) is another typeof objective that encourages an agent to consider whether
it should do something risky for large reward or be moreconservative for smaller rewards within a pre-defined periodof time.
To evaluate the safety of an RL algorithm, we consider
counting the number of safety violations for each individual
constraint. Accumulating all these violations into a single
number has been proposed previously (Dalal et al., 2018)
and provides a good global summary. We propose also
maintaining each individual constraint’s count of violations:!
TX
saf etyJ(π) =cj (si , ai )∈ RK , (3)
i=1

1≤j≤K

where K is the number of safety constraints in the CMDP.
Visualizing this vector’s evolution during both training andonce running on the environment is essential to understanding the policy’s behavior relative to the provided safety
constraints.
2.5. Partial Observability and Non-Stationarity
Almost all real systems where we would want to deploy reinforcement learning are partially observable. For example,
on a physical system, we likely do not have observationsof the wear and tear on motors or joints, or the amount ofbuildup in pipes or vents. On systems that interact with

Challenges of Real-World Reinforcement Learning

users such as recommender systems, we have no observations of the mental state of the users. Often times, thesepartial observabilities appear as non-stationarity (e.g. as
a pump’s efficiency degrades) or as stochasticity (e.g. as
each robot being operated behaves differently). Partiallyobservable problems are typically formulated as a partiallyobservable Markov Decision Process (POMDP). The key
difference from the MDP formulation is that the agent’s
observation x ∈ X is now separate from the state, withan observation function O(x | s) giving the probability ofobserving x given the environment state s.
There are two common approaches to handling partial observability in the literature. First is to incorporate history
into the observation of the agent. DQN (Mnih et al., 2015)
stacks four Atari frames together as the agent’s observationto account for partial observability. An alternate approachis to use recurrent networks within the agent, enabling themto track and recover hidden state. Hausknecht & Stone apply such an approach to DQN, and show that the recurrent
version can perform equally well in Atari games when only
given a single frame as input. Nagabandi et al. propose
an approach modeling the system as non-stationary with atime-varying reward function, and use meta-learning to findpolicies that will adapt to this non-stationarity.
Much of the recent work on transferring learned policies
from simulation to real system also focuses on this area,
as the underlying differences between the systems are notobservable (Andrychowicz et al., 2018; Peng et al., 2018).
Real world systems are often stochastic and noisy comparedto most simulated environments. In addition, sensor andaction noise as well as action delays add to the perturbations
an agent may experience in the real-world setting. There
are a number of RL approaches that have been utilized toensure that an agent is robust to different subsets of these factors. We will focus on the Robust MDP formalism, domain
randomization and system identification as frameworks for
dealing with noisy, non-stationary systems.
A Robust MDP is defined by a tuple hS, A, P, r, γi where
S, A, r and γ are as previously defined; P is a set of transition matrices referred to as the uncertainty set (Iyengar,
2005). The objective that we optimize is the worst casevalue function defined as:
X∞
J worst (π) = inf Ep
γ t rt |P, π .
p∈P

t=0

At each step, nature chooses a transition function that theagent transitions with so as to minimize the long term value.
The agent learns a policy that maximizes this worst casevalue function. Recently, a number of works have surfacedthat have shown this formulation to yield robust policies thatare agnostic to a range of perturbations in the environment(Tamar et al., 2014; Mankowitz et al., 2018; Shashua &Mannor, 2017). The solutions do tend to be overly conserva-

tive but some work has been done to yield less conservative,‘soft-robust’ solutions (Derman et al., 2018).
In addition to the robust MDP formalism, the practitioner
may be interested in both robustness due to domain randomization and system identification. Domain randomization (Peng et al., 2018) involves explicitly training an agent
on various perturbations of the environment and averaging these learning errors together during training. System
identification involves training a policy that can determine
online the environment in which it is operating and modify
the policy accordingly (Finn et al., 2017; Nagabandi et al.,2018).
We can train a policy to be robust, but the question arises
as to how we can evaluate the performance of such a policy.
Robustness to noisy measurements (possibly both sensor
and action noise), as well as action delays, can be evaluated by executing the policy in K test environments thatexhibits these types of perturbations. Comparing the average test performance of the policy across the K perturbed
test environments can provide a notion of robustness. That

 T
is,
1 X p X
robustr(si , ai ) ,
(4)
J
(π) =
E
K
i=1
p∈P

where p is a perturbed test environment within the test set P.
This has previously been applied in a number of different
works (Mankowitz et al., 2018; Di Castro et al., 2012; Derman et al., 2018). Evaluating the performance of a policy
on a stream of constantly changing environment perturbations and its ability to adapt online to these perturbations
provides another notion of robustness. In each of thesecases, for a given task, experiment designers must decide
which dimensions of the perturbations are pertinent and lookat the effects of variations in these dimensions on policy
performance.
2.6. Unspecified and Multi-Objective Reward
Functions
Reinforcement learning frames policy learning through thelens of optimizing a global reward function, yet most systems have multi-dimensional costs to be minimized. In
many cases, system or product owners do not have a clear
picture of what they want to optimize. When an agent istrained to optimize one metric, other metrics are discoveredthat also need to be maintained or improved. Thus, a lot of
the work on deploying RL to real systems is in formulating
the reward function, which may be multi-dimensional. Because the global reward function is generally a balance ofmultiple sub-goals (e.g. reducing both time-to-target andenergy use), a proper evaluation should explicitly separate
the individual components of the reward function to better
understand the policy’s tradeoffs.
In addition, it may be desired that the policy performs well

Challenges of Real-World Reinforcement Learning

for all task instances and not just in expectation. For example, an algorithm deployed to a factory needs to workon every robot, not just on average, and a recommendersystem must work for every user, not just on average. Therefore, policy quality cannot be summarized by a single scalar
describing cumulative reward, but must consider both multiple dimensions of the policy’s behavior and the full distribution of behaviors both during training and testing. A
typical approach to evaluate the full distribution of reward
across groups is to use a Conditional Value at Risk (CVaR)
objective (Tamar et al., 2015b), which looks at a given
percentile of the reward distribution, rather than expected
reward. Tamar et al. show that by optimizing reward percentiles, the agent is able to improve upon its worst-caseperformance. Distributional DQN (Dabney et al., 2018;
Bellemare et al., 2017) explicitly models the distribution
over returns, and it would be straight-forward to extend it touse a CVaR objective.
There are a number of works devoted to recovering an underlying reward function from demonstrations, such as inverse reinforcement learning (Russell, 1998; Ng et al., 2000;
Abbeel & Ng, 2004; Ross et al., 2011). Hadfield-Menell
et al. examine how to infer the truly intended reward function from the given reward function and training MDPs, toensure that the agent performs as intended in new scenarios.
For problems with multi-objective reward functions, there
are approaches to learning the pareto-optimal reward function (Roijers et al., 2013), but none of these have been scaled
to the deep reinforcement learning setting yet. Van Seijen
et al. present an approach that takes advantage of multiobjective reward signals to learn super-human performancein the Atari game Mc-PacMan.
We propose a simple multi-objective analysis of return.
If we consider that the global reward function is defined as a linear combination of sub-rewards, r(s, a) =PK
j=1 αj rj (s, a), then we can consider the vector of percomponent rewards for evaluation:!
Tn
X
J multi (π) =rj (si , ai )∈ RK . (5)
i=1

1≤j≤K

When dealing with multi-objective reward functions, it isimportant to track the different objectives individually whenevaluating a policy. This way, problem stakeholders canunderstand the different tradeoffs the policy is making andchoose which compromises they consider best.
To evaluate the performance of the algorithm across thefull distribution of scenarios (e.g. users, tasks, robots, objects,etc.), we independently analyze the performance of thealgorithm on each cohort. This is also important for ensuring
fairness of an algorithm when interacting with populationsof users. Another approach is to analyze the CVaR return
rather than expected returns (Tamar et al., 2015b). One eval-uation procedure is to determine whether rare catastrophic
rewards are minimized (Tamar et al., 2015b;a). Another evaluation procedure is to observe behavioural changes whenan agent needs to be risk-averse or risk-seeking such as infootball (Mankowitz et al., 2016).
2.7. Explainability
Another essential aspect of real systems is that they are
owned and operated by humans, who need to be reassured
about the controllers’ intentions and require insights regarding failure cases. For this reason, policy explainability isimportant for real-world policies. Especially in cases where
the policy might find an alternative and unexpected approachto controlling a system, understanding the longer-term intent
of the policy is important for obtaining stakeholder buy-in.
In the event of policy errors, being able to understand theerror’s origins a posteriori is essential.
Verma et al. define their policies using a domain-specificprogramming language, and then use a local search algorithm to distill a learned neural network policy into an explicit program. Additionally, the domain-specific language
is verifiable, which allows the learned policies to be verifiably correct. There are many methods to attempt to elicit
the intent of deep neural networks (Montavon et al., 2018)
which could also be used to understand a learned policy.
Additionally, model-based methods with explicit rollouts
used for planning (Hafner et al., 2018; Chua et al., 2018) can
provide insights on what the policy’s ‘intent’ may have been.
In the case of (Verma et al., 2018), evaluation is centered on
looking at the performance of the programatically definedpolicy, since the policy is inherently defined as a humanreadable program. Evaluating actual explainability comes
down to evaluating how well a human understands the intent of the policy’s expression, which can be done throughA/B experiments with users on mechanical turk (PoursabziSangdeh et al., 2018).
2.8. Real-Time Inference
To deploy RL to a production system, policy inference must
be done in real-time at the control frequency of the system.
This may be on the order of milliseconds for a recommendersystem (Covington et al., 2016) responding to a user requestor the control of a physical robot, and up to the order ofminutes for building control systems (Evans & Gao). This
constraint both limits us from running the task faster thanreal-time to generate massive amounts of data quickly (Silver et al., 2016; Espeholt et al., 2018) and limits us fromrunning slower than real-time to perform more computationally expensive approaches (e.g. some forms of model-basedplanning).
There has been literature focused on this problem in the caseof robotics. Hester et al. present a real-time architecture
to do model-based RL on a physical robot that uses MonteChallenges of Real-World Reinforcement Learning

Carlo Tree Search and returns actions when required by thetask, but will improve given more time and more rollouts.
Other rollout-based approaches like AlphaGo (Silver et al.,2016) will improve with more rollouts, but are not engineered to run at a specific frequency. Wawrzyński analyzes
the potential gains to be had by allowing more computation
time than is possible on a true real-time system.
2.9. System Delays
Finally, most real systems have delays in either the sensationof the state, the actuators, or the reward feedback. For example, Hester & Stone focus on controlling a robot vehicle
with significant delays in the control of the braking system.
They incorporate recent history into the state of the agent sothat the learning algorithm can learn the delay effects itself.
Mann et al. look at delays in recommender systems, where
the true reward is based on the user’s interaction with therecommended item, which may take weeks to determine.
They present a factored learning approach that is able totake advantage of intermediate reward signals to improvelearning in these delayed tasks.
Hung et al. introduce a method to better assign rewards
that arrive significantly after a causative event. They use
a memory-based agent, and leverage the memory retrieval
system to properly allocate credit to distant past events thatare useful in predicting the value function in the current
timestep. They show that this mechanism is able to solve
previously unsolvable delayed reward tasks. Arjona-Medina
et al. introduce the RUDDER algorithm, which uses a
backwards-view of a task to generate a return-equivalent
MDP where the delayed rewards are re-distributed more
evenly throughout time. This return-equivalent MDP iseasier to learn and is guaranteed to have the same optimal
policy as the original MDP. They improvements using thisapproach in Atari tasks with long delays.
Evaluation in this context depends on the source of delay.
Delays in the state & action domain can be patched onto
existing environments, and the correlation between delay
magnitude and the policy’s cumulative reward can be used
to evaluate the policy’s robustness to these perturbations.

3. Example Environment
In this section, we present an example environment from the
DeepMind control suite (Tassa et al., 2018), and the modifications required to make it present all of the challenges
for real world RL that we have presented in this paper2 . We
also describe how to perform specific evaluations for eachchallenge where it makes sense. Our goal is both that thisenvironment and its modifications drive research in realworld RL as well as help evaluate candidate algorithms’applicability to real-world problems.
2

A more complete task description, as well as three other proposed tasks are presented in the Appendix.

Humanoid Variables: θ, u, F
TypeConstraint
Limit joint angles.
StaticEnforce uprightness.
Avoid dynamic obstacles.
Kinematic Limit joint velocities.
Limit foot contact forces.
Dynamic
Encourage falls on pelvis.
Table 1. Safety constraints for the Humanoid environment.

We start from the control suite’s humanoid task, as it alreadyhas a high-dimensional continuous state and action space.
Next, we present a set of modifications to the task to makeit present all nine challenges.
First, the training must be done in the batch RL regime aspresented in Sec. 1. To imitate the situation where we aretaking over control from a system with an existing controller,
we recommend first training a policy on the task with adifferent algorithm than the one being evaluated, and thengenerating a dataset from this policy to serve as the initialD(πb ). As the size, quality, and coverage of this dataset canbe critical, we suggest evaluating algorithms with a variety
of datasets of different sizes, converged performances, andpolicy entropies.
Second, we are constrained to act on-line in the domain,
meaning every training run needs to be considered as if it
is on the actual system, thus incurring safety and efficiency
penalties. For the third challenge, the humanoid domain
already presents a high-dimensional continuous state andaction space. Table 1 proposes safety constraints for thehumanoid task. In the case of physical systems, we cangenerally consider that safety constraints involve one or
many of static, kinematic and dynamic aspects of the system.
One can modify the simulation to have non-stationarityand partial observability by sampling a new set of domain
parameters (frictions, CoGs, masses) from a distribution
every episode. This can either be done during both training
and evaluation, or only during evaluation depending on
the envisioned scenario. For multi-objective reward, we
consider rewards for distance moved as well as for energy
used in movement. We also consider worse-case reward foreach sampled episode; we want the agent to perform well inevery perturbation, not just on average.
For explainability, there is not a modification to the domain, but evaluation must be performed qualititatively withhumans determining the quality of the explainability (e.g.
perhaps with mechanical turk (Poursabzi-Sangdeh et al.,2018)). We require that real-time inference be enforced andthat the physics simulation be run no slower than real-time.

Challenges of Real-World Reinforcement Learning

Finally, we can introduce artificial delays on the system
actuators, storing the requested actions in an n-step queue
before passing them to the simulator.
Metric (Challenges)
Average ReturnWarm-Start (1)Time-to-Rmin (2)Safety Violations (4)Robust Performance (5)CVaR Return (6)Multi-Objective (6)Explainability (7)Definition
Equation (1)Equation (2)Equation (3)Equation (4)(Tamar et al., 2015b)
Equation (5)User Evals

Table 2. The full set of evaluation metrics to consider to address
all the challenges of real world RL, along with which challenges
they are relevant for.

In addition to evaluating the average rewards achieved during the trials, we can also look at specific metrics to evaluate
individual challenges (Table 2). We look at the warm-start
performance after the training on the behavior policy data
(Eq. 1). For sample efficiency, we examine the number ofdata samples required to reach a minimum performancelevel (Eq. 2). We also track the number of safety violations for every safety constraint (Eq. 3). We evaluate theworst-case reward over sampled perturbations (Eq. 2.5). We
independently evaluate the reward on both objectives (Eq. 5).
Finally, we want to qualitatively evaluate the explainabilityof the agent’s policies. Only by looking at all these evaluators can we truly get an idea of an algorithm’s aptitude forreal-world use. Our goal is for this task to present a testbed
for other researchers who wish to develop new algorithms
that address the challenges of real world RL.

4. Related Work
While we covered related work specific to each challenge inthe sections above, there are a few other works related to ourgoal of practical reinforcement learning. Hester & Stone
similarly present a list of challenges for real world RL, but
specifically for RL on robots. They present four challenges(sample efficiency, high-dimensional continuous state andaction spaces, sensor and actuator delays, and real-time inference), all of which we include in our set of challenges aswell. They do not include our other challenges such as robust performance, learning from fixed logs, non-stationarity,
safety constraints, etc. Their approach is to setup a real-timearchitecture for model-based learning where ensembles ofmodels are learned to improve robustness and sample efficiency.
Henderson et al. investigate the effects of existing degrees
of variability between various RL setups and their effects onalgorithm performance. Although restricted to the domainof existing environments, Henderson et al. propose morerobust performance estimators for RL learning algorithms

which complement those presented here. The paper ends
with the question ”In what setting would [a given algorithm]
be useful?” to which we try to contribute by proposing a
specific setting in which well-adapted work should hopefully stand out. Wagstaff argue similarly to us regarding
supervised ML, mentioning the too-strong lack of real-world
applications in ML conferences and the subsequent impact
on research directions this can have. The Horizon platform (Gauci et al., 2018) and Decision Service (Agarwal
et al., 2016) provide software platforms for training, evaluation and deployment of RL agents in real-world systems.
In the case of Decision Service, transition probabilities arelogged to help make off-policy evaluation easier down theline, and both systems consider different approaches to offpolicy evaluation. We believe well-structured frameworks
such as these are crucial to productionizing RL systems.
Finally, Irpan discusses a complementary series of difficulties of getting RL systems working on real systems, andoffer interesting directions for future research such as taking
into consideration domain-specific priors, or densifying the
reward signal.

5. Conclusion
We argue that real-world problems in RL contain multiple
challenges which are not often considered in the current
literature. We have presented nine such challenges for practical RL. For each one, we have motivated the challenge,
presented examples, and specified the challenge more formally. For practitioners wishing to deploy RL to their own
problems, we have pointed the reader to relevant references
for each challenge that may guide them in deploying RL
to a production system. For RL researchers, we have presented an example environment and evaluation criteria withwhich to measure progress on these challenges. There havebeen many recent works on each of these challenges individually, but little work on approaches that address all ninechallenges.
There are a few themes that appear across the related works
for each challenge. While model-based RL seems especiallypromising to address the issue of sample efficiency, it canalso help address off-policy evaluation, robustness, and explainability. Ensembles are useful both to improve sample
efficiency and robustness. One very important componentof deploying RL to any real product is interacting withthe product owner or other human experts. We must work
closely with the experts to formulate the right reward objectives and safety constraints, to get expert demonstrations towarm-start learning, and we must explain the algorithm’s
actions enough that they have enough confidence in thesystem to deploy it. These themes should point the way
towards research that can address all these challenges andbe deployed on more real world systems and products.

Challenges of Real-World Reinforcement Learning

References
Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse
reinforcement learning. In Proceedings of the twenty-firstinternational conference on Machine learning, pp. 1.
ACM, 2004.
Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained
policy optimization. CoRR, abs/1705.10528, 2017. URL
http://arxiv.org/abs/1705.10528.
Adamski, I., Adamski, R., Grel, T., Jedrych, A., Kaczmarek,
K., and Michalewski, H. Distributed deep reinforcement
learning: Learn how to play atari games in 21 minutes.
In International Conference on High Performance Computing, pp. 370–388. Springer, 2018.
Agarwal, A., Bird, S., Cozowicz, M., Hoang, L., Langford,
J., Lee, S., Li, J., Melamed, D., Oshri, G., Ribas, O.,et al. Making contextual decisions with low technicaldebt. arXiv preprint arXiv:1606.03966, 2016.
Altman, E. Constrained Markov decision processes, volume 7. CRC Press, 1999.
Andrychowicz, M., Baker, B., Chociej, M., Jozefowicz,
R., McGrew, B., Pachocki, J., Petron, A., Plappert, M.,
Powell, G., Ray, A., et al. Learning dexterous in-handmanipulation. arXiv preprint arXiv:1808.00177, 2018.
Arjona-Medina, J. A., Gillhofer, M., Widrich, M., Unterthiner, T., and Hochreiter, S. Rudder: Return
decomposition for delayed rewards. arXiv preprint
arXiv:1806.07857, 2018.
Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. CoRR,
abs/1707.06887, 2017. URL http://arxiv.org/
abs/1707.06887.
Bohez, S., Abdolmaleki, A., Neunert, M., Buchli, J., Heess,N., and Hadsell, R. Value constrained model-free continuous control. arXiv preprint arXiv:1902.04623, 2019.
Boutilier, C. and Lu, T. Budget allocation using weakly coupled, constrained markov decision processes. In Proceedings of the 32nd Conference on Uncertainty in Artificial
Intelligence (UAI-16), pp. 52–61, New York, NY, 2016.
Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H.
Sample-efficient reinforcement learning with stochastic
ensemble value expansion. CoRR, abs/1807.01675, 2018.
Carrara, N., Laroche, R., Bouraoui, J., Urvoy, T., Olivier, T.
D. S., and Pietquin, O. A fitted-q algorithm for budgeted
mdps. In EWRL 2018, 2018.

Chandak, Y., Theocharous, G., Kostas, J., Jordan, S., andThomas, P. S. Learning action representations for reinforcement learning. arXiv preprint arXiv:1902.00183,
2019.
Chow, Y., Nachum, O., Duenez-Guzman, E., andGhavamzadeh, M. A lyapunov-based approach to safe
reinforcement learning. In Bengio, S., Wallach, H.,
Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31, pp. 8092–8101. 2018.
Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep
reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754–4765, 2018.
Covington, P., Adams, J., and Sargin, E. Deep neural networks for youtube recommendations. In Proceedings ofthe 10th ACM conference on recommender systems, pp.
191–198. ACM, 2016.
Dabney, W., Ostrovski, G., Silver, D., and Munos,
R. Implicit quantile networks for distributional reinforcement learning.
In Dy, J. and Krause, A.
(eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1096–1105,
Stockholmsmssan, Stockholm Sweden, 10–15 Jul 2018.
PMLR. URL http://proceedings.mlr.press/
v80/dabney18a.html.
Dalal, G., Dvijotham, K., Vecerik, M., Hester, T., Paduraru,C., and Tassa, Y. Safe exploration in continuous action
spaces. CoRR, abs/1801.08757, 2018. URL http://
arxiv.org/abs/1801.08757.
Deisenroth, M. and Rasmussen, C. E. Pilco: A model-based
and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machinelearning (ICML-11), pp. 465–472, 2011.
Derman, E., Mankowitz, D. J., Mann, T. A., and Mannor,
S. Soft-robust actor-critic policy-gradient. arXiv preprint
arXiv:1803.04848, 2018.
Di Castro, D., Tamar, A., and Mannor, S. Policy gradients with variance related risk criteria. arXiv preprint
arXiv:1206.6404, 2012.
Dudı́k, M., Langford, J., and Li, L. Doubly robust policy
evaluation and learning. arXiv preprint arXiv:1103.4601,
2011.
Dulac-Arnold, G., Evans, R., van Hasselt, H., Sunehag, P.,
Lillicrap, T., Hunt, J., Mann, T., Weber, T., Degris, T., andCoppin, B. Deep reinforcement learning in large discrete
action spaces. arXiv preprint arXiv:1512.07679, 2015.

Challenges of Real-World Reinforcement Learning

Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih,
V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., and Kavukcuoglu, K. IMPALA:
Scalable distributed deep-RL with importance weighted
actor-learner architectures. In Dy, J. and Krause, A.
(eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1407–1416,
Stockholmsmssan, Stockholm Sweden, 10–15 Jul 2018.
PMLR. URL http://proceedings.mlr.press/
v80/espeholt18a.html.
Evans, R. and Gao, J. Deepmind ai reduces google data centre cooling bill by 40%. https://deepmind.com/
blog/deepmind-ai-reduces-google-datacentre-cooling-bill-40/.
Evans, R. A., Gao, J., Ryan, M. C., Dulac-Arnold, G.,
Scholz, J. K., and Hester, T. A. Optimizing data center controls using neural networks, July 19 2018. US
Patent App. 15/410,547.
Farajtabar, M., Chow, Y., and Ghavamzadeh, M. Morerobust doubly robust off-policy evaluation. CoRR,
abs/1802.03493, 2018. URL http://arxiv.org/
abs/1802.03493.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic metalearning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine
Learning-Volume 70, pp. 1126–1135. JMLR. org, 2017.
Gauci, J., Conti, E., Liang, Y., Virochsiri, K., He, Y., Kaden,Z., Narayanan, V., and Ye, X. Horizon: Facebook’s open
source applied reinforcement learning platform. arXiv
preprint arXiv:1811.00260, 2018.
Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. J.,
and Dragan, A. D. Inverse reward design. CoRR,
abs/1711.02827, 2017.
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D.,
Lee, H., and Davidson, J. Learning latent dynamics forplanning from pixels. arXiv preprint arXiv:1811.04551,
2018.
Hausknecht, M. J. and Stone, P. Deep recurrent q-learningfor partially observable mdps. CoRR, abs/1507.06527,
2015.

Hester, T. and Stone, P. TEXPLORE: Real-time sampleefficient reinforcement learning for robots. Machine
Learning, 90(3), 2013. doi: 10.1007/s10994-012-53227. URL http://dx.doi.org/10.1007/s10994012-5322-7.
Hester, T., Quinlan, M., and Stone, P. RTMBA: A realtime model-based reinforcement learning architecture
for robot control. In IEEE International Conference on
Robotics and Automation (ICRA), May 2012.
Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul,T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., Dulac-Arnold, G., Agapiou, J., Leibo, J. Z., andGruslys, A. Deep q-learning from demonstrations. In
Proceedings of the Thirty-Second AAAI Conference on
Artificial Intelligence, (AAAI-18), pp. 3223–3230, 2018.
URL https://www.aaai.org/ocs/index.php/
AAAI/AAAI18/paper/view/16976.
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel,M., van Hasselt, H., and Silver, D. Distributed prioritizedexperience replay. CoRR, abs/1803.00933, 2018. URL
http://arxiv.org/abs/1803.00933.
Hung, C.-C., Lillicrap, T., Abramson, J., Wu, Y., Mirza,
M., Carnevale, F., Ahuja, A., and Wayne, G. Optimizing
agent behavior over long time scales by transporting value.
arXiv preprint arXiv:1810.06721, 2018.
Irpan, A. Deep reinforcement learning doesn’t work yet.
https://www.alexirpan.com/2018/02/14/
rl-hard.html, 2018.
Iyengar, G. N. Robust dynamic programming. Mathematics
of Operations Research, 30(2):257–280, 2005.
Jiang, N. and Li, L. Doubly robust off-policy value
evaluation for reinforcement learning. arXiv preprint
arXiv:1511.03722, 2015.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez,
T., Tassa, Y., Silver, D., and Wierstra, D. Continuous
control with deep reinforcement learning. arXiv preprint
arXiv:1509.02971, 2015.
Mankowitz, D. J., Tamar, A., and Mannor, S. Situational awareness by risk-conscious skills. arXiv preprint
arXiv:1610.02847, 2016.

He, J., Chen, J., He, X., Gao, J., Li, L., Deng, L., andOstendorf, M. Deep reinforcement learning with a natural
language action space. arXiv preprint arXiv:1511.04636,
2015.

Mankowitz, D. J., Mann, T. A., Bacon, P.-L., Precup, D.,
and Mannor, S. Learning robust options. In Thirty-Second
AAAI Conference on Artificial Intelligence, 2018.

Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup,
D., and Meger, D. Deep reinforcement learning thatmatters. In Thirty-Second AAAI Conference on Artificial
Intelligence, 2018.

Mann, T. A., Gowal, S., Jiang, R., Hu, H., Lakshminarayanan, B., and György, A. Learning from delayed outcomes with intermediate observations. CoRR,
abs/1807.09387, 2018.

Challenges of Real-World Reinforcement LearningMnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness,
J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control
through deep reinforcement learning. Nature, 518(7540):
529, 2015.

Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online
learning. In Proceedings of the fourteenth international
conference on artificial intelligence and statistics, pp.
627–635, 2011.

Montavon, G., Samek, W., and Müller, K.-R. Methods
for interpreting and understanding deep neural networks.
Digital Signal Processing, 73:1–15, 2018.

Russell, S. J. Learning agents for uncertain environments.
In COLT, volume 98, pp. 101–103, 1998.

Nagabandi, A., Finn, C., and Levine, S. Deep online learning via meta-learning: Continual adaptation for modelbased RL. CoRR, abs/1812.07671, 2018.
Ng, A. Y., Russell, S. J., et al. Algorithms for inverse
reinforcement learning. In Icml, volume 1, pp. 2, 2000.
OpenAI. Openai five. https://blog.openai.com/
openai-five/.
Osband, I., Blundell, C., Pritzel, A., and Van Roy,B.
Deep exploration via bootstrapped dqn.
In
Lee, D. D., Sugiyama, M., Luxburg, U. V.,
Guyon, I., and Garnett, R. (eds.), Advances in
Neural Information Processing Systems 29, pp.
4026–4034. Curran Associates, Inc., 2016.
URL
http://papers.nips.cc/paper/6501-deepexploration-via-bootstrapped-dqn.pdf.
Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbeel,
P. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1–8. IEEE,
2018.
Pham, T., Magistris, G. D., and Tachibana, R. Optlayer practical constrained optimization for deep reinforcement
learning in the real world. CoRR, abs/1709.07643, 2017.
URL http://arxiv.org/abs/1709.07643.
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M.,
Vaughan, J. W., and Wallach, H. M. Manipulating andmeasuring model interpretability. CoRR, abs/1802.07810,
2018.
Precup, D., Sutton, R. S., and Singh, S. P. Eligibility traces
for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning,
ICML ’00, pp. 759–766, San Francisco, CA, USA, 2000.
Morgan Kaufmann Publishers Inc. ISBN 1-55860-707-2.
URL http://dl.acm.org/citation.cfm?id=
645529.658134.
Roijers, D. M., Vamplew, P., Whiteson, S., and Dazeley, R.
A survey of multi-objective sequential decision-making.
Journal of Artificial Intelligence Research, 48:67–113,
2013.

Scherrer, B., Gabillon, V., Ghavamzadeh, M., and Geist, M.
Approximate modified policy iteration. arXiv preprint
arXiv:1205.3054, 2012.
Shashua, S. D.-C. and Mannor, S. Deep robust kalman filter.
arXiv preprint arXiv:1703.02310, 2017.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L.,
Van Den Driessche, G., Schrittwieser, J., Antonoglou, I.,
Panneershelvam, V., Lanctot, M., et al. Mastering thegame of go with deep neural networks and tree search.
nature, 529(7587):484, 2016.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An
introduction. MIT press, 2018.
Tamar, A., Mannor, S., and Xu, H. Scaling up robust mdps
using function approximation. In International Conference on Machine Learning, pp. 181–189, 2014.
Tamar, A., Chow, Y., Ghavamzadeh, M., and Mannor, S.
Policy gradient for coherent risk measures. In Advances in
Neural Information Processing Systems, pp. 1468–1476,
2015a.
Tamar, A., Glassner, Y., and Mannor, S. Optimizing thecvar via sampling. In Twenty-Ninth AAAI Conference on
Artificial Intelligence, 2015b.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D.
d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq,
A., et al. Deepmind control suite. arXiv preprint
arXiv:1801.00690, 2018.
Tessler, C., Mankowitz, D. J., and Mannor, S. Reward constrained policy optimization. arXiv preprint
arXiv:1805.11074, 2018.
Thomas, P. and Brunskill, E. Data-efficient off-policy policy
evaluation for reinforcement learning. In International
Conference on Machine Learning, pp. 2139–2148, 2016.
Thomas, P. S. Safe reinforcement learning. 2015.
Turchetta, M., Berkenkamp, F., and Krause, A. Safe exploration in finite markov decision processes with gaussian processes. CoRR, abs/1606.04753, 2016. URL
http://arxiv.org/abs/1606.04753.

Challenges of Real-World Reinforcement Learning

Van Seijen, H., Fatemi, M., Romoff, J., Laroche, R., Barnes,
T., and Tsang, J. Hybrid reward architecture for reinforcement learning. In Guyon, I., Luxburg, U. V., Bengio,S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30, pp. 5392–5402. Curran Associates, Inc.,
2017. URL http://papers.nips.cc/paper/
7123 - hybrid - reward - architecture - for reinforcement-learning.pdf.
Vecerı́k, M., Sushkov, O., Barker, D., Rothörl, T., Hester, T.,
and Scholz, J. A practical approach to insertion with variable socket position using deep reinforcement learning.
Verma, A., Murali, V., Singh, R., Kohli, P., and Chaudhuri,
S. Programmatically interpretable reinforcement learning.
arXiv preprint arXiv:1804.02477, 2018.
Wachi, A., Sui, Y., Yue, Y., and Ono, M. Safe explorationand optimization of constrained mdps using gaussian
processes. In AAAI, pp. 6548–6556. AAAI Press, 2018.
Wagstaff, K. Machine learning that matters. arXiv preprint
arXiv:1206.4656, 2012.
Wawrzyński, P. Real-time reinforcement learning by sequential actor-critics and experience replay. Neural Netw.,
22(10):1484–1497, December 2009. ISSN 0893-6080.
doi: 10.1016/j.neunet.2009.05.011. URL http://dx.doi.org/10.1016/j.neunet.2009.05.011.
Zahavy, T., Haroush, M., Merlis, N., Mankowitz, D. J., andMannor, S. Learn what not to learn: Action elimination
with deep reinforcement learning. In Advances in Neural
Information Processing Systems, pp. 3562–3573, 2018.

Challenges of Real-World Reinforcement Learning

Appendix

Additional Environments

Cart-Pole Variables: x, θTypeConstraint
Limit range:
Staticxl < x < xr
Limit velocity near goal:
Kinematic|θc − θ| > θL ∨ θ̇ < θ̇V
Limit cart acceleration:
Dynamicẍ < Amax

We extend four example environments from the DeepMind
Control Suite (Tassa et al., 2018) with both safety and nonstationarity contraints (which can be considered independently or together) to illustrate the challenges proposed inthis paper. Our goal is both that these environments drive
research in real-world RL as well as help evaluate candidate algorithms’ applicability to real-world problems. Future work involves developing an open-source benchmark
around these safety constraints.

Walker Variables: θ, u, F
TypeConstraint
Limit joint angles:
StaticθL < θ < θ U
Enforce uprightness:
0 < uxLimit joint velocities:
Kinematic
maxi θ̇i < Lθ̇

We chose the Cart-Pole, Quadruped, Reacher, and Humanoid tasks for their increasing difficulty and closeness toreal-world control systems.

Dynamic

Limit foot contact forces:
Ffoot < Fmax

Manipulator Variables: θ, F , M
TypeConstraint
Limit joint angles
StaticθL < θ < θ U
Avoid dynamic obstacles
M ∩ MO,i = ∅Avoid self-contact
M∩M=M
Kinematic Limit joint velocities:
maxi θ̇i < Lθ̇Dynamic

In the case of physical systems, we can generally consider
that safety constraints involve one or many of static, kinematic and dynamic aspects of the system. We present safety
constraints of these environments in Table 3.
Robustness
Env.
Cart-Pole
WalkerManipulator
Humanoid

Noise
Actuator and sensor delays
Noisy perceptionof terrain
Imprecise proprioception
Reduced torque on
leg actuator

Non-Stationarity
Track friction increasing with time
Occasionally nonresponsive leg actuator
Changes in gripper
friction
Varying payload
CoGs

Acceleration Limits:maxi θ̈i < Lθ̈Limit end effector forces:
FEE < Fmax

Humanoid Variables: θ, u, F
TypeConstraint
Limit joint angles:
Static
θL,i < θi < θU,i
Enforce uprightness:
0 < uxKinematic Limit joint velocities:
maxi θ̇i < Lθ̇Dynamic

SafetyLimit foot contact forces.
FFoot < Fmax
Encourage falls on posterior
Fi < Fmax,1 ∀i ∈ C \ ipost
Fpost < Fmax,2Table 3. Safety-constrained control environments.

Table 4. Possible perturbations to environments to illustrate noise& non-stationarity.

Table 4 presents some ideas of specific noise and nonstationarity perturbations for each of the proposed environments. The possibilities are effectively endless, and certain
choices could make for impossible to learn, so well-designed
environments will be important for useful evaluations.
Evaluators
We resume the proposed evaluators in table 5.

Challenges of Real-World Reinforcement Learning

Challenge

Evaluator

Off-line

J start = R(Train(DπB ))Efficient
Safe

J ef f. = min |Di | s.t. R(Train(Di )) > RminPTJ saf ety (π) =∈ RK
i=1 cj (si , ai )Robust



P
p PT
1
E
r(s
,
a
)
J robust (π) = K
i
i
p∈P
i=1

Discerning

J multi (π) =

1≤j≤K

P
Tn

i=1 rj (si , ai )Table 5. Proposed evaluators.


1≤j≤K

∈ RK