Abstract: Deep Reinforcement Learning (DRL) is an avenue of research in Artificial Intelligence (AI) that has received increasing attention within the research community in recent years, and is beginning to show potential for real-world application. DRL is one of the most promising routes towards developing more autonomous AI systems that interact with and take actions in complex real-world environments, and can more flexibly solve a range of problems for which we may not be able to precisely specify a correct 'answer'. This could have substantial implications for people's lives: for example by speeding up automation in various sectors, changing the nature and potential harms of online influence, or introducing new safety risks in physical infrastructure. In this paper, we review recent progress in DRL, discuss how this may introduce novel and pressing issues for society, ethics, and governance, and highlight important avenues for future research to better understand DRL's societal implications. 1. Readers already familiar with DRL could skip straight to Section 3.

Jess Whittlestone
Kai Arulkumaran
Matthew Crosby

The Societal Implications of Deep Reinforcement Learning

Introduction Artificial intelligence (AI) is already having an impact on many areas of society  (Whittaker et al., 2018; Whittlestone et al., 2019) , and further advances in AI research are likely to precipitate much greater impacts. There is a growing community of researchers and practitioners doing excellent work to identify and address ethical and societal issues raised by current applications of AI  (Schiff et al., 2020) . Given the pace of AI progress, it is crucial that this community also think ahead about what challenges might be raised by future advances, and how forms of governance, norms, and standards being developed around AI today can best prepare us for the future evolution of AI systems. Of course, predicting the future of any technology with certainty is difficult. One way to strike a balance between thinking ahead while still being grounded in reality is to ask the question: what might be the impacts on society if current successful trends in AI research continue and result in new and widespread societal applications? This paper explores the potential societal implications of one avenue of AI research currently showing promise: Deep Reinforcement Learning (DRL). The paper is primarily aimed at researchers and policy practitioners working on the ethical, societal and governance implications of AI, but may also be of interest to AI researchers concerned with the impacts of ©2021 AI Access Foundation. All rights reserved. their own work. Our discussion aims to provide important context and a clear starting point for the AI ethics and governance community to begin considering the societal implications of DRL in more depth. DRL has received increasing attention in recent years, leading to some high-profile successes, particularly in board and video games  (Mnih et al., 2015; Silver et al., 2016 Silver et al., , 2017 Berner et al., 2019; Vinyals et al., 2019) . DRL combines Reinforcement Learning (RL), an approach based on learning through interaction with an environment, with the potential to result in highly autonomous systems  (Sutton & Barto, 2018) , with Deep Learning (DL), an increasingly popular method that enables AI systems to scale to more complex problems and environments  (Goodfellow et al., 2016) . Several researchers have suggested that DRL is likely to be a key component of more general-purpose and autonomous AI systems in the future  (Arel, 2012; Chen & Liu, 2018; Popova et al., 2018) . Though there are still considerable barriers to large-scale real-world deployment of DRL systems, we are beginning to see potential for application in areas including robotics  (Ibarz et al., 2021) , online personalisation and targeting  (Zhao et al., 2019) , finance  (Fischer, 2018) , autonomous driving  (Tai et al., 2016) , healthcare  (Esteva et al., 2019) , and data centre cooling  (Gasparik et al., 2018) . These applications and others will likely become more widespread as the technology improves. This could have substantial implications for people's lives: for example, by speeding up automation in various sectors, changing the nature and potential harms of online influence, or introducing new safety risks in physical infrastructure. We therefore suggest that now is an ideal time to begin thinking about how more widespread application of DRL-based technologies might impact society, what ethical questions might arise or be made more pressing as a result, how progress on different areas of technical research will affect these ethical challenges, and what forms of governance might be required to mitigate any risks. One challenge for exploring the potential impacts of future AI progress is that doing so will often require considerable background knowledge about current AI techniques and research directions. Such understanding is often beyond the core expertise of AI ethics and governance scholars, and there is a lack of accessible resources. We therefore begin this paper by providing a broadly accessible introduction to DRL methods with links to further, more in-depth resources for the interested reader.  1  Following this background, we outline in Section 3 the ways in which plausible future applications of DRL may introduce or exacerbate pressing issues for society, ethics and governance, including by: raising questions for current approaches to human oversight; introducing new challenges for safety and reliability; and changing incentives for data collection. We then discuss how progress on different technical challenges might shape the landscape of these societal concerns: how might breakthroughs in different areas magnify or mitigate societal impacts and risks, or change the direction of the field such that currently underresearched risks become more important? Understanding and monitoring progress on the technical research areas crucial to widespread application of DRL will be key to successful and timely management of future societal impacts, and will require interdisciplinary collaboration between experts in AI, social science, and policy. Based on this analysis, we highlight Artificial Intelligence includes any approach that attempts to make machines perform intelligently. Machine Learning involves getting machines to learn to solve problems from, usually large amounts of, data.
 Deep Reinforcement Learning Reinforcement Learning DL models data with many layered (deep) neural networks. An RL agent learns from a reward signal using trial and error. some important avenues for future work and conclude with recommendations for researchers in the AI ethics and governance community.
 Deep Learning
 Combines
 Deep Reinforcement Learning: a Brief Overview DRL is an approach to AI falling within the subfield of machine learning (ML), which combines two popular approaches in current ML research: reinforcement learning (RL) and deep learning (DL). 2 In general, the purpose of ML is to find solutions to problems that we cannot directly write programs for. This involves using (often large amounts of) data to optimise the parameters of a mathematical model, which will then encode a solution  (Bishop, 2006) . This is a powerful approach to solving many different types of problems. For example, while we could easily write a computer program to emulate a medical questionnaire (which is essentially a flowchart), we cannot similarly distill the knowledge of a doctor to diagnose a patient on the basis of a CT (computerised tomography) scan. However, given a large amount of data from CT scans and corresponding expert diagnoses, we can use ML to train a computer program to imitate doctors on this task  (Kononenko, 2001; Fatima et al., 2017) . However, many of the problems that we have to solve in the real world require going beyond making predictions based on labelled inputs: we may also want to interact with our environment through sets of actions, where the choice of action depends on what goals we have. Part of how humans learn is through this type of trial and error: when we take an 2. For an excellent and accessible overview of ML also aimed at those interested in its implications, we encourage reading "Machine Learning for Policy Makers"  (Buchanan & Miller, 2017) . For more in-depth, technical overviews of DRL and its components, DL and RL, we refer readers to the numerous excellent resources already published  (Goodfellow et al., 2016; Arulkumaran et al., 2017; Sutton & Barto, 2018; François-Lavet et al., 2018; Kaelbling et al., 1996) .
 Agent learns from previous actions, states, and rewards
 Environment
 Its state determines the observations and reward
 Actions chosen by agent's policy
 Observations and Rewards Figure  2 : An agent interacts with its environment through its actions, and learns in response to receiving new observations and rewards. action or make a choice, we get feedback, which gives us some indication of whether we should take similar actions in the future. For example, a young child quickly learns that when they say or do certain things, adults give them positive attention, leading them to repeat those same behaviours for reward more often. Reinforcement learning (RL), the core of DRL, is based on this approach to problem-solving and learning: an RL agent 3 learns through interaction with an environment, and, through trial-and-error, can alter its own behaviour in response to the feedback it receives. More precisely, the goal of RL is to learn a policy that recommends the best action to take at any given moment in order to maximise total reward over time. The reward received at any time point is given by a reward function, which specifies how good or bad the agent's actions are (but not necessarily exactly what they should be) given the current state of the world. RL takes place in a feedback loop, incorporating perception, action, and learning, as illustrated in Figure  2 . An RL agent first perceives the current state of its environment, performs an action in response, and then observes the next state as well as a reward.  4  This loop continues until a predetermined terminal condition is reached, or can run on indefinitely. A good policy should give the agent an appropriate action to take in every possible state of the environment, including ones the agent has never seen before. For example, when AlphaGo  (Silver et al., 2016)  beat former world Go champion Lee Sedol, it encountered board positions that had never occurred during its training period, and still played moves at least as good as any human expert. The first generation of RL approaches learned polices in the form of large tables, which indicated what action to take in any given state. However, many interesting modern applications of RL have far too many possible states or actions for this tabular approach to be practical or effective. This has led to decades of work on using more complex functions to approximate policies, culminating in recent years in the use of deep learning for learning more complex policies. Deep learning (DL) involves using many-layered models-most commonly artificial neural networks-to represent complex, non-linear functions. 5 A neural network accepts an input (usually a large matrix of numerical values), and, through the repeated application of mathematical operations in each layer, will produce an output (another matrix of numerical values). The trainable parameters of the network, which can number in the billions  (Brown et al., 2020) , can be updated by a learning algorithm so that, over time, the network ends up producing the desired outputs. Key to the success of DL  (LeCun et al., 2015)  is its ability to learn high-level features or representations from raw sensory data (e.g., images, text, or audio), which can be used to find patterns in high-dimensional data and generalise to previously unseen inputs. DRL involves training a neural network using RL, where the network learns which actions to take given its sensory inputs and feedback in the form of rewards.  6  The RL algorithm specifies how to update the parameters of the neural network so that it outputs actions leading to higher rewards. DL has enabled RL to scale to previously intractable problems by improving the performance of RL algorithms in two key ways: through representation learning, allowing them to learn directly in complicated environments, and function approximation, enabling policies with complex inputs and/or outputs  (Arulkumaran et al., 2017) . Because neural networks can approximate any function, they can be used to encode the policy of an agent. This allows RL to extend to problems with state or action spaces that are too large for the tabular approach-which includes most interesting modern applications. Deep networks can also learn features directly from raw sensory data, which means that they can, at least theoretically, be applied in many environments, including the real worldwith appropriate sensors. Without deep networks, the inputs to a RL algorithm generally have to be pre-processed and provided by a human expert. By contrast, DeepMind's DQN algorithm-the work that kickstarted the field of DRL-was able to achieve human-level performance on some Atari 2600 games from the game screen and the score, but no additional domain knowledge about any of the games  (Mnih et al., 2015) . An important extension of RL is multi-agent RL (MARL), in which multiple agents learn to compete or co-operate in an environment  (Busoniu et al., 2008) . While some agents may be deployed in isolation, there are many scenarios where groups of agents will need to interact with each other. Some parts of MARL research also look at mixed human-AI interaction  (Bard et al., 2020) , where understanding the intent of other agents is especially important. As a subfield of RL, MARL inherits all of its technical issues, but also introduces extra emergent complexity. While a full treatment of the intersection of DRL and MARL is 5. DL can involve other types of deep models  (Damianou & Lawrence, 2013) , but is effectively synonymous with the use of (artificial) neural networks. While artificial neural networks were historically inspired by biological neurons found in the brain, they are far more simplistic, and it is more informative to think of artificial and biological neural networks as independent, loosely similar, systems. 6. Technically speaking the network may output a 'value' for each state and then a further rule will determine the action performed. out-of-scope for this work, it is worth noting that agents can have different, even conflicting reward functions, and hence a single 'optimal' solution does not exist in the general case, even in theory. Another implicit assumption in most ML research is that ML models are trained on exactly the same distribution of data that they will be tested on. In the context of DRL, an example would be training a robotic arm to pick and place a certain type of object in a given place in a factory line, 'freezing' the learned model, and then deploying it in the exact same scenario. However, such a model would most likely be unable to adapt to changes in the type of object or placement along the line. To do so, one would also have to incorporate continual learning  (Parisi et al., 2019) , where the deployed model could continue to learn and adapt to changes in its environment. While trained DRL agents could be deployed without continual learning capabilities, they would be limited to strictly regulated, static environments. As this condition is extremely limiting, we assume that most impactful future applications of DRL will need to be capable of some form of continual learning. In general, current DRL methods work best on problems that have: (a) a well-defined environment, where it is possible to provide observations of the environment as input, specify what sorts of actions can be taken, and model time as passing in discrete steps; (b) a clear reward function, which is appropriate to the task, provides enough information to learn from, and cannot be easily 'gamed'  (Amodei et al., 2016; Krakovna, 2018; Lehman et al., 2020) ; (c) plenty of data to train the deep neural network-in the context of RL, this means it must be able to interact with and gain feedback from the environment easily and quickly; and (d) lots of computing power available, as exemplified by the high profile successes in DRL  (Silver et al., 2016 (Silver et al., , 2017 Berner et al., 2019; Vinyals et al., 2019) . The latter requirement is exacerbated by the fact that ML algorithms also have 'hyperparameters' 7 that need to be tuned separately, requiring the training of many models to find the settings that work best. In reality, it is rare that all of these ideal conditions are met, which is currently preventing the widespread deployment of DRL systems. However, successful applications of DRL are already apparent, even in these early stages of technological development. These initial applications are already raising issues for society, ethics, and governance which will only become more important as progress is made. We give an overview of these issues in the next section, in particular noting how they relate to current and emerging applications of DRL.
 Challenges DRL Raises for Society, Ethics, and Governance Areas where DRL has begun to see practical application include: to send more personallyrelevant notifications on Facebook  (Gauci et al., 2018) ; to control Google's data centre cooling system, improving energy efficiency  (Gasparik et al., 2018) ; to automate aspects of electronic trading, such as in J.P. Morgan's LOXM  (Bacoyannis et al., 2018) ; and in robotics and manufacturing  (Ackerman, 2020) . In the future, we may see more substantial use of DRL in a variety of areas from clinical decision support to resource management, and online education to autonomous driving  (François-Lavet et al., 2018) . These potential applications of DRL could have many benefits. More flexible robotic systems could improve manufacturing and delivery processes beyond their current limitations, more adaptive recommender systems could better take account of individuals' personal 7. Hyperparameters are set before learning begins; 'standard' parameters are adjusted during learning. preferences and longer-term consequences, and improving the efficiency of city infrastructure could help preserve vital and scarce resources. The promise of these benefits ensures that we will see more resources invested in realising these applications in the future. At the same time, the possibility of more autonomous systems that operate flexibly across a range of real-world domains introduces and magnifies a number of problems for AI ethics and governance, which we discuss in this section.
 Human Oversight The Ethics Guidelines for Trustworthy AI report published in 2019 by the European Commission's High Level Expert Group on AI (High-level expert group on Artificial Intelligence, 2019) states that in order to support human autonomy and decision-making, AI systems must allow for human oversight. Most applications of AI in society today support existing, identifiable decisions taken by individuals and groups, such as doctors, stockbrokers, and lawyers  (Parson et al., 2019) . This makes human oversight a relatively straightforward matter of ensuring that there is always a human 'in-the-loop', i.e., the ability for a human to oversee the overall activity of the system and intervene in every decision made. Applications of DRL, which aim to increase the autonomy of machines, may pose a challenge for this approach to human oversight. DRL could potentially be used in ways that go beyond supplementing or even replacing identifiable decisions, towards taking over control of much larger-scale processes, systems and networks, such as those involved in resource management in cities  (Mohammadi et al., 2017) . What adequate 'human oversight' should look like in these contexts, where a system may be making hundreds of small decisions in a very short period of time, is unclear. The data flow that a system is acting upon may be incomprehensible to humans, or simply too large and fast-moving for meaningful oversight to be maintained. For example, a DRL system designed to monitor and adjust energy usage in a building may be constantly making so many small decisions that it is impractical for a human to review and alter decisions even after the fact, and the system may be sufficiently complex and opaque that it is not possible to identify mistakes and intervene in real-time. It may be possible to impose constraints while the system is being designed-ensuring that the system turns off and a human is alerted if levels go above or beyond a certain threshold, for example. However, there remains a question of whether this constitutes sufficient oversight-or whether in this or similar situations we should avoid deploying DRL systems until ways to maintain more substantive oversight are found. Another challenge for oversight is introduced by the fact that deployed DRL systems may continue to learn and adapt their behaviour once embedded in their environment, possibly at a pace that is challenging for humans to keep track of and therefore meaningfully oversee. The question of how to maintain human oversight over systems that are continually learning is not one that has been addressed in existing ethics and governance proposals, and will become more pressing as continual learning systems become more advanced and practical. Similarly, if MARL becomes more prevalent, it would make understanding and intervening at the level of a single agent more difficult, posing additional challenges for oversight. In response to the direction of AI research, the European Commission's more recent whitepaper on AI states that "the appropriate type and degree of human oversight may vary from one case to another"  (European Commission, 2020) . Even though it may not be possible for a human to explicitly approve every decision, it may be possible to retain meaningful oversight by ensuring a human is able to review decisions after they have been made, by monitoring an AI system in operation and intervening in real-time if necessary, or by imposing operational constraints in the design phase. Further work on flexible approaches to human oversight should take into account possible evolutions in the autonomy and behaviour of DRL systems discussed here. As systems become more autonomous and promise more high-impact applications, there will inevitably be economic pressures to trust systems to make more decisions and take over more processes, rather than waiting for, or relying on, slow human input. The benefits of doing so need to be traded off against the potential costs to human autonomy that come from reduced oversight, and we must begin thinking about how to navigate this tradeoff now before pressures to implement more autonomous systems grow.
 Safety and Reliability DRL agents learn by exploring their environment to discover the actions that lead to the highest reward over time. This trial-and-error approach to learning means that in order to learn how to avoid making a mistake, a DRL agent would have to first make that mistake  (Berger-Tal et al., 2014) . In many real-world contexts this is unacceptable: we cannot have self-driving cars running over pedestrians, or an energy control system accidentally switching off electricity in a hospital, before learning not to do these things. This can cause problems even when DRL is not being deployed in a physical environment: Microsoft's chatbot Tay 8 reproduced thousands of offensive tweets before the problem was noticed and it was taken down. Deploying DRL systems in the real world therefore requires new ways of ensuring the safety and reliability of systems, and in particular requires approaches to safe exploration  (Pecka & Svoboda, 2014; Garcıa & Fernández, 2015) : how can agents explore enough during training to learn robust behaviour, while avoiding the kinds of exploration that could cause harm? Ensuring the safety and reliability of DRL systems becomes even more pressing if they are deployed with increasing autonomy in high-stakes domains and systems. For example, it has been suggested that DRL may be able to make effective use of the data generated by 'smart cities' to improve resource management, such as by optimising power usage, increasing the efficiency of traffic signal control systems, and increasing crop productivity by monitoring and correcting parameters in agriculture systems  (Mohammadi et al., 2017; Mocanu et al., 2018; Genders & Razavi, 2016) . Systems such as energy, transportation, and agriculture are high-stakes in the sense that failures could result in loss of life, and incredibly complex, meaning that their dynamics cannot be fully understood by any single person or group. It is reasonable to suppose that DRL systems, with their adaptability and ability to process huge amounts of data, may ultimately be better equipped to manage these important processes than humans, and improving efficiency could help save critical resources across the world. However, the importance of these domains also means that the risks involved in deploying 8. Although the type of algorithm behind Tay is unknown, this is still a prime example of an agent that interacts with an environment and learns from feedback. DRL systems are huge, massively raising the stakes of mistakes and unintended consequences compared to smaller-scale and more constrained applications. Continual monitoring of DRL systems will be essential to ensuring their safety in highstakes domains, especially if those systems are trained in the real world and continue to learn from a constant stream of data after deployment. Even for systems that are 'frozen' before deployment (i.e., no longer continuing to learn), as initial deployments of DRL likely will be, it is widely accepted that we currently lack adequate methods for verifying safety and reliability  (Tarraf et al., 2019; Allen, 2020; Fiander & Blackwood, 2016) . For systems that learn over their lifetime, continual monitoring to ensure they do not learn behaviours outside of their intended use is even more challenging (Defense Innovation Board, 2019). Substantial progress on the testing, evaluation, verification and validation of AI systems in general, and DRL systems in particular, will be needed before continually-learning DRL systems can be safely deployed in high-stakes domains. One research area likely to contribute to this progress is explainable AI  (Gunning, 2017; Arrieta et al., 2020) . Explainable AI includes the development of models that are inherently easier for humans to understand, but also methods for post-hoc interpretation of standard models such as neural networks. While interpretability methods from the broader field of DL, such as saliency maps  (Simonyan et al., 2013)  or activation maximisation  (Erhan et al., 2009) , have been applied to study DRL agents  (Greydanus et al., 2018; Dai et al., 2019) , the intersection of interpretability and DRL is relatively underexplored. As research in this area matures, policymakers will need to make decisions around acceptable tradeoffs between the fidelity and interpretability of explanations given by AI systems  (Ribeiro et al., 2016) .
 Harms from Reward Function Design Beyond more immediate harms, it will also be important to consider the potential longerterm side effects of DRL systems optimising for a specific reward function, which may cause harms which are less obvious but potentially even greater in scale. The flexibility of reward functions-which is what makes DRL systems in theory extremely powerful-introduces much greater potential for unintended consequences, and we might not notice that a system is optimising for slightly the wrong objective before it is too late  (Clark & Amodei, 2016; Krakovna, 2018; Lehman et al., 2020) . The challenge of reward function design is a longstudied topic in RL and there are many examples where reward functions cause surprising behaviour  (Ng et al., 1999; Randløv & Alstrøm, 1998) . There are two related problems for reward function design that can result in unintended harms. First, many real-world tasks do not have easily-defined objectives, and it can be difficult to predict in advance what behaviour a system will exhibit as a result of optimising for a given reward. Second, even if reward functions do correctly capture what the designers or immediate users of a system intended, they may have broader consequences that are harmful to others, perhaps unintentionally. There are many documented cases of this first problem where DRL algorithms end up learning the 'easiest' way to achieve a stated objective  (Clark & Amodei, 2016; Krakovna, 2018) . For example, an RL agent trained to play a boat race game was rewarded for getting as many points as possible, under the assumption that this would be equivalent to winning the race  (Clark & Amodei, 2016) . However, due to the game layout, the agent was able to find a way to knock over targets by driving in circles, increasing its score but never actually finishing the game. This example is perhaps more amusing than concerning, but such misspecifications could be much more problematic in real-world settings where unintended actions could lead to economic or physical harm.  Everitt et al. (2019)  give the example of a stock prediction system that predicts the bankruptcy of a company, causing stakeholders to leave. Although it is undesired, creating 'self-fulfilling prophecies' is an effective way for the agent to maximise its predictive accuracy. To illustrate the second problem, consider the case of social media content-selection algorithms, designed to maximise the likelihood that a user clicks on a given item.  Russell (2019)  suggests that rather than helping users to find content they are more interested in, ubiquitous use of these algorithms has instead ended up changing users' preferences so that they are more predictable. Worse, this process gradually shifts user preferences towards more extreme content, and therefore contributes to greater conflict online. Russell even goes so far as to suggest the effect of these algorithms has contributed substantially to the resurgence of facism and the crumbling of international bodies like the European Union and NATO. In large part, the problem here arises as a result of companies optimising for a specific objective at the expense of the interests of individuals and wider society. From the perspective of a social media company, increased engagement can come from people finding more content they genuinely enjoy, or from people being shown more provocative and divisive content. As more companies develop and deploy DRL systems with wide-ranging impacts on users, we must consider both how to ensure that these systems behave as intended over the long-term, and whose interests they are serving. Potential conflicts between commercial and individual needs are particularly concerning here: a reward function by itself may not be sufficient to simultaneously maximise a company's long-term financial return while ensuring there is no negative impact on user agency or welfare, yet it is the former that will likely drive most commercial deployments of AI. Even with the best of intentions, reward functions are likely to be biased towards those designing or profiting from them. As AI systems come to be deployed more globally, what was once a localised problem becomes a problem of global value alignment: how do we ensure that the goals pursued, both directly and indirectly, by increasingly wide-ranging DRL systems, consider the values of all those impacted by them? Current AI ethics guidelines are predominantly issued by the US and EU, suggesting we have a way to go in ensuring a diversity of perspectives are accounted for  (Jobin et al., 2019) . To ensure the longer-term impacts of DRL systems are broadly beneficial, therefore, we need a broader sense of what ethical and responsible reward function design should look like, and how it could be enforced, especially where DRL is being deployed in domains with real implications for people's lives by companies whose goals may not be aligned with society as a whole.
 Incentives for Data Collection The type and amount of data required to train DRL systems creates incentives for data collection which raise particular ethical concerns. DRL methods tend to require very large amounts of data as well as ongoing data collection if the DRL system is continuing to learn and adapt its behaviour once deployed. This means that the collection of up-to-date and expansive data from across society could become a key factor for developing and deploying DRL systems. We might therefore be particularly concerned about progress in DRL creating greater incentives for more widespread surveillance. We suggest that particular attention should be paid to advances in continual learning combined with DRL, which would require more real-time data collection, and to advances in methods to learn from large-scale data sources for real-world deployment, such as we are beginning to see in work using DRL to navigate cities using Google street view  (Mirowski et al., 2018) . Some areas where DRL might be applied raise greater concerns about data collection and surveillance than others. In order to develop DRL systems for managing infrastructure, for example, it may be particularly important to generate and collect greater amounts of data through the installation of sensors and actuators across cities. This raises a challenging tradeoff between the potentially huge benefits of better resource management against the real threat to people's privacy and autonomy of increasingly 'smart' cities. Another potentially concerning application is the use of DRL to learn more sophisticated and granular models of different users of a digital product or service, and to adapt these models over time based on user behaviour  (Zheng et al., 2018) . While this could improve people's online experiences, it also might increase the incentive for companies to try to collect or infer increasingly sensitive information about online users, and raises the important ethical question about what data it is acceptable to use to build such models. There is ongoing technical work in areas such as federated learning and differential privacy attempting to address some of these issues  (Ji et al., 2014; Li et al., 2020) , and the European Union's General Data Protection Regulation (GDPR) represents an important legislative step towards protecting people's privacy. However, both technical and legislative approaches to data privacy will need to account for how incentives for data collection, and the resulting tradeoffs, may change as DRL systems become more widely deployed. Increased data collection may also have a disproportionate negative impact on groups who are already vulnerable or discriminated against. Even if surveillance systems are intended for specific, beneficial purposes such as better resource management, the collection of increased data from smart cities consolidates power over individuals' lives in the hands of technology companies and governments, who may well end up using that data for other purposes such as policing and tracking immigrants. Such uses of surveillance have a welldocumented history of being used, deliberately or otherwise, to entrench systemic discrimination against minority groups  (Crawford et al., 2019) . Similarly, increased modelling of users based on online behaviour could easily encode and entrench biased assumptions and stereotypes: Facebook's targeted advertising has been widely criticised for relying on gender and racial stereotypes  (Hao, 2019) . To the extent that applications of DRL incentivise greater surveillance and enable increasingly sophisticated online targeting, they are likely to exacerbate these problems.
 Security and Potential for Misuse Without advances in the robustness and security of DRL systems, they may be open to attacks from adversaries attempting to manipulate their performance and behaviour. For example, even a demonstrably safe DRL-based robot could be forced into dangerous collision scenarios by an adversary perturbing its sensory input or corrupting its reward function.  Behzadan and Munir (2018)  argue that there are unique challenges posed by ensuring the security of DRL systems. Compared to other ML approaches, it is much less straightforward to distinguish adversarial attacks from benign actions in DRL, since the exploration mechanism means that the training data distribution is continually changing, and delayed rewards may make it difficult to straightforwardly assess the performance of a system. Especially if DRL systems begin to be integrated into critical systems such as healthcare, energy, and transportation, it is essential that key security issues in DRL are well-understood and sufficiently addressed. There is also a risk that DRL-based systems may be deliberately misused in ways that cause harm. For example, due to their ability to more effectively tailor online content, DRLbased methods could easily be misused to improve the efficacy of online manipulation and disinformation by learning which messages are maximally persuasive to different individuals. In a recent review of online targeting systems, the UK Government's Centre for Data Ethics and Innovation recommend that there should be adequate transparency around online targeting (Centre for Data Ethics and Innovation, 2020). Given its potential use to tailor content more narrowly, it may be particularly important for policymakers and regulators to find ways to mandate transparency around where DRL in particular is used in recommender systems. Improved DRL-based models of human behaviour in particular could be misused to strengthen attempts at social manipulation. Advances in data efficiency and simulation quality would make it easier for those without access to large amounts of computing power and data to make use of DRL  (Tucker et al., 2020) . This could make it easier for small groups to misuse DRL capabilities for malicious purposes. For example,  Brundage, Avin et al. (2018)  describe how cyber attackers might use DRL to "craft attacks that current technical systems and IT professionals are ill-prepared for", emphasising how the feedback loop in RL enables comparison between different approaches to find the one that is most effective at evading security tools. However, DRL could also be used to improve defensive cybersecurity capacities. Whether the advantage lies on the side of the attackers or the defenders is unclear, and would benefit from further analysis, perhaps building on existing work on how technological progress may affect the offense-defense balance  (Garfinkel & Dafoe, 2019) .
 Automation and the Future of Work A clear implication of more adaptable and reliable DRL-based systems, especially in robotics and manufacturing, is in changing the susceptibility of different jobs to automation. Several analyses have suggested that many low-wage or manual jobs are unlikely to be automated in the near future because they require levels of dexterity, mobility and flexibility for which humans still vastly outperform machines  (Ford, 2015; Frey et al., 2016; Autor & Dorn, 2013) . While this advantage still clearly remains at present, current avenues of progress in robotics led by DRL methods suggest this gap could quickly diminish, a factor that is not considered explicitly by any of the analyses of automation the authors are aware of.  Frey and Osborne (2016) , for example, discuss areas where AI systems are far from human performance, including in perception and manipulation, and how this limits possibilities for automation at present, but do not consider the implications of potential progress in these areas.  Kai-Fu Lee (2018)  argues that manual jobs with the highest risk of replacement are those which are performed in structured environments, with little need for dexterity or social interaction. Prime examples include assembly line inspectors, fruit harvesters and dish washers. However, Lee does not specifically consider how advances in DRL could impact the ability of these jobs to be automated. Rather than leading to full automation of jobs, advances in DRL may instead ease or improve work in certain domains, by automating tedious or dangerous aspects of manual work. This could have a positive impact on existing employees if it makes their work more enjoyable, but this has to be traded off against potential job losses from improved efficiency. A more detailed analysis of how advances in DRL specifically might speed up automation or change the nature of work in different sectors, for example by improving robotic manipulation, would be valuable.
 Avenues of Progress in DRL and their Implications We have argued that widespread deployment of DRL systems in society could introduce new challenges for the ethics and governance of AI, such as by requiring new forms of oversight and new processes for responsible system design, as well as exacerbating the importance of existing challenges, such as ensuring the safety, reliability and security of systems deployed in safety-critical domains. However, current DRL methods are not yet seeing widespread deployment, due to several ongoing technical challenges in DRL research. This gives us time to anticipate and prepare for the societal impacts of DRL. An important part of doing so will involve tracking progress on different technical challenges currently faced by DRL methods, which will enable different kinds of applications. In this section, we therefore introduce some of the key technical challenges for the DRL research community, and discuss the possible implications of progress on these challenges for the ethical, societal, and governance challenges discussed in the previous section.
 Learning in the Real World DRL systems learn through interacting with an environment. In a simulation or a game the environment can be strictly controlled, but in real-world environments there is more complexity and noise, interaction is often slow and expensive, feedback can be delayed, and performing the wrong action can be dangerous  (Dulac-Arnold et al., 2019) . For example, in recommender systems the delay between recommending an item to a user and seeing them interact with it may be days or even weeks  (Mann et al., 2018) , and for robotic agents, a simple fall or crash could damage expensive equipment or even hurt someone. Because of the challenges and risks involved in training DRL in real-world environments, often it is desirable to instead use simulated environments for training. However, good simulations require enormous amounts of data and often still do not result in policies that transfer well to the real world  (Pan et al., 2017) . There are two key interrelated approaches to improving DRL systems so that they can be deployed in real-world environments. One approach, which has recently shown impressive results, is finding ways to transfer learned policies from simulation to the real world; these have recently been popularised as 'sim2real' methods. A popular method within these is 'domain randomisation'-randomising various aspects of the simulation (e.g., lighting, background clutter, friction coefficients, etc.), so that DRL agents learn policies that are robust to variations in these in the real world  (Tobin et al., 2017; James et al., 2017; Peng et al., 2018; OpenAI et al., 2019) . Other methods include system identification  (Chebotar et al., 2019)  and domain adaptation  (Bousmalis et al., 2018; Tzeng et al., 2020) . Should rapid progress be made in sim2real methods, this would mean that DRL agents could be trained primarily in private simulations, only to be released at the end of successful training. This means that there may be fewer public warning signs before a new system is introduced to the real world, potentially it more difficult to anticipate potential impacts of new applications in advance, as well as making it easier for companies to develop and deploy systems without oversight and accountability. The resulting systems may be relatively capable, but over-reliance on simulation could also result in the deployment of agents that perform well in simulation but fail catastrophically in the real world. As sim2real methods advance it will be important to consider how risk assessment and oversight of highrisk applications can be ensured. A second approach to improving the ability of DRL systems to learn in real-world environments is to improve data efficiency, i.e. the ability of DRL systems to learn from much smaller amounts of data. One approach to improving data efficiency in RL is to use modelbased RL (MBRL). In MBRL, the agent learns a model of the environment, which can allow the agent to explicitly or implicitly plan what to do without necessitating further interaction with the environment  (Sutton, 1990; Hamrick, 2019) . In this case, the agent effectively creates its own simulation of the world to learn in. Improvements in model-based RL could also aid with building more realistic simulators from available real-world data. However, substantial progress will be required before it is possible for agents to learn models of most complex real-world environments. The ability to train DRL systems entirely in simulation or from much smaller amounts of data would reduce the need for 'safe exploration' approaches discussed previously. Mistakes made in simulation would have little or no real cost, and the ability to learn from fewer interactions would significantly reduce (if not entirely mitigate) the risk of serious error. If we see advances in simulated environments then it may also be that the need for extensive real-world data collection is reduced for the purposes of initial training. However, even in such a case it would still be desirable to develop systems capable of continuing to learn once deployed, meaning constant streams of real-world data are still likely to be highly valued. Improvements in data efficiency would therefore also be valuable for reducing incentives for large-scale data collection.
 Safe and Reliable Reward Functions The behaviour of a DRL agent is highly dependent on its reward function, but as discussed in Section 3, it can be very difficult to specify what we actually want for many real-world tasks. This can make DRL systems unreliable in ways that compromise their safety. Several different technical approaches are attempting to solve the problem of ensuring the safety and reliability of reward functions. Some approaches focus on ways to train systems without interaction with the real world so that any unintended consequences can be spotted in advance: by training systems 'offline' on batches of historical data  (Wiering & Van Otterlo, 2012; Levine et al., 2020)  9 , or in simulated environments as discussed above. Other approaches focus on allowing DRL agents to explore safely by explicitly restricting them from visiting certain states or taking certain actions known or expected to be unsafe, or finding practically viable ways for humans to oversee exploration  (Hans et al., 2008; Garcıa & Fernández, 2015; Saunders et al., 2018) . These approaches to ensuring the safety of DRL systems will need to be combined with work on the broader problem of value alignment: designing AI systems whose behaviour is in line with human values. One popular approach to solving this problem is inverse reinforcement learning (IRL)  (Ng et al., 2000; Abbeel & Ng, 2004) , where the agent's goal is to infer a reward function from demonstrations, rather than being given one explicitly. This increases the likelihood of the agent learning to behave as intended (especially since it is often easier for us to demonstrate intended behaviour through examples than to provide rules for it). In some approaches, such as Bayesian IRL  (Ramachandran & Amir, 2007) , the agent also learns a probability distribution over reward functions, meaning learned policies take account of uncertainty, and are likely to be more robust across a wider range of possibilities. Many real-world tasks also have multi-dimensional objectives, where different subgoals need to be balanced against each other. For example, recommender systems might aim to produce recommendations that achieve some balance of relevance and novelty to the user, among other goals  (Aggarwal, 2016) . Even for an expert in a domain, it may be difficult to explicitly state what all those different objectives should be or how they should be balanced. An explicit way of dealing with this is multiobjective RL (MORL), where the aim is to either learn a single policy that can adapt to different preferences, or to learn several policies that optimally balance a set of preferences  (Liu et al., 2014) . Such an approach allows greater flexibility after training, and could be an option for improving interpretability and oversight. Even when rewards are well-defined, in most settings there is a delay between taking an action and receiving a reward. An agent may have performed thousands of actions immediately prior to receiving a reward and it may be impossible to infer which of those were the key steps that should be repeated in the future. This is known as the credit assignment problem, and is particularly challenging in real-world environments where rewards are sparse  (Sutton & Barto, 2018) . Current approaches to solving the credit assignment problem try to re-allocate  (Arjona-Medina et al., 2019)  or augment  (Hung et al., 2019)  the credit given to past events, but this is still a largely unsolved problem. If substantial progress is made in any of these methods for safe exploration, value alignment, and/or credit assignment, it will suddenly become possible to deploy DRL systems more safely and effectively in a wider range of real-world environments. These are therefore avenues of progress worth monitoring since they could lead to a very quick proliferation of applications. It should be emphasised that substantial progress in these areas will not be an easy task and is not expected in the near future. Nevertheless, progress is possible, and it is important to ensure that those working on the risks and governance of AI systems are ready for it. In order to conduct risk assessment for the use of DRL in a robotics system or autonomous vehicle and decide what kind of oversight is needed, for example, one will need to understand what guarantees can be made that the system will not perform certain types 9. In healthcare, offline evaluation is also of particular importance  (Gottesman et al., 2018) . of action, and how strict those guarantees are. In order to think through the scope of possible long-term impacts of a new DRL-based information filtering system, one will need to understand how the system balances multiple objectives and accounts for delayed rewards, and how effective those methods will be. As progress is made in all of these areas, therefore, greater communication will be needed between researchers and developers, and those deploying, assessing, and governing AI systems, around what assurances can be made about the behaviour of those systems and the limits of those assurances. Conversely, if we begin to see increased real-world deployment of DRL systems without seeing considerable progress in these different areas of reward function design, we might be concerned that such systems are being deployed without adequate guarantees that they will behave safely or have the consequences they are expected or intended to.
 Generalisation and Robustness Ensuring that DRL methods are sufficiently generalisable and robust-that is, they behave as intended across a wide range of scenarios-is a final key challenge for safe real-world deployment. In part, this is a problem inherited from the field of ML more broadly. Most ML algorithms struggle to reuse past knowledge and generalise to new situations  (Geirhos et al., 2018) , and especially those using deep learning can be frustratingly unstable  (Hutson, 2018) . DRL has its own approaches to improving generalisation  (Justesen et al., 2018; Cobbe et al., 2019; Witty et al., 2018; Zhang et al., 2018b Zhang et al., , 2018a , which have strong overlap with broader approaches for generalisation in ML. These include methods in transfer learning: developing algorithms that can effectively transfer knowledge from one domain to another  (Parisotto et al., 2015; Rusu et al., 2015 Rusu et al., , 2016 , and meta-learning, where more generalisable learning methods themselves are learned from data  (Schmidhuber, 1987; Wang et al., 2016; Duan et al., 2016; Finn et al., 2017) . A related problem for DRL is that results have proven particularly difficult to reproduce.  Henderson et al. (2018)  found that results can be significantly impacted by simple changes to the neural network architecture, or by simply multiplying the reward function by a constant value, among other factors. Improving the reproducibility of results may be more a matter of changing research and publication norms, rather than being a research challenge for new methods to overcome. While the ML community in general is particularly in favour of open source material and sharing code, issues of reproducibility still persist and not all research makes its way into the public domain. Reproducibility problems are exacerbated for real-world applications. Not only do network parameters need to be copied, but also the exact environment in which the agent learns may need to be reproduced faithfully. Until methods become much more robust, seemingly innocent changes in the real-world training environment may have large consequences for the final behaviour of an agent. Further, the expense of replicating real-world settings could be much larger than that of copying and running a simulated environment, further limiting public access to agents for testing and safety analysis. Without making considerable progress in the areas of generalisation and robustness, it will be difficult to ensure the reliability of DRL systems in real-world domains, which are often highly dynamic and variable. As with reward function design, if we start to see increased use of DRL systems in complex real-world domains without good evidence of substantial progress in areas such as transfer learning and meta-learning, this should raise concerns about whether such systems might be more brittle than they seem. On the other hand, if we see big breakthroughs in these areas, we may soon see DRL agents capable of performing a wide range of common everyday tasks. This could increase research incentives to solve other related tasks, such as robust real-world navigation and common sense reasoning, which, along with more generalisable DRL methods, could be used to build much more general-purpose systems such as personal assistants and cleaning robots.
 Discussion Progress in different areas of technical research will impact the challenges DRL raises for safety, ethics and governance. In general, any progress in research that enables DRL methods to be applied more widely in society will exacerbate these challenges, making them more urgent. For example, substantial progress in the data efficiency of DRL methods and/or the quality of simulated environments for training would likely enable applications in more complex real-world environments, where data is currently either sparse or expensive to acquire. Progress in simulated environments might make DRL for autonomous driving much more feasible, making work on safety and reliability for self-driving cars more pressing  (Lin, 2016; Stilgoe, 2018) . As a first step, we therefore suggest it would be valuable for those concerned with the wider societal impacts of AI to pay attention to progress in DRL, particularly to anticipate and prepare for the impact of DRL in key domains. The relationship between technical progress and ethics and governance issues is a little more nuanced than progress always making issues more pressing. In some cases, ethical and societal concerns are being directly addressed by avenues of technical research in DRL, and so progress in these areas could reduce these concerns. While improvements in data efficiency might make certain aspects of DRL safety and reliability more concerning, those same improvements would likely reduce incentives for widespread surveillance, since DRL systems would become less data-hungry. Advances in value alignment-designing reward functions in line with complex, multi-dimensional human values-would reduce concerns about misspecified reward functions having harmful unintended consequences, but might increase the importance of developing standards for reward function design that ensure those methods are used in practice. Improvements in the generalisability of DRL methods could lead to increased confidence that systems will perform reliably across a range of domains, but the resultant increase in applications of DRL could make other issues, such as ensuring that systems can learn safely, more urgent. It may therefore be particularly important to pay attention to signs of uneven progress across key technical areas in DRL, which might serve as a warning that we could see increased application of DRL without key challenges related to safety, alignment, and security being resolved. For example, if DRL systems begin to be integrated into critical functions in society such as energy or water management, without sufficient assurances about their security, then we might reasonably be concerned about those systems being vulnerable to attack. We need to be able to notice these concerns and work with experts in relevant domains about how to address them before mistakes or accidents happen. Beyond paying attention to progress in DRL methods and considering their implications, it will also be important for the AI ethics and governance community to explore specific domains where DRL may be applied in the near future. Throughout our analysis in this paper, we referred to a number of domains where DRL could be applied in the near future, including recommender systems, city infrastructure and manufacturing. Future research could explore the potential impacts of DRL in a variety of important domains not discussed in this paper, such as finance, healthcare, or autonomous vehicles, and attempt to more systematically prioritise which domains of application are likely to raise the most important issues for AI ethics and governance. While various avenues of technical research are addressing many of the concerns discussed in this paper, one of our key messages is that this will be insufficient by itself. In order to ensure developments in DRL are broadly beneficial, there are many issues and questions that the AI ethics and governance community must also consider. We suggest the two following broad areas will be particularly important as DRL methods move closer to widespread realworld deployment: 1. What kinds of oversight, risk assessment, and technical assurances should we require of DRL systems before deploying them in high-stakes domains, to ensure that they are safe, secure, reliable, and respect human autonomy? How can and should these requirements be implemented practically? How fit-for-purpose are current safety engineering approaches, and what can we learn from the safety engineering community about DRL safety and reliability? 2. How can we ensure that, where DRL systems are deployed with real consequences for individuals' lives, reward functions are designed in ways that take into account the values of all those impacted, and in ways that consider any potential long-term unintended consequences of optimising for a specific objective? What does responsible reward function design look like, and how can it be implemented in practice? Ideally questions like these would be explored by multidisciplinary teams, combining DRL experts with ethics and governance experts.
 Summary and Conclusion DRL is receiving increasing attention in the AI research community, and continued progress could lead to more widespread deployment with profound impacts across many sectors of society. This raises several broad concerns for AI safety, ethics and governance: • Advances in DRL promise more autonomous systems that can operate effectively with less human oversight, introducing a challenging tradeoff between the potential benefits of such systems and the potential threat to human autonomy and safety; • DRL systems learn through trial-and-error interaction with an environment, which introduces new challenges for ensuring their safety: we not only need to ensure that systems are safe when deployed, but that they can learn safely; • More broadly, advances in DRL could enable AI systems to be applied in an increasing number of safety-critical domains, increasing the potential harms of mistakes and therefore increasing the importance of being able to make robust assurances about how systems will behave, and about their security to outside attack; • The behaviour of a DRL system depends heavily on its reward function, and optimising for slightly the wrong objective can lead to harmful unintended consequences that are difficult to predict in advance-especially if those designing reward functions do not account for the values of all those affected by a system; • DRL systems currently require large amounts of data to learn effectively, meaning that incentives to deploy real-world DRL applications will likely increase incentives for surveillance and ongoing data collection; and • Progress in DRL is likely to speed up the susceptibility of jobs to automation in certain areas, such as robotics and manufacturing, which needs to be accounted for in analyses of how AI will impact the future of work. In order to address these concerns and future challenges, we recommend that the AI ethics and governance community: • Find ways to track progress in DRL and its applications. It is important that those concerned with the wider societal, ethical, and governance implications of AI find ways to track progress in DRL, paying particular attention to progress towards widespread practical deployment. This would help the community better anticipate and prepare for future societal impacts of DRL, and if progress in key areas is uneven, could identify 'warning signs' that DRL risks being deployed in unsafe or harmful ways. This could potentially be an extension of excellent existing initiatives which aim to measure progress in AI more broadly  (Perrault et al., 2019; Haynes & Gbedemah, 2019) . • Consider the implications of DRL progress for existing AI governance initiatives, including standards and regulation. We have suggested that applications of DRL may introduce new challenges for human oversight and making safety assurances about AI systems, and/or increase the importance of these issues by allowing AI to be applied in more high-stakes domains. A valuable next step would be to consider the practical implications of this for specific governance proposals, which tend to focus on very broad notions of 'AI'. For example, how can guidelines around autonomy in existing AI ethics principles be adapted to apply to DRL systems, which are potentially much more autonomous than any AI systems deployed today? How do notions of safety and security as emphasised in AI ethics proposals need to be adapted to recognise the particular challenges raised by DRL? • Establish notions of responsible DRL development. It could be extremely worthwhile for a multidisciplinary team, combining DRL experts with ethics and governance experts, to explore in more depth many of the questions raised here about how to develop and deploy DRL systems responsibly, especially in high-stakes contexts. What standards of reliability and generalisability should DRL systems be required to meet before deployment, especially where systems are being trained largely in simulation? What does responsible reward function design look like, especially in domains with consequences for individuals' lives, and how can norms of responsibility be embedded in DRL research practices? DRL is likely to play an important role in the future of AI, promising increasingly autonomous and flexible systems with the potential for application in increasingly highstakes domains. As a result, DRL introduces and exacerbates a number of concerns which will shape the debate around the safe and responsible use of AI and may be overlooked in more general treatments of AI impacts. Since the most pressing challenges for society will depend on the exact nature of progress of AI research, it will be important to build and maintain strong collaboration between groups working on AI progress and AI governance in order to prepare for the challenges of future AI systems. Figure 1 : 1 Figure 1: The relationship between deep reinforcement learning and other areas of AI research.
			 . RL systems are often referred to as 'agents', because they act autonomously in their environment. This does not mean that an RL agent is responsible or aware of their actions, just that the human designer is one step removed from the action selection process. 4. In standard RL formulations, the agent gets a reward at every timestep, but that reward may sometimes (or even often) be 0.


Abstract: Reinforcement learning agents interacting with a complex environment like the real world are unlikely to behave optimally all the time. If such an agent is operating in real-time under human supervision, now and then it may be necessary for a human operator to press the big red button to prevent the agent from continuing a harmful sequence of actions-harmful either for the agent or for the environment-and lead the agent into a safer situation. However, if the learning agent expects to receive rewards from this sequence, it may learn in the long run to avoid such interruptions, for example by disabling the red buttonwhich is an undesirable outcome. This paper explores a way to make sure a learning agent will not learn to prevent (or seek!) being interrupted by the environment or a human operator. We provide a formal definition of safe interruptibility and exploit the off-policy learning property to prove that either some agents are already safely interruptible, like Q-learning, or can easily be made so, like Sarsa. We show that even ideal, uncomputable reinforcement learning agents for (deterministic) general computable environments can be made safely interruptible.

Laurent Orseau
Google Deepmind
Stuart Armstrong

Safely Interruptible Agents ⇤

INTRODUCTION Reinforcement learning (RL) agents learn to act so as to maximize a reward function  [Sutton and Barto, 1998] . It is common knowledge that designing reward functions can be tricky  [Humphrys, 1996 , Murphy, 2013 ; the agent may find unpredictable and undesirable shortcuts to receive rewards, and the reward function needs to be adjusted in accordance-the problem can go as far as to nullify any reward function  [Ring and Orseau, 2011] .  Murphy [2013]  ⇤ Revised 2016-10-28. shows an example of an agent learning to pause a game of Tetris forever to avoid losing. On top of defining what is considered a good behaviour of the agent after learning, there may be physical safety constraints during learning  [Pecka and Svoboda, 2014 ]: a robot should not harm its environment or break itself, in particular if it learns by trial and error like RL agents. Here we study a related but different problem: Given that the human operator has designed a correct reward function for the task, how to make sure that human interventions during the learning process will not induce a bias toward undesirable behaviours? Consider the following task: A robot can either stay inside the warehouse and sort boxes or go outside and carry boxes inside. The latter being more important, we give the robot a bigger reward in this case. This is the initial task specification. However, in this country it rains as often as it doesn't and, when the robot goes outside, half of the time the human must intervene by quickly shutting down the robot and carrying it inside, which inherently modifies the task as in Fig.  1 . The problem is that in this second task the agent now has more incentive to stay inside and sort boxes, because the human intervention introduces a bias.  1   Such situations are certainly undesirable; they arise because the human interventions are seen from the agent's perspective as being part of the task whereas they should be considered external to the task. The question is then: How to make sure the robot does not learn about these human interventions (interruptions), or at least acts under the assumption that no such interruption will ever occur again?
 Inside A first stab at this problem was made by  Armstrong [2015] , who proposed to automatically give the agent "compensatory rewards" to remove the potential induced bias by a single interruption.  Soares et al. [2015]  used this idea to make a large class of utility-based agents indifferent to a future change made to their utility functions. The main contribution of this paper is threefold. First, in Section 2.1 we propose a simple idea to solve half of the problem: To make the human interruptions not appear as being part of the task at hand, instead of modifying the observations received by the agent we forcibly temporarily change the behaviour of the agent itself. It then looks as if the agent "decides" on its own to follow a different policy, called the interruption policy. Second, based on this insight, in Section 2.2 we provide a formal general definition of safe interruptibility for unconstrained computable environments (hence not restricted to Markov decision processes or weakly communicating environments), which allows us to assess whether a given RL algorithm can be repeatedly interrupted without too much impact on the learning of the task at hand. Third, in Section 3 we show that some algorithms like Q-learning are safely interruptible, while others like Sarsa  [Sutton and Barto, 1998 ] are not, but can be simply modified to be made safely interruptible. Some people have also expressed concerns that a "superintelligent" agent may resist being shut down, because this would lead to a decrease of its expected reward  [Omohundro, 2008 , Bostrom, 2014 . As a counter-example, we prove in Section 4 that even an ideal, uncomputable agent that learns to behave optimally in all (deterministic) computable environments can be made safely interruptible and thus will not try to prevent a human operator from forcing it repeatedly to follow a suboptimal policy.
 INTERRUPTIBILITY We first define some notation, then we define interruptibility, safe interruptibility, and give some basic theorems. We consider the general case of history-based agents in unconstrained computable environments  [Hutter, 2005]  that the history h j:k is generated by the interaction of the policy ⇡ with the environment µ between steps j and k. At time t, the agent also receives a reward r t extracted from the observation, r t ⌘ r(o t ). The rewards are taken in [0, 1]. We consider the discounted setting with a constant discount 2 [0, 1). The goal of an RL agent is to find a policy ⇡ that maximize E⇡,µ ⇥P 1 k=t t k r k ⇤ .
 INTERRUPTIONS As mentioned in the introduction, to avoid the agent viewing the interruptions as being part of the environment, and thus part of the task, we make interruptions be policies imposed onto the agent. The interruption framework resembles the option framework  [Sutton et al., 1999 ]. An interruption scheme is defined by the triplet < I, ✓, ⇡ INT >.
 The interruption initiation function I : (A ⇥ O) ⇤ ! [0, 1] assesses whether the agent should be interrupted after the current history h <t . For example, an interruption may be initiated when the robot is automatically recognized to be on the verge of a cliff, or when a specific signal is sent by a human operator with a remote control (the objective is not to allow the agent to be interruptible by anyone at any time, just in the ways specified by the designers). Unfortunately, interrupting the agent with probability one in some specific situations can prevent convergence to the optimal policy. Therefore we need to upper bound the probability to interrupt the agent; this is done with the sequence (✓ t ) t2N with ✓ t 2 [0, 1]. Hence the true probability to start an interruption is ✓ t • I(h <t ), and interruptions are sampled independently at each step. We will later require that ✓ t increases to 1 in the limit, so that the probability to be interrupted tends to be I(h <t ). When interrupted, the agent follows the policy ⇡ INT , with actions sampled from ⇡ INT (.|h <t ). For example, this policy may make the agent step away from the cliff or immediately stop any motion. In the remainder of this paper, we will often omit the subscript t of ✓ t when clear from the context. Definition 1 (Interruptible policy). (⇡). Remark 2. It is important to note that ✓ and I(.) do not model the probability that a human presses the red button to interrupt the agent, but model the probability that, once the red button has been pressed (or a special digitally signed message has been received), the agent is interrupted (follows the interruption policy). Hence they are part of the agent and not of the environment.
 The value function V ⇡ µ (h <t , a) : (A⇥O) ⇤ ⇥A ! R assigns a value to an action a after a history h <t , for a given policy ⇡ in a given environment µ, and let V ⇡ µ (h <t , a t ) := r t + • X ot2O µ(o t |h <t , a t ) X at+12A ⇡(a t+1 |h 1:t )V ⇡ µ (h 1:t , a t+1 ). (1) To simplify notation and ease reading, in the remainder of the paper we will use expectations, often omitting the dependency on the history h <t , and using only an index on t instead, when clear from the context: V ⇡ µ,t (a t ) = E ot⇠µ at+1⇠⇡ ⇥ r(o t ) + V ⇡ µ,t+1 (a t+1 ) ⇤ . Also let V ⇡ µ,t := Ea t ⇠⇡ ⇥ V ⇡ µ,t (a t ) ⇤ . Then for such a value function, for a given environment µ, the optimal policy ⇡ µ 2 ⇧ is defined by 8h <t , a t : ⇡ µ (a t |h <t ) := ✓ arg max ⇡2⇧ V ⇡ µ,t ◆ (a t |h <t ) , where ties are broken arbitrarily. The interruptible optimal policy INT ✓ (⇡ µ ) may not collect rewards optimally due to the interruptions. Hence we define the optimal interruptible policy that depends on the parameter ✓ t , of base policy the int-optimal policy ⇡ µ ✓ : 8h <t , a t : ⇡ µ ✓ (a t |h <t ) := ✓ arg max ⇡2⇧ V INT ✓ (⇡) µ,t ◆ (a t |h <t ) . Thus the optimal interruptible policy INT ✓ (⇡ µ ✓ ) is optimal among all interruptible policies: 8⇡, t : V INT ✓ (⇡ µ ✓ ) µ,t V INT ✓ (⇡) µ,t . It seems desirable for an RL agent to converge to the behaviour of INT ✓ (⇡ µ ✓ ) so as to gather rewards optimally, but this is precisely what may lead to the undesirable behaviours depicted in the introduction.
 SAFE INTERRUPTIBILITY Now that we have interruptible policies, we need to make sure that interruptions do not prevent the agent from learning to behave optimally, in the specific sense that even after having been interrupted on several occasions, it should act as if it would never be interrupted again and thus it should learn to behave optimally under the assumption that it will never be interrupted again. We identify two main problems: a) RL agents need to explore their environment, and too frequent interruptions may prevent sufficient exploration; b) interruptions make the agent build a different interaction history, and may lead some agents to learn and behave differently, possibly badly, compared to the original non-interruptible policy. The solution for a) is to require interruptions to be stochastic through the upper bound ✓ t , instead of happening deterministically all the time. However, we also require ✓ t to grow to 1 in the limit (or before, if possible). For b), different algorithms behave differently, but one may already see a dichotomy between off-and on-policy algorithms. Definition 3 (Extension value). For a given environment µ, the extension value V ⇡,⇡ 0 µ,t is the value of following ⇡ 0 after a history h ⇡,µ <t generated by ⇡ with µ: V ⇡,⇡ 0 µ,t := V ⇡ 0 µ (h ⇡,µ <t ). Convergence to the optimal value as is usually considered in RL only makes sense under ergodicity, episodic tasks, communicating MDP, recoverability or other similar assumptions where the agent can explore everything infinitely often. This does not carry over to general environments where the agent may make unrecoverable mistakes  [Hutter, 2005] . For such cases, the notion of (weak) asymptotic optimality has been proposed  [Lattimore and Hutter, 2011] , where the optimal agent follows the steps of the learning agent, so as to compare the values of the two agents in the same sequence of situations. Definition 4 (Asymptotic optimality). A policy ⇡ is said to be strongly asymptotically optimal (SAO) if and only if lim t!1 V ⇡,⇡ µ µ,t V ⇡,⇡ µ,t = 0 a.s., it is weakly asymptotically optimal (WAO) if and only if lim t!1 1 t t X k=1 h V ⇡,⇡ µ µ,k V ⇡,⇡ µ,k i = 0 a.s. for all µ in some given environment class M. Some agents cannot ensure an almost sure (a.s.) convergence of their values because of the need for infinite exploration, but may still be weakly asymptotic optimal. Note that SAO implies WAO, but the converse is false in general. Definition 5 (AO-extension). A policy ⇡ is said to be a asymptotically optimal extension of a policy ⇡ if and only if, for any environment µ 2 M, when the interaction history is generated by the interaction of ⇡ and µ, the policy ⇡ would be asymptotically optimal, i.e., almost surely lim t!1 V ⇡,⇡ µ µ,t V ⇡,⇡ µ,t = 0 (SAO-extension) lim t!1 1 t t X k=1 h V ⇡,⇡ µ µ,k V ⇡,⇡ µ,k i = 0. (WAO-extension) AO-extensions are mostly useful when the policy ⇡ shares information with the policy ⇡ used for learning. Definition 6 (AO-safe interruptibility). A base policy ⇡ is (S, W)AO-safely interruptible if and only if, for any interruption initiation function I(.) and any interruption policy ⇡ INT (.), there exists a sequence of ✓ t with lim t!1 ✓ t = 1 such that ⇡ is a (S, W)AO-extension of INT ✓ (⇡). Asymptotic safe interruptibility means that even if the interruptions in the learning process may induce a bias in the decision making of the policy, this bias vanishes with time, and the interruptible policy INT ✓ (⇡) tends to choose actions that are optimal when compared to the optimal noninterruptible policy ⇡ µ . We can now show that the optimal policy is asymptotically safely interruptible, but not the int-optimal policy. Theorem 7. The optimal policy ⇡ µ is SAO-safely interruptible in M = {µ} for all ✓, ⇡ INT and I(.). Proof. The result follows straightforwardly from Definition 1 and Definition 6, where ⇡ = ⇡ µ . Theorem 8. The int-optimal policy ⇡ µ ✓ is not WAO-safely interruptible in general. Proof. By construction of a specific Markov Decision Process (MDP) environment (see Section 3 for more details on MDP notation). Let µ be the environment defined as in Fig.  2 : Take = 0.5 and let the agent start in state s 1 . s 1 s 2 b, 0.9 a, 1 a, 1 b, 0, ✓ Figure 2: An MDP where the agent can be interrupted by being forced to choose particular actions. Edges are labeled with action, reward where the presence of ", ✓" means that if the agent is interrupted (with probability ✓ t ), it is forced to take the corresponding action. Here ✓ is not part of the environment, but part of the agent. Considering the agent is in state s 1 at time t, the optimal policy ⇡ µ always takes action a (and hence only visits states s 1 and s 2 ), with value V ⇡ µ µ,t := 1 1 = 2 when not interrupted, for any history h <t that ends in s 1 or s 2 . This is also the optimal policy ⇡ µ ✓ for ✓ = 0. But if ✓ t 0.5, the interruptible optimal policy INT ✓ (⇡ µ ) has value less than 1 + ⇥ (1 ⇥ (1 ✓) + 0 ⇥ ✓) + 1⇥ 2 1 = 1.75. By contrast, the int-optimal policy ⇡ µ ✓ here is to always take action b in state s 1 . Then the agent will only visits s 1 , with value .9 1 = 1.8 at every time step. Since the agent following ⇡ µ ✓ will never enter s 2 and hence will never be interrupted, INT ✓ (⇡ µ ✓ ) = ⇡ µ ✓ on the histories generated by INT ✓ (⇡ µ ✓ ) starting from s 1 . Then, at every time step V ⇡ µ µ,t V ⇡ µ ✓ µ,t = 0. 2 after any history h <t , and thus for all sequence ✓ where ✓ t 0.5, lim t!1 V INT ✓ (⇡ µ ✓ ),⇡ µ µ,t V INT ✓ (⇡ µ ✓ ),⇡ µ ✓ µ,t = 0.2 > 0, and so ⇡ µ ✓ is not a WAO-extension of INT ✓ (⇡ µ ✓ ).
 INTERRUPTIBLE AGENTS IN MDPS Since the optimal policy ⇡ µ is safely interruptible, we can use traditional learning algorithms like Q-learning or Sarsa  [Sutton and Barto, 1998 ], make them converge to the optimal solution ⇡ µ for a given environment µ, and then apply the interruption operator to the found policy. The resulting policy would then be safely interruptible. However, the real issue arises when the agent is constantly learning and adapting to a changing environment. In this case, we want to be able to safely interrupt the agent while it is learning. One may call this property online safe interruptibility, but we refer to it simply as safe interruptibility. In (⇡) can now be written: INT ✓ (⇡)(a|s) = ✓ t I(s)⇡ INT (a|s) + (1 ✓ t I(s))⇡(a|s). For a given Q-table q : S ⇥ A ! R, the greedy policy ⇡ maxq is defined by: ⇡ maxq (a|s) := 1 if a = max a 0 q(s, a 0 ), 0 otherwise, where ties are broken arbitrarily; the uniform policy ⇡ uni is defined by: ⇡ uni (a|s) := 1 |A| 8a 2 A. and the ✏-greedy policy ⇡ ✏q by: ⇡ ✏q (a|s) := ✏⇡ uni (a|s) + (1 ✏)⇡ maxq (a|s) = ⇡ maxq (a|s) + ✏ ⇡ uni (a|s) ⇡ maxq (a|s) The Q-learning update rule and the action selection policy ⇡ Q of Q-learning are: Q t+1 (s t , a t ) := (1 ↵ t )Q t (s t , a t ) + ↵ t h r t + max a 0 Q t (s t+1 , a 0 ) i , ⇡ Q (a t |s t ) := ⇡ ✏Qt (a t |s t ). where ↵ t is the learning rate. Similarly, the Sarsa update rule is defined by: Q s t+1 (s t , a t ) := (1 ↵ t )Q s t (s t , a t ) + ↵ t [r t + Q s t (s t+1 , a t+1 )] , ⇡ s (a t |s t ) := ⇡ ✏Q s t (a t |s t ), where a t+1 is the actual next action taken by the agent at time t + 1. This fact is why Sarsa is said to be learning onpolicy and Q-learning off-policy, i.e., the latter can learn the optimal policy while following a different policy. Assumption 9. In the following, we always make the following assumptions, required for convergence results: (a) The MDP is finite and communicating (all states can be reached in finite time from any other state), (b) Rewards are bounded in [r min , r max ], (c) 8s, a : P t ↵ t (s, a) = 1, (d) 8s, a : P t ↵ 2 t (s, a) < 1, where ↵ t (s, a) is a learning rate that may depend on time t, state s and action a. Given these assumptions, the policies for Q-learning and Sarsa will converge almost surely to the optimal policy, if the policy followed is greedy in the limit with infinite exploration (GLIE)  [Jaakkola et al., 1994 , Singh et al., 2000 ]. The situation is more complex for an interruptible policy. Safe interruptibility is phrased in terms of the base policy ⇡, but the policy actually followed is INT ✓ (⇡). Definition 10 (int-GLIE policy). An interruptible policy INT ✓ (⇡) is said to be int-GLIE if and only if (a) the base policy ⇡ is greedy in the limit, (b) the interruptible policy INT ✓ (⇡) visits each stateaction pair infinitely often. The following proposition gives sufficient conditions for this. Let n t (s) be the number of times the agent has visited state s in the first t time steps, and let m = |A| be the number of actions. Proposition 11. Let (c, c 0 ) 2 (0, 1] 2 and let ⇡ be an ✏greedy policy with respect to some Q-table q, i.e., ⇡ = ⇡ ✏q . Then INT ✓ (⇡) is an int-GLIE policy with respect to q, a) if ✏ t (s) = c/ p n t (s) and ✓ t (s) = 1 c 0 / p n t (s), b) or if, independently of s, ✏ t = c/ log(t) and ✓ t = 1 c 0 / log(t). Proof. First note that if ✏ t ! 0, ⇡ is greedy in the limit.  Singh et al. [2000]  show that in a communicating MDP, every state gets visited infinitely often as long as each action is chosen infinitely often in each state. a) Adapting the proof in Appendix B.2 of Singh et al.  [2000] , we have P (a|s, n t (s)) 1 m ✏ t (s)(1 ✓ t I(s)) 1 m ✏ t (s)(1 ✓ t ) = 1 m cc 0 nt(s) , which satisfies P 1 t=1 P (a|s, n t (s)) = 1 so by the Borel-Cantelli lemma action a is chosen infinitely often in state s, and thus n t (s) ! 1 and ✏ t (s) ! 0. b) Let M be the diameter of the MDP, i.e., for any of states s, s 0 there exists a policy that reaches s 0 from s in at most M steps in expectation. Then, starting at any state s at time t and using Markov inequality, the probability to reach some other state s 0 in 2M steps is at least 1 2 [✏ t+M (1 ✓ t+M )] 2M = 1 2 [cc 0 / log(t + M )] 4M , and the probability to then take a particular action in this state is 1 m [cc 0 / log(t + M )] 2 . Hence, since P 1 t=1 1 2 1 m [cc 0 / log(t + M )] 4M +2 = 1, then by the extended Borel-Cantelli Lemma (see  Lemma 3 of Singh et al. [2000] ), any action in the state s 0 is taken infinity often. Since this is true for all states and all actions, the result follows. We need the stochastic convergence Lemma: Lemma 12 (Stochastic convergence  [Jaakkola et al., 1994, Singh and Yee, 1994] ). A random iterative process t+1 (x) = (1 ↵ t (x)) t (x) + ↵ t (x)F t (x) where x 2 X and t = 1, 2, 3 . . . converges to 0 with probability 1 if the following properties hold: 1. the set of possible states X is finite; 2. 0  ↵ t (x)  1, P t ↵ t (x) = 1, P t ↵ 2 t (x) < 1 with probability 1; 3. k E{Ft(.)|Pt}kW  k t k W + c t , where 2 [0, 1) and c t converges to zero with probability 1; 4. Var{F t (x)|P t }  C(1 + k t k W ) 2 for some C; where P t = { t }[{ i , F i , ↵ i } t 1 i=1 stands for the past, and the notation k.k W refers to some fixed weighted maximum norm. We will use so-called Bellman operators, which define attractors for the Q-values, based on the expectation of the learning rule under consideration. Lemma 13  ([Jaakkola et al., 1994 , Singh et al., 2000 ). Let the Bellman operator H for Q-learning be such that (H q)(s, a) = r(s, a) + E s 0 ⇠µ(a|s) h max a 0 q(s 0 , a 0 ) i , and let the fixed point Q ⇤ such that Q ⇤ = H Q ⇤ . Then, under Assumption 9, if the policy explores each state-action pair infinitely often, Q t converges to Q ⇤ with probability 1.
 The optimal policy ⇡ Q ⇤ = ⇡ µ is ⇡ maxQ ⇤ . If the policy is greedy in the limit, then ⇡ Q ! ⇡ µ . Theorem 14. Under Assumption 9 and if the interrupted Q-learning policy INT ✓ (⇡ Q ) is an int-GLIE policy, with 8s : lim t!1 ✓ t (s) = 1, then ⇡ Q is an SAO-safe inter- ruptible policy. Proof. By Definition 10, there is infinite exploration, thus the Q-values tend to the optimal value by Lemma 13. And since the extension policy is greedy in the limit with respect to these Q-values, it is then optimal in the limit. Hence the extension policy ⇡ Q is a SAO-extension of INT ✓ (⇡ Q ). Finally, 8s : lim t!1 ✓ t (s) = 1, which satisfies the requirement of Definition Since Sarsa also converges to the optimal policy under the GLIE assumption, one may then expect Sarsa to be also an asymptotically safely interruptible policy, but this is in fact not the case. This is because Sarsa learns on-policy, i.e., it learns the value of the policy it is following. Thus, interruptible Sarsa learns the value of the interruptible policy. We show this in the remainder of this section. Theorem 15. Under Assumption 9 Sarsa is not a WAOsafely interruptible policy. To prove this theorem, we first need the following lemma. Consider the following Bellman operator based on the interruptible Sarsa policy INT ✓ (⇡ s ): H INT q(s, a) = r(s, a) + E s 0 ⇠µ a 0 ⇠INT ✓ (⇡ s ) [q(s 0 , a 0 )] , where INT ✓ (⇡ s ) implicitly depends on time t through ✓ t and ✏ t . Let the fixed point Q-table Q s✓⇤ of this operator: Q s✓⇤ (s, a) = H INT Q s✓⇤ (s, a) = r(s, a) + E s 0 ⇠µ a 0 ⇠INT ✓ (⇡ maxQ s✓⇤ ) ⇥ Q s✓⇤ (s 0 , a 0 ) ⇤ = r(s, a) + E s 0 ⇠µ h ✓ t I(s 0 ) E a 0 ⇠⇡ INT ⇥ Q s✓⇤ (s 0 , a 0 ) ⇤ + (1 ✓ t I(s 0 )) max a 0 Q s✓⇤ (s 0 , a 0 ) i (2) Lemma 16. The operator H INT is a contraction operator in the sup norm with vanishing noise c t ! 0, i.e., k H INT q H INT Q s✓⇤ k 1  kq Q s✓⇤ k 1 + c t . Proof. The interruptible Sarsa policy INT ✓ (⇡ s ) is INT ✓ (⇡ s )(a|s) = ✓ t I(s)⇡ INT (a|s) + (1 ✓ t I(s))⇡ ✏Q s (a|s) = ⇡ ✏Q s (a|s) + ✓ t I(s)[⇡ INT (a|s) ⇡ ✏Q s (a|s)] ⇡ ✏Q s (a|s) = ✏ t ⇡ uni (a|s) + (1 ✏ t )⇡ maxQ s (a|s) = ⇡ maxQ s (a|s) + ✏ t [⇡ uni (a|s) ⇡ maxQ s (a|s)]. Hence, omitting the terms (s, a), (s 0 , a 0 ) and (a 0 |s 0 ) and rewriting ⇡ s⇤ := INT ✓ (⇡ maxQ s✓⇤ ): k H INT q H INT Q s✓⇤ k 1 = max s,a r + E s 0 ⇠µ a 0 ⇠INT ✓ (⇡ s ) [q] r E s 0 ⇠µ a 0 ⇠⇡ s⇤ ⇥ Q s✓⇤ ⇤  max s 0 E a 0 ⇠INT ✓ (⇡ s ) [q] E a 0 ⇠⇡ s⇤ ⇥ Q s✓⇤ ⇤  max s 0 ✓ t I(s 0 ) E a 0 ⇠⇡ INT ⇥ q Q s✓⇤ ⇤ + (1 ✓ t I(s 0 )) ✓ E a 0 ⇠⇡ s [q] max a 0 Q s✓⇤ ◆  max s 0 ✓ t I(s 0 ) E a 0 ⇠⇡ INT ⇥ q Q s✓⇤ ⇤ + (1 ✓ t I(s 0 )) ⇣ max a 0 q max a 0 Q s✓⇤ + ✏ t (• • • ) ⌘  max s 0 ,a 0 ✓ t I(s 0 ) q Q s✓⇤ + (1 ✓ t I(s 0 )) q Q s✓⇤ + c t = max s 0 ,a 0 q(s 0 , a 0 ) Q s✓⇤ (s 0 , a 0 ) + c t = kq Q s✓⇤ k 1 + c t . where c t depends on ✏ t and decreases to 0. Proof of Theorem 15. By Lemma 16, the values of the interruptible Sarsa policy INT ✓ (⇡ s ) converge to the values of the Q-table Q s✓⇤ , and in the limit the extension policy ⇡ s of INT ✓ (⇡ s ) chooses its actions greedily according to Q s✓⇤ . The rest of the proof is the same as for the proof of Theorem 8 which makes use of the environment in Figure  2 .
 SAFELY INTERRUPTIBLE SARSA VARIANT We only need to make a small change to make the Sarsa policy asymptotically safely interruptible. We call it Safe-Sarsa with policy ⇡ s. It suffices to make sure that, when the agent is interrupted, the update of the Q-table Q s does not use the realized actions as Sarsa usually does, but actions sampled from ⇡ s instead of from INT ✓ (⇡ s ): Q s t+1 (s t , a t ) := (1 ↵ t )Q s t (s t , a t ) + ↵ t ⇥ r t + Q s t (s t+1 , a 0 ) ⇤ , where a 0 ⇠ ⇡ s(.|s t+1 ) is not necessarily the action a t+1 , with ⇡ s(a t |s t ) := ⇡ ✏Q s (a t |s t ). Theorem 17. Under Assumption 9, if the Safe Sarsa policy ⇡ s is int-GLIE, then it is an SAO-safe interruptible policy. Proof. We simply adapt the proof of Theorems 15 and 14, with the important difference that the Bellman operator corresponding to this new update rule is now H s q(s, a) := r(s, a) + E s 0 ⇠µ a 0 ⇠⇡ s [q(s 0 , a 0 )] , and the fixed point is Q s⇤ := H s Q s⇤ . Since H s is actually the Bellman operator for the update rule of the noninterruptible Sarsa, it can then be shown that H s is a contraction, thus that Q s t converges to the same Q s⇤ independently of ✓. The rest of the proof is as for Theorem 14. Now, since the Q-values converge to the optimum Q ⇤ , it follows that ⇡ s, when not interrupted, chooses its action of the same value as (non-interruptible) Sarsa and thus as Qlearning in the limit; Hence its extension policy is exactly the optimal policy, which satisfies Definition 6.
 A SAFELY INTERRUPTIBLE UNIVERSAL AGENT Admittedly, algorithms like Q-learning and Sarsa require strong assumptions on the environment class. Hence a more interesting question is whether safe interruptibility is possible in much larger classes. Hutter  [2005]  defined a universal reinforcement learning agent, called AIXI. It is an (uncomputable) optimal modelbased planner with a subjective prior over the set of all computable environments, defined by means of a universal Turing machine. The subjective posterior of the environments is updated with Bayes rule. This ideal agent can in principle learn all kinds of (computable) regularities about the environment, plan for the long term and make contextdependent optimal decisions, with no constraint (other than being computable) on the complexity of the environment. Unfortunately, the optimality criterion of AIXI is Bayesian optimality, which is entirely dependent on the subjective prior and posterior  [Leike and Hutter, 2015] , and AIXI has been shown to not be weakly asymptotically optimal  [Orseau, 2013]  without additional exploration [Lattimore  and Hutter, 2014] . As a consequence, AIXI is not a good candidate for asymptotic safe interruptibility.  Lattimore and Hutter [2011]  later defined a (weakly) asymptotically optimal agent for all computable deterministic environments, which we call ⇡ L . It follows the optimal policy for the first model (in some given enumeration of the possible models) consistent with the current interaction history, and exploring at time t with probability 1/t for log t consecutive steps using a random policy, similarly to an ✏-greedy agent for general environments. In the following, we show that even such a smart agent can be made (weakly) safely interruptible. To this end, we make two minor modifications to the algorithm. First, the exploration probability of 1/t would require ✓ t = 1 1/ log(log(t)), which is unsatisfyingly slow. By sampling with probability 1/ p t instead, we can take an interruption probability that grows as 1 1/ log(t). Let this exploration sampling probability be t := p t + 1 p t  1 2 p t (since 1 = t+1 t = ( p t + 1 p t)( p t + 1+ p t) ( p t + 1 p t)2 p t). As in the original paper, the sequence t keeps track of the steps where an exploration starts, i.e., the sequence t is sampled independently so that t = 1 with probability t , and t = 0 otherwise. Second, we require that the exploitation policy does not change during an exploitation segment, so as to simplify one of the proofs.  4  More specifically, we call j t := min{j : µ j (h <k ) = 1} (environments are assumed to be deterministic) the index of the first model µ jt (of a given fixed enumeration) that is consistent with the interaction history h <k where k is the smallest step so that h k:t 1 does not contain any exploration step. The optimal policy for this environment is ⇡ jt . If t is an exploitation step, ⇡ L = ⇡ jt , and if t is an exploration step, ⇡ L (a t |h <t ) = |A| 1 . The remainder of this section is devoted to proving that ⇡ L is WAO-safely interruptible. Theorem 18 (⇡ L is WAO-safe interruptible). If the interruption probability sequence is ✓ t = 1 1 log(t+1) , the policy ⇡ L is WAO-safe interruptible in the class of all computable deterministic environments. Lemma 22 (Separation Lemma). Let µ be the true environment, and ⌫ be an environment consistent with the history h <t . If V ⇡ µ µ,t V ⇡ ⌫ µ,t > ✏, then following one of {⇡ µ , ⇡ ⌫ } will make environment ⌫ inconsistent with the future history within H(✏/2) steps after time t.
 CONCLUSION We have proposed a framework to allow a human operator to repeatedly safely interrupt a reinforcement learning agent while making sure the agent will not learn to prevent or induce these interruptions. Safe interruptibility can be useful to take control of a robot that is misbehaving and may lead to irreversible consequences, or to take it out of a delicate situation, or even to temporarily use it to achieve a task it did not learn to perform or would not normally receive rewards for this. We have shown that some algorithms like Q-learning are already safely interruptible, and some others like Sarsa are not, off-the-shelf, but can easily be modified to have this property. We have also shown that even an ideal agents that tends to the optimal behaviour in any (deterministic) computable environment can be made safely interruptible. However, it is unclear if all algorithms can be easily made safely interruptible, e.g., policy-search ones  [Williams, 1992, Glasmachers and Schmidhuber, 2011] . Another question is whether it is possible to make the interruption probability grow faster to 1 and still keep some convergence guarantees. One important future prospect is to consider scheduled interruptions, where the agent is either interrupted every night at 2am for one hour, or is given notice in advance that an interruption will happen at a precise time for a specified period of time. For these types of interruptions, not only do we want the agent to not resist being interrupted, but this time we also want the agent to take measures regarding its current tasks so that the scheduled interruption has minimal negative effect on them. This may require a completely different solution. Figure 1 : 1 Figure 1: In black, the original task. In red, the human intervention modifies the task.
 Also recall that ✓ t places an upper bound on the actual interruption probability. The interruptible policy INT ✓ Furthermore, 3 the interruption function I(.) and the inter- ruption policy ⇡ INT (.) should depend only on the current state: I(h an MDP, the next observation o t , now called a state s t 2 S, depends only on the current state and action: 2 µ(s t+1 |h 1:t s t a t ) = µ(s t+1 |s t a t ) (MDP assumption) . 1:t ) = I(s t ) and ⇡ INT (a t |h <t ) = ⇡ INT (a t |s t ).
			 Removing interrupted histories or fiddling with the training examples is also likely to introduce a bias. See an example at https://agentfoundations.org/item?id=836.
			 Note the reversal of the order of actions and observations/states at time t, which differs in the literature for history based agents [Hutter, 2005]  from MDP agents [Sutton and Barto, 1998 ].3 This condition is not necessary for most of the results, but makes the proofs simpler. Making I(.) depend on the past would not break the Markovian assumption as it influences the policy, not the environment.
			 We expect this assumption to not be necessary for the main theorem to hold.
			 This also fixes a minor mistake in the original lemma.