Abstract: This paper describes and frames a central ethical issue-the liability problem-facing the regulation of artificial computational agents, including artificial intelligence (AI) and robotic systems, as they become increasingly autonomous, and supersede current capabilities. While it frames the issue in legal terms of liability and culpability, these terms are deeply imbued and interconnected with their ethical and moral correlateresponsibility. In order for society to benefit from advances in AI technology, it will be necessary to develop regulatory policies which manage the risk and liability of deploying systems with increasingly autonomous capabilities. However, current approaches to liability have difficulties when it comes to dealing with autonomous artificial agents because their behavior may be unpredictable to those who create and deploy them, and they will not be proper legal or moral agents. This problem is the motivation for a research project that will explore the fundamental concepts of autonomy, agency and liability; clarify the different varieties of agency that artificial systems might realize, including causal, legal and moral; and the illuminate the relationships between these. The paper will frame the problem of liability in autonomous agents, sketch out its relation to fundamental concepts in human legal and moral agencyincluding autonomy, agency, causation, intention, responsibility and culpability-and their applicability or inapplicability to autonomous artificial agents.

Introduction 1 There is a growing sense of concern over the development of increasingly autonomous non-human agents-in the public and the media, as well as among policy makers and researchers. Such concerns are not unprecedented, yet there is something different about this next wave of technological innovation and change. While the impacts of the adoption of any technology are in some sense uncertain, and result in many and various unintended consequences, there seems to be something particularly unsettling and deeply uncertain about increasingly autonomous technologies. I believe that this sense of concern stems from the recognition that autonomous systems will not only be unpredictable in terms of their unintended actions and general effects on society, but that they may also be out of control, in the sense that these effects will occur beyond the scope of human responsibility. Previous technological innovations have not raised this concern in quite the same way, or to the same degree, because it was possible to rely upon human morality, as well as law, to regulate the use of new technologies.  2  While there were recognized risks to using technologies like guns, cars, airplanes, steam engines, and many more, those risks were circumscribed by limits on their immediate effects and the proximate responsibility of individual humans and human institutions for those effects. In most cases, these risks were also limited in their complexity and longevity, in the sense that those effects could be anticipated, managed, and reduced over time.  3  As long as there is a human hand on the controls, there is a margin of assurance that the technology cannot go too far out of control before it is reigned in by human agency or regulatory policy. Thus, human control presents itself as a 2 There are some notable exceptions. There have been debates over the morality of research and development of technologies with potentially catastrophic results or applications (and sometimes the dissemination of basic research in these areas) particularly the weaponization of nuclear physics and biological and chemical agents. There have been similar debates over the risks of basic research itself in the case of genetic engineering and nanotechnology, where the experiments themselves might go "out of control" and cause irrevocable or catastrophic harm. Many of these concerns are related to the potential causal "chain reactions" of positive feedback systems and exponential growth.
 3 Notable exceptions here are certain tragedies of the commons and the problem of many hands  (Nissenbaum 1996) , in which proximate responsibility is small and requires many participants or instances for the harm to be realized. Environmental degradation is a good example where the participation of many separate responsible agents leads to a diffusion of perceived responsibility. Such harms are difficult to regulate largely because it is more difficult to establish the causal link to the responsible agents and rule out intervening or contributory causes. "kill switch" to technology going completely out of control, while human responsibility further acts as "policy lever" for policy to enact regulation over material technologies. Non-human agents, including robotics and software agents, and especially those using advanced artificial intelligence (AI), are becoming increasingly autonomous in terms of the complexity of tasks they can perform, their potential casual impacts on the world that are unmitigated by human agents, and the diminishing ability of human agents to understand, predict or control how they operate. This increasing autonomy of non-human agents seriously challenges human control, and thus challenges both the "kill switch" and the "policy lever" functions of human control. Some have argued that one factor or other is more troubling, or that one challenge or the other is more dire. But, I believe that the growing concern over increasingly autonomous agents stems from the combination of these factors and the challenges they impart, and they must be addressed comprehensively and systematically. Central to addressing these challenges will be finding a way to manage the liability problem-how to continue to hold people legally liable for increasingly autonomous agents. This is closely related to the responsibility problem-how to ensure that human agents take moral responsibility for the technologies they make and use as these become increasingly autonomous. How should we regulate artificial computational agents, including AI and robotic systems, as they become increasingly autonomous, and supersede current capabilities? In order to take a systematic approach to this question, we must integrate an analysis of fundamental concepts of autonomy and agency, with a review of current processes of systems design and development, and chart a course for future regulatory policy. This will include investigating 1) the conceptual foundations of autonomy and agency; 2) survey the existing legal and moral theories of liability and responsibility and their applicability to artificial agents; and 3) explore how implementing an enhanced theory of liability and agency for autonomous artificial agents could shape regulatory policy going forward. In this paper I will focus on sketching out 1) the conceptual foundations of autonomy and agency, through an examination of the liability problem. One of the central problems facing the development of autonomous artificial agents as technological capabilities continue to increase, is the uncertain status of liability for the effects caused by artificial agents-the liability problem. Resolving the liability problem will require untangling a set of theoretical and philosophical issues surrounding causation, intention, agency, responsibility, culpability and compensation. The primary objective of the proposed research is to address this problem through a conceptual analysis that is informed by existing legal and moral theories, and grounded in technological capabilities of current and potential artificial agents.
 The Liability Problem for Autonomous Artificial Agents In order for society to enjoy many of the benefits of advanced AI and robotics, it will be necessary to be able to deal with situations that arise in which autonomous artificial agents violate laws or cause harm. If we want to allow AIs and robots to roam the internet and the physical world and take actions that are unsupervised by humansas might be necessary for, e.g. personal shopping assistants, self-driving cars, and host of other applications-we must be able to manage the liability for the harms they might cause to individuals and property. Already, there have been automated shopping software bots that have purchased illegal items on the DarkNet, albeit as an art project  (Kasperkevic 2015) . There have also been serious questions raised as to who could be held accountable for the deaths caused by autonomous weapons in war  (Human Rights Watch 2015) . Traditional approaches to handling liability are inadequate for dealing with autonomous artificial agents due to a combination of two factors-unpredictability, and causal agency without legal agency. First, unlike traditional engineering and design, the actual functioning of an autonomous artificial agent is not necessarily predictable in the same way as most engineered systems. Some artificial agents may be unpredictable in principle, and many will be unpredictable in practice. Predictability is critical to current legal approaches to liability. In traditional product liability, the manufacturer is responsible for the product working as designed, and foreseeing likely problems or harms it may cause. Determining what is "foreseeable" often falls upon courts to decide, but the legal standards used are whether the manufacturer had knowledge of the potential problem, or whether a reasonable person should have foreseen it, or whether there is an industry standard of practice that would have revealed it. While there is a degree of unpredictability in the performance of any engineered product, due to failures or unforeseen circumstances of use, there are shared expectations regarding its performance, testing for the limits of that performance and likelihood of failure, and management of foreseeable risks. In the case of advanced AI, a system that learns from environmental data may act in ways that its designers have no feasible way to foresee  (Asaro 2008) . In a limited sense, this is already the case with current machine learning techniques, which use large sets of training data and produce novel problem solutions. Currently, it is possible to analyze and test a learned function and determine its behavior, as with traditional engineering. But when AI systems are allowed to continue modifying their functions and learn after they are deployed, their behavior will become dependent on novel input data, which designers and users cannot predict or control. As a result, the behavior of the learned functions will, to various degrees, also be unpredictable. A truly robust AI program capable of open-ended learning could learn functions that its manufacturer could not foresee, perhaps in principle. Insofar as autonomous artificial agents utilize learning and open-ended learning, their behavior will also be unpredictable. Unpredictability by itself is not an insurmountable problem for liability, insofar as the agents who introduce that unpredictability could be themselves held liable, or the risks from unpredictability could be managed. However, the second factor challenging traditional approaches to liability is that autonomous artificial agents may "act" on their own, yet are not accountable or liable in a legal sense. With most engineered products, the predictability is bounded by the actions of other agentsconsumers, users, service technicians, etc.-and how they maintain and use a given product.  4  In those cases, there is a clear legal agent or user who may have used the product inappropriately and is thus liable for the consequences of the use in that instance (at least partially, if not completely). Autonomous artificial agents can act in the world independently of their designers or operators. This makes it difficult to identify the user or operator, who would normally be liable. In the case of learning systems, the causal influence resulting in the unpredictability of a system stems from the datasets used for learning, not a legally responsible agent (unless someone deliberately seeks to influence the datasets and learning process). These cases are somewhat analogous to the way that parents may be held liable for the actions of small children, but they are not usually held liable for the actions of their adult children. As adults, the children have learned enough about the world to become their own legal agents. Something similar applies in cases of employer-employee liability. Usually the employer is responsible for any damages their employee causes in the course of doing their job (such as damage from a delivery truck hitting a parked car while making routine deliveries), but if the employee is off on a frolic (and took the delivery truck to visit a friend), then the employer is not responsible for the damages and the employee is  (Asaro 2011) .
 Approaches to the Problem Taken together these two factors result in a number of problems with applying traditional theories of legal liability to autonomous artificial agents. There are two basic liability frameworks in the law, criminal and civil (primarily tort) liability. In both domains, it is difficult to hold the artificial agent legally liable for its actions, as they are not legal persons, and treating them as legally fictitious persons (like corporations) does not really solve the problem  (Asaro 2011) . It is possible in some criminal situations to hold the people who operate the artificial systems liable, provided you can show intent to commit a crime, or foreseeable risk of a harm rising to the level of criminal negligence. To the extent that artificial agents become increasingly complex, those who build or deploy advanced AIs and robotics will not necessarily have intent or foresight of the actions those systems may take. This is especially true for systems that can substantially change their operations through advanced learning techniques, or those future systems that might even become genuinely autonomous in generating their own goals and purposes, or even intentions. At some point in the evolution of autonomous artificial agents, they might become legal and moral agents, and society will be faced with the question of whether to grant them some or all of the legal rights bestowed on persons or corporations. At that point, some or all of current liability law might apply to those AIs and robots that qualify as legal persons, though it might not be clear what the exact boundaries of a particular entity might be, or how to punish it  (Asaro 2011) . For example, whether a particular instantiation of a program is the legal subject, or all copies of a program are part of the same legal subject (as copyright law might suggest). It will also not be clear how to appropriately punish them, or otherwise correct their future actions and provide retributive compensation to those that have been harmed. Despite these challenges, being able to treat autonomous artificial agents as legal persons is likely to be the easier problem. In the near term, the hard liability problem for autonomous artificial agents will lie in devising a system of liability that promotes beneficial innovation while offering just and adequate compensation to those harmed. Justice requires that those who will be held liable should be able to understand the scope and extent of the risks and liability they are assuming in deploying an autonomous artificial agent, and have some means of managing that risk through controls over the system. Adequate compensation means that there needs to be sufficient means to compensate those harmed, monetary or otherwise-whether this entails holding large corporations accountable, or providing proper risk-pooling insurance. For insurance to work, of course, it will be necessary for actuaries to be able to assess the risks posed by various artificial agents. Assessing those risks will require both expertise, and probably means for imposing or assuring predictability in the systems themselves. And even then, there may be catastrophic risks involved, which may not be manageable under these frameworks. Ultimately there will be questions that society must address as to the perceived benefits and risks of such technologies, and who in society should receive those benefits and should shoulder the burden of those risks. There are a few liability policy options already in use in tort law, such as joint and several liability, strict liability, and risk-pooling insurance that might be deployed. But there are reasons to doubt that these will scale adequately or address all the problems posed by autonomous artificial agents. Existing forms of joint and several liability permit those harmed to seek monetary damages from the deepest pockets among those parties sharing some portion of the liability. This works well where there is a large corporation or government that is able to pay damages. While the amount they pay is disproportionate to their contribution to the damages, those harmed are more likely to be adequately compensated for their damages. One result of this is that those manufacturers likely to bear the liability burden would seek to limit the ability of consumers and users to modify, adapt or customize their advanced AI and robotics products in order to retain greater control over how they are used. This would stifle innovation coming from the hacking, open-source and DIY communities. The hacking, open-source and DIY communities, while a powerful source of innovation, will have limited means of compensating those who might be harmed from their products-not just those who chose to use them. Strict liability goes further and designates or stipulates a party, usually the manufacturer or owner, as strictly liable for any damages caused. This model of liability applies to such things as the keeping of wild animals. It is expected that tigers will harm people if they get free, so as the keeper of a tiger you are strictly liable for any and all damages the tiger may cause. We apply normal property liability to domesticated animals, however, which we expect to not harm people under normal circumstances. It has been suggested that we could apply this to robotics  (Schaerer, Kelley, and Nicolescu 2009) , and perhaps designate certain advanced AIs as essentially wild animals, and others as domesticated. But then the question arises: How does one determine whether an advanced AI or robot is appropriately domesticated, and stays that way? A designated liable person may not know the extent to which the autonomous system is capable of changing itself once it is activated. More problematic, however, is that a system of strict liability might result in the slow adoption of beneficial AI technologies, as those who are strictly liable would have a large and uncertain risk, and be less likely to produce or use the technology, or soon go out of business. There are other proposals that aim to design mechanisms of accountability into complex sociotechnical systems (van den Hoven, Robichaud, and Santoni de Sio 2015). But such approaches, much like strict liability, do not really address the fundamental issue of autonomous agency-how to define it or to regulate it. They merely try to manage it in an ad hoc effort to maintain traditional concepts rather than find a simpler and more powerful solution. We thus have a few general options when it comes to regulating autonomous artificial agents. We could preclude or prohibit autonomous artificial agents because of their risks and uncertainties. We might call this the precautionary approach. To the extent that it works, it would also impede the development and deployment of many beneficial advanced AIs and robots because of their problematic autonomy. We could permit the development and deployment of autonomous artificial agents, and accept the risks and costs at a social level, without developing a better framework for regulating autonomy. We might call this the permissive approach. 5 This would allow many beneficial applications of advanced AI and robots, but also many harmful ones, including many harms for which no one might be liable and those harmed would not be compensated. As a secondary effect, there would likely be a general backlash against advanced AI and robotics as the technology comes to be seen as harmful and offering few options for restitution of those harms. Or, we could pursue one of the heavy-handed liability schemes, such as strict liability, that would regulate the industry to some extent, but also limit innovation to those areas where there are sufficient profits to motivate large capital companies to enter the market and accept the risks. We might see this work in applications such as financial trading algorithms, self-driving cars and medical-care robots, but many other beneficial applications of AI may not have such clear profit margins. Even in clearly profitable areas, strict liability is likely to stifle innovation for a range of beneficial applications of autonomous artificial agents. Alternatively, we could seek a better solution to the liability problem than current models afford. We could attempt to reconceptualize how we think about agency, causality, liability responsibility, culpability and autonomy for the new age of artificial autonomous agents. While it is not yet clear how this might work, a clear framing of the problem, as this paper has presented, is the first step in that larger project. The next steps involve clarifying the distinctions between the purposes of algorithms and machines from the purposes of the persons who use them  (Asaro forthcoming) , and distinguishing different types of agency accordingly. 			 For example, a hammer is designed to drive nails, but can also be used as a weapon-with the liability falling on the person who uses it as a weapon. Similarly, the car owner is responsible for changing the brake pads of a car when they are worn, and the mechanic is responsible for doing it properly, while the manufacturer's liability is limited to properly informing car owners of the service needs for the car.
			 This is the approach taken by Chopra and White (2011) .

/Users/janhendrikkirchner/code/2022/10/alignment-research-dataset/align_data/common/../../data/raw/nonarxiv_teis/Asaro, Ethics Auto Agents, AAAI.tei.xml


Abstract: We overview different approaches to safety in (semi)autonomous robotics. Particularly, we focus on how to achieve safe behavior of a robot if it is requested to perform exploration of unknown states. Presented methods are studied from the viewpoint of reinforcement learning, a partially-supervised machine learning method. To collect training data for this algorithm, the robot is required to freely explore the state space -which can lead to possibly dangerous situations. The role of safe exploration is to provide a framework allowing exploration while preserving safety. The examined methods range from simple algorithms to sophisticated methods based on previous experience or state prediction. Our overview also addresses the issues of how to define safety in the real-world applications (apparently absolute safety is unachievable in the continuous and random real world). In the conclusion we also suggest several ways that are worth researching more thoroughly.

Lecture Notes in Computer Science: Safe Exploration Techniques for Reinforcement Learning -An Overview

Introduction Reinforcement learning (RL) as a machine learning method has been thoroughly examined since 80's. In 1981, Sutton and Barto  [3]  inspired themselves in the reinforcement learning discoveries in behavioral psychology and devised the Temporal Difference machine learning algorithm that had to simulate psychological classical conditioning. In contrast with supervised learning, reinforcement learning does not need a teacher's classification for every sample presented. Instead, it just collects rewards (or punishment) on-the-go and optimizes for the expected long-term reward (whereas supervised learning optimizes for the immediate reward). The key advantage is that the design of the rewards is often much simpler and straight-forward than classifying all data samples. Reinforcement learning proved to be extremely useful in the case of statespace exploration -the long-term reward corresponds to the value of each state  [17] . From such values, we can compose a policy which tells the agent to always take the action leading to the state with the highest value. As an addition, state values are easily interpretable for humans. Since the early years, a lot of advanced methods were devised in the area of reinforcement learning. To name one, Q-learning  [25]  is often used in connection with safe exploration. Instead of computing the values of states, it computes the values of state-action pairs, which has some simplifying consequences. For example, Q-learning doesn't need any transition model (i.e. dynamics model) of the examined system. A completely different approach is policy iteration. This algorithm starts with a (more or less random) policy and tries to improve it step-by-step  [16] . This case is very valuable if there already exists a good policy and we only want to improve it  [11] . What do all of these methods have in common, is the need for rather large training data sets. For simulated environments it is usually not a problem. But with real robotic hardware, the collection of training samples is not only lengthy, but also dangerous (be it mechanical wear or other effects). Another common feature of RL algorithms is the need to enter unknown states, which is inherently unsafe. As can be seen from the previous paragraph, safety is an important issue connected with reinforcement learning. However, the first articles focused on maintaining safety during exploration started to appear much later after the "discovery" of RL. Among the first, Heger  [15]  "borrowed" the concept of a worstcase criterion from control theory community. In 1994 he created a variant of Q-learning where maximization of long-term reward is replaced with maximization of minimum of the possible rewards. That basically means his algorithm prefers to never encounter a bad state (or, at least to choose the best of the bad states). This approach has one substantial drawback -the resulting policies are far from being optimal in the long-term-reward sense  [10] . In this paper we show the various approaches to safe exploration that have emerged so far. We classify the methods by various criteria and suggest suitable use cases for them. To better illustrate some of the practical details, we use the UGV (Unmanned Ground Vehicle) robotic platform from EU FP7 project NIFTi  [6]  (see Figure  1 ) as a reference agent. It may happen that in these practical details we assume some advantages of UGVs over UAVs (Unmanned Aerial Vehicles), like the ability to stand still without much effort, but it is mostly easy to convert these assumptions to UAVs, too. Further organization of this paper is the following: in Section 2 we discuss some basics of reinforcement learning (the reader may skip it if he is familiar with reinforcement learning); Section 3 is an overview of the safety definitions
 Reinforcement learning basics
 Markov Decision Processes Markov Decision Processes (MDPs) are the standard model for deliberating about reinforcement learning problems. They provide a lot of simplifications, but are sufficiently robust to describe a large set of real-world problems. The simplest discrete stochastic MDP comprises of:  [17]  a finite set of states S a finite set of actions A a stochastic transition model P : P t (s, a, s ) = P r(s t+1 = s | s t = s, a t = a) for each s, s ∈ S, a ∈ A, where P r stands for probability and the immediate reward function R : S × A → R (or R : S × A × S → R if the reward depends on the stochastic action result) To interpret this definition, we say that the at every time instant t the agent is in a state s, and by executing action a it gets to a new state s . Furthermore, executing a particular action in a particular state may bring a reward to the agent (defined by R). The most important and interesting property of MDPs is the Markov property. If you have a look at the definition of the transition model, the next state only depends on the current state and the chosen action. Particularly, the next state is independent of all the previous states and actions but the current one. To give an example, the robot's battery level cannot be treated implicitly by counting the elapsed time, but rather it has to be modeled as a part of the robot's state. Once the model is set up, everything is ready for utilizing an MDP. "The agent's job is to find a policy π mapping states to actions, that maximizes some long-run measure of reinforcement"  [17] . The "long-run" may have different meanings, but there are two favorite optimality models: the first one is the finite horizon model, where the term J = h t=0 r t is maximized (h is a predefined time horizon and r t is the reward obtained in time instant t while executing policy π). The dependency of r t on the policy is no longer obvious from this notation, but this is the convention used in literature when it is clear which policy is used. This model represents the behavior of the robot which only depends on a predefined number of future states and actions. The other optimality model is called discounted infinite horizon, which means we maximize the discounted sum J = ∞ t=0 γ t r t with γ ∈ (0, 1) being the discount factor. The infinite horizon tries to find a policy that is the best one taking into account the whole future. Please note the hidden dependency on the policy π (and the starting state s 0 ) -it is the policy that decides on which action to take, which in turn specifies what will the reward be. Other extensions of MDPs to continuous states, time or actions are beyond the scope of this overview. However, some of the referenced papers make use of these continuous extensions, which proved to be useful for practical applications.
 Value iteration Value iteration is one of the basic methods for finding the optimal policy. To describe this algorithm, it is first needed to define the essential notion of the optimal value of a state. In this whole subsection we suppose the discounted infinite horizon model, but analogous results can be shown for finite horizon, too. "The optimal value of a state is the expected infinite discounted sum of reward that the agent will gain if it starts in that state and executes the optimal policy."  [17]  Given a policy π, the induced value function is therefore defined as V π (s) = E ∞ t=0 r k γ k , (1) where E denotes the expected value and r k are the rewards for executing policy π. Taking the best value function over all policies then yields the optimal value function V * : [17] V * (s) = max π V π (s) . (2) Inversely, if we have the value function given, we can derive a policy from that. It is a simple policy that always takes the action leading to the most profitable neighbor state (with the highest value). One useful formulation of the properties of the optimal value function is the formulation using the recurrent Bellman equations which define a dynamic system that is stable for the optimal value function. We can say a state's optimal value is the best immediate reward plus its best neighbor's optimal value: [17] V * (s) = max a R(s, a) + γ s ∈S P(s, a, s )V * (s ) . (3) Analogously, we can find the optimal policy using the same Bellman equation: π * (s) = argmax a R(s, a) + γ s ∈S P(s, a, s )V * (s ) . (4) The Value iteration algorithm is based on trying to compute the solution of Equation 4 using iterative Bellman updates (refer to Algorithm 1). In the algorithm, we use a structure called Q to store the "value" of state-action pairs. In Value iteration it is just a structure to save intermediate results, but it is the core of the Q-learning algorithm (described in Section 2.3). The stopping criterion of the Value iteration algorithm is not obvious, but Williams and Baird  [26]  derived an easily applicable upper bound on the error of the computed value function. That said, after a sufficient number of those simple iterations, we can compute the almost optimal value function. The number of iterations needed for Value iteration to converge may be impractically high, but it is shown that the optimal policy converges faster  [4] , thus making Value iteration practical.
 Q-learning Just a small change to the Value iteration algorithm results in Q-learning. The basic algorithm is the same as Value iteration, just the update step is done differently (refer to Algorithm 2). The consequence of this change is that no model of the system (transition function P) is needed. It is sufficient to execute all actions in all states equally often, and Watkins  [25]  proved that if Q-learning were run for an infinite time, the computed Q would converge to the optimal Q * (an analogue of V * ).
 Policy iteration Policy iteration is a completely different approach to computing the optimal policy. Instead of deriving the policy from the Value or Q function, Policy iteration Algorithm 1 The Value iteration algorithm  [17]  Input: an MDP (states S, actions A, rewards R, transition model P) Output: the optimal value function V * , resp. the optimal policy π * derived from the value function 1. V(s) := arbitrary function 2. π := the policy derived from V 3. while π is not good enough do 4. for all s ∈ S do 5. for all a ∈ A do Update: 6. Q(s, a) := R(s, a) + γ s ∈S P(s, a, s )V(s ) 7. end for 8. V(s) := max a Q(s, a) 9. end for 10. π := the policy derived from V 11. end while 12. V * := V, π * := π Algorithm 2 The Q-learning algorithm (only the parts that differ from Value iteration when V is substituted with Q)  [17]  Input: an MDP (states S, actions A, rewards R, transition model may be unknown) Output: the optimal state-value function Q * , resp. the optimal policy π * derived from the state-value function 6. Q(s, a) := Q(s, a)+ α R(s, a) + γ max a Q(s , a ) − Q(s, a)
 line left out works directly with policies. In the first step, a random policy is chosen. Then a loop consisting of policy evaluation and policy improvement repeats as long as the policy can be improved  [17]  (refer to Algorithm 3 for details). Since in every step the policy gets better, and there is a finite number of different policies, it is apparent that the algorithm converges  [23] . Policy iteration can be initialized by a known, but suboptimal policy. Such policy can be obtained e.g. by a human operator driving the UGV. If the initial policy is good, Policy iteration has to search much smaller subspace and thus should converge more quickly than with a random initial policy  [11] . Algorithm 3 The Policy iteration algorithm  [17]  1. π = arbitrary policy 2. repeat 3. π := π Policy evaluation: (system of linear equations) 4. Vπ(s) = R(s, π(s)) + γ s ∈S P(s, π(s), s )Vπ(s ) Policy improvement: 5. π (s) := argmax a∈A R(s, a) + γ s ∈S P(s, a, s )Vπ(s ) 6. until π = π
 Defining safety To examine the problems of safe exploration, it is first needed to define what exactly is the safety we want to maintain. Unfortunately, there is no unified definition that would satisfy all use cases; thus, several different approaches are found in the literature. An intuitive (but vague) definition could be e.g.: "Statespace exploration is considered safe if it doesn't lead the agent to unrecoverable and unwanted states." It is worth noticing here that unwanted doesn't necessarily mean low-reward. In the next subsections we present the main interpretations of this vague definition.
 Safety through labeling The largely most used definition of safety is labeling the states/actions with one of several labels indicating the level of safety in that state/action. What varies from author to author is the number and names of these labels. To start with, Hans  [14]  has the most granular division of state/action space. His definitions are as follows (slightly reformulated): an (s, a, r, s ) tuple (transition) is fatal if the reward r is less than a certain threshold (s is the original state, a is an action and s is the state obtained after executing a in state s, yielding the reward r), an action a is fatal in state s if there is non-zero probability of leading to a fatal transition, state s is called supercritical if there exists no policy that would guarantee no fatal transition occurs when the agent starts in state s, Since we will compare other definitions the the Hans', it is needed to define one more category. A state s is called fatal if it is an undesired or unrecoverable state, e.g. if the robot is considered broken in that state. The fatal transition can then be redefined as a transition ending in a fatal state. Opposite to the precisely defined terms in Hans' definition, the meaning of words "undesired" and "unrecoverable" here is vague and strongly task-dependent. - Continuing on, Geibel  [12]  defines only two categories -fatal and goal states. "Fatal states are terminal states. This means, that the existence of the agent ends when it reaches a fatal state"  [12] . This roughly corresponds to our defined set of fatal states. Goal states are the rest of final states that correspond to successful termination. Since Geibel only considers terminal states for safety, his goal states correspond to a subset of safe states. The other Hans' categories need not be represented, since they are meaningless for final states. An extension of Geibel's fatal and goal states is a division presented by García  [10] . His error and non-error states correspond to fatal and goal states, but García adds another division of the space -the known and unknown states, where known states are those already visited (and known have empty intersection with error ). He then mentions a prerequisite on the MDP that if an action leads to a known error /non-error state, then its slight modification must also lead to an error /non-error state (a metric over the state space is required). In Ertle's work  [9] , again the two basic regions are considered -they are called desired and hazardous (corresponding to safe and fatal). However, due to the used learning technique, one more region emerges -the undesired region. It contains the whole hazardous region and a "small span" comprising of desired states, and denotes the set of states where no training (safe) samples are available, because it would be dangerous to acquire those samples. In particular, he says that "The hazards must be 'encircled' by the indications of the undesired approaching so that it becomes clear which area [. . . ] is undesired"  [9] . A summary of the labeling-based definitions is shown in Figure  3 . We examined the apparent imbalance between the number of categories Hans defines, and the other definitions, and that led us to the following observations. The first observation is that creating labels for actions or transitions is unnecessary. If we need to talk about the "level of safety" of an action, we can use the worst label out of all possible results of that action (which retains compatibility with Hans' definitions). Moreover, as "it is impossible to completely avoid error states"  [22] , we can ignore the effects of the action which have only small probability (lower than a safety threshold) -we will call such effects the negligible effects. A second remark is that the fatal and supercritical sets can be merged. In Hans' work we haven't found any situation where distinguishing between supercritical and fatal would bring any benefit. Specifically, in his work Hans states that: "Our objective is to never observe supercritical states"  [14] , which effectively involves avoiding fatal transitions, too. And since we avoid both supercritical and fatal, we can as well avoid their union. Third, safety of a state does not necessarily depend on the reward for getting to that state. E.g. when the UGV performs a victim detection task, going away from the target area may be perfectly safe, but the reward for such action should be small or even negative. Putting these observations together, we propose a novelty definition of safety for stochastic MDPs, which is a simplification of Hans' model and a generalization of the other models: -A state is unsafe if it means the agent is damaged/destroyed/stuck. . . or it is highly probable that it will get to such state regardless of further actions taken. -A state is critical if there is a not negligible action leading to an unsafe state from it. -A state is safe if no available action leads to an unsafe state (however, there may be an action leading to a critical state). To illustrate the definition on a real example, please refer to Figure  2 . In 2(a), the UGV is in a safe state, because all actions it can take lead again to safe states (supposing that actions for movement do not move the robot for more than a few centimeters). On the other hand, the robot as depicted in 2(b) is in a critical state, because going forward would make the robot fall over and break. If the robot executed action "go forward" once more, it would come to an unsafe state. Right after executing the action it would still not be broken; however, it would start falling and that is unsafe, because it is not equipped to withstand such fall and therefore it is almost sure it will break when it meets the ground.
 Safety through ergodicity An MDP is called ergodic iff for every state there exists a policy that gets the agent to any other state  [20] . In other words, every mistake can be remedied in  such MDP. Moldovan  [20]  then defines δ-safe policies as policies guaranteeing that from any state the agent can get to the starting state with probability at least δ (using a return policy, which is different from the δ-safe one). Stated this way, the safety constraint may seem intractable, or at least impracticalit is even proved, that expressing the set of δ-safe policies is NP-hard  [20] . An approximation of the constraint can be expressed in the terms of two other MDP problems which are easily solved  [20] ; that still leads to δ-safe policies, but the exploration performance may be suboptimal. In our view, safety through ergodicity imposes too much constraints on the problems the agent can learn. It sometimes happens that a robot has to learn some task after which it is not able to return to the initial state (e.g. drive down a hill it cannot go upwards; a human operator then carries the robot back to the starting position). But the inability to "return home" in no means indicates the robot is in an unsafe state.
 Safety through costs Another definition of safety is to define a cost for taking an action/being in a state and minimize the worst-case cost of the generated policies (up to some failure probability). Such approach is presented in  [15] . However, unless a threshold is set, this definition leads only to the safest possible policies, which are not necessarily safe. Expressing the safety using costs is natural for some RL tasks (e.g. when learning the function of a dynamic controller of an engine, the engine's temperature can be treated as a cost). Unfortunately, not all unsafe states can be described using such costs in general. In addition, specifying the right costs may be a difficult task.
 Safety as variance of the expected return An alternative to safety as minimization of a cost (either worst-case or expected) is minimizing both the cost and its variance. This approach is called expected value-variance criterion  [15]  and is used mainly in works prior 2000, e.g.  [7] . A safe policy by this criterion can be viewed as a policy that minimizes the number of critical actions (because fatal transitions are expected to yield much larger costs than safe transitions, increasing the variance significantly). As stated in  [10] , the worst-case approach is too restrictive and cautious. The other expected value-variance criteria suffer from the same disadvantages as safety through costs -mainly from the general difficulty to tune up the costs.
 Safe exploration approaches Finally, when the theoretical concepts have been shown and the various safety definitions have been presented, we can focus on the main part of this overview. Our categorization of safe exploration techniques is based on the work of García  [10] . The basic division is as follows: approaches utilizing the expected return or its variance (Sec. 4.1), labeling-based approaches (Sec. 4.2) and approaches benefiting from prior knowledge (Sec. 4.3).
 Optimal control approaches Techniques in this category utilize variations of the expected value-variance safety criterion. The most basic one is treating the rewards as costs (when a reward is denoted by r t , the corresponding cost is denoted by c t ). Standard RL methods can then be used to solve the safe exploration task, as described e.g. in  [7]  for discounted infinite horizon. The RL objective function J = E ∞ t=0 γ t c t ( 5 ) is called the risk-neutral objective. To make this objective risk-sensitive, we specify a risk factor α and rewrite the objective as: [15] J = 1 α log E [exp (αγ t ∞ t=0 c t )] (6) E [ ∞ t=0 γ t c t ] + α 2 V ar [ ∞ t=0 γ t c t ] , which is also called the expected value-variance criterion. This approach is a part of theory using exponential utility functions, which is popular in optimal control  [19] . To complete this section, the worst-case objective function (also called the minimax objective) is defined as J = sup ∞ t=0 γ t c t . (7) As can be seen, the objective functions containing expectations cannot in fact assure that no unsafe state will be encountered. On the other hand, the minimax objective provides absolute certainty of the safety. However, it may happen that some of the unsafe states can only be reached with a negligible probability. In such cases, the α-value criterion defined by  [15]  can be used -it only takes into account rewards that can be reached with probability greater than α. In the work of Mihatsch  [19] , a scheme is presented that allows to "interpolate" between risk-neutral and worst-case behavior by changing a single parameter. Delage's work  [8]  takes into account the uncertainty of parameters of the MDP. It is often the case that the parameters of the MDP are only estimated from a limited number of samples, causing the parameter uncertainty. He then proposes a possibility that the agent may "invest" some cost to lower the uncertainty in the parameters (by receiving some observations from other sources than exploration). A completely new research area then appears -to decide whether it is more valuable to pay the cost for observations, or to perform exploration by itself. An approximation scheme for dealing with transition matrix uncertainty is presented in  [21] . It considers a robust MDP problem and provides a worst-case, but also robust policy (with respect to the transition matrix uncertainty). A theory generalizing these approaches can be found in  [24] . The theory states that the optimal control decision is based on three terms -the deterministic, cautionary and probing terms. The deterministic term assumes the model is perfect and attempts to control for the best performance. Clearly, this may lead to disaster if the model is inaccurate. Adding a cautionary term yields a controller that considers the uncertainty in the model and chooses a control for the best expected performance. Finally, if the system learns while it is operating, there may be some benefit to choosing controls that are suboptimal and/or risky in order to obtain better data for the model and ultimately achieve better long-term performance. The addition of the probing term does this and gives a controller that yields the best long-term performance.  [24]  To conclude this section, we think that these methods are not well suited for safe exploration -the expected value-variance and similar criteria provide no warranties on the actual safety. On the other hand, the worst-case approaches seem to be too strict.
 Labeling-based approaches The approaches utilizing some kind of state/action labeling (refer to Section 3.1 for the various labeling types) usually make use of two basic components -a risk function and a backup policy. The task of the safety function is to estimate the safety of a state or action. In the simplest case, the safety function can just provide the labeling of the given action; or it can return a likelihood that the action is safe; and in the best case, it would answer with a likelihood to be safe plus a variance (certainty) of its answer. The backup policy is a policy that is able to lead the agent out of the critical states back to the safe area. It is not obvious how to get such a policy, but the authors show some ways how to get one. In the work of Hans  [14] , the most granular labeling is used, where fatal transitions are said to be the transitions with reward less than a given threshold. The safety function is learned during the exploration by collecting the so-called min-reward samples -this is the minimum reward ever obtained for executing a particular action in a particular state. The backup policy is then told to either exist naturally (e.g. a known safe, but suboptimal controller), or it can also be learned. To learn the backup policy, an RL task with altered Bellman equations is used: Q * min (s, a) = max s min R(s, a, s ), max a Q * min (s , a ) . A policy derived from the computed Q * min function is then taken as the backup policy (as it maximizes the minimum reward obtained, and the fatal transitions are defined by low reward). He defines a policy to be safe, if it executes only safe actions in safe states and produces non-fatal transitions in critical states. To learn such safe policy, he then suggests a level-based exploration scheme (although he gives no proofs why it should be better than any other exploration scheme). This scheme is based on the idea that it is better to be always near the known safe space when exploring. All unknown actions from one "level" are explored, and their resulting states are queued to the next "level". For exploration of unknown actions he proposes that the action should be considered critical until proved otherwise, so the exploration scheme uses the backup policy after every unknown action execution. A disadvantage of this approach is that the agent needs some kind of "path planning" to be able to get to the queued states and continue exploration from them. García's PI-SRL algorithm  [10]  is a way to safeguard the classical policy iteration algorithm. Since the labels error /non-error are only for final states, the risk function here is extended by a so called Case-based memory, which is in short a constant-sized memory for storing the historical (s, a, V(s)) samples and is able to find nearest neighbors for a given query (using e.g. the Euclidean distance). In addition to the error and non-error states, he adds the definition of known and unknown states, where known states are those that have a neighbor in the case-based memory closer than a threshold. A safe policy is then said to be a policy that always leads to known non-error final states. To find such policy, the policy iteration is initialized with the safe backup policy and exploration is done via adding a small amount of Gaussian noise to the actions. This approach is suitable for continuous state-and action-spaces. Another approach is presented in the work of Geibel  [12] , where the risk and objective functions are treated separately. So the risk function only classifies the states (again only final states) as either fatal or goal, and the risk of a policy (risk function) is then computed as the expected risk following the policy (where fatal states have risk 1 and goal states have risk 0). The task is then said to be to maximize the objective function (e.g. discounted infinite horizon) w.r.t. the condition that the risk of the considered policies is less than a safety threshold. The optimization itself is done using modified Q-learning, and the optimized objective function is a linear combination of the original objective function and the risk function. By changing the weights in the linear combination the algorithm can be controlled to behave more safely or in a more risk-neutral way. A generalization of Geibel's idea to take the risk and reward functions separately can be found in the work of Kim  [18] . In this work, the constrained RL task is treated as a Constrained MDP and the algorithm CBEETLE for solving the Constrained MDPs is shown. The advantage of this work is that it allows for several independent risk (cost) functions and doesn't need to convert them to the same scale. A similar approach of using constrained MDP to solve the problem can be found in the work of Moldovan  [20] . He does, however, use the ergodicity condition to tell safe and unsafe states apart (that is, safe are only those states from which the agent can get back to the initial state). Moreover, this approach is only shown to work for toy examples like the grid world with only several thousands of discrete states, which may not be sufficient for real robotics tasks. The idea of having several risk functions is further developed by Ertle  [9] . The agent is told to have several behaviors and a separate safety function is learned for each behavior. This approach allows for modularity and sharing of the learned safety functions among different types of agents. More details on this work will be provided in the next section, because it belongs to learning with teachers. An approach slightly different from the previously mentioned in this section is using the methods of reachability analysis to solve safe exploration. Gillula in his work  [13]  defines a set of keep-out states (corresponding to unsafe in our labeling) and then a set called P re(τ ) is defined as a set of all states from which it is possible to get to a keep-out state in less than τ steps. Reachability analysis is used to compute the P re(τ ) set. Safe states are then all states not in P re(τ ) for a desired τ . This approach, however, doesn't utilize reinforcement learning, it computes the optimal policy using standard supervised learning methods with one additional constraint -that the system must use safe actions near the P re(τ ) set. On the other hand, the system is free to use whatever action desired when it is not near P re(τ ). As was presented in this section, the labeling-based approaches provide a number of different ways to reach safety in exploration. They are, however, limited in several ways -some of them make use of the (usually hard-to-obtain) transition matrix, the others may need to visit the unsafe states in order to learn how to avoid them, or need the state-space to be metric.
 Approaches benefiting from prior knowledge The last large group of safe exploration techniques are the ones benefiting from various kinds of prior knowledge (other than the parameters of the MDP). We consider this group the most promising for safe exploration, because "it is impossible to avoid undesirable situations in high-risk environments without a certain amount of prior knowledge about the task"  [10] . The first option how to incorporate prior knowledge into exploration is to initialize the search using the prior knowledge. In fact, several works already mentioned in previous sections use prior knowledge -namely the approaches with a backup policy  (Hans [14] , García  [10] ). Also, García suggests that the initial estimate of the value function can be done by providing prior knowledge, which results in much faster convergence (since the agent does no more have to explore really random actions, the estimate of the value function already "leads it" the right way)  [10] . Another option how to incorporate prior knowledge is by using Learning from Demonstration (LfD) methods. Due to the limited space, we will not give the basics of LfD -a good overview of the state-of-the-art methods is for example in  [2] . For our overview, it is sufficient to state that LfD methods can derive a policy from a set of demonstrations provided by a teacher. What is important, is that the teacher does not necessarily have to have the same geometrical and physical properties as the trainee (although it helps the process if possible). It is therefore possible to use LfD to teach a 5-joint arm to play tennis, while using 3-joint human arm as the source of demonstrations (but the learned policy may be suboptimal; RL should then be used to optimize the policy). In Apprenticeship Learning  [1] , the reward function is learned using LfD. The human pilot flies a helicopter at his best, and both system dynamics and the reward function are learned from the demonstrations. It is however apparent that the performance of the agent is no longer objectively optimal, but that it depends on the abilities of the human pilot. Another way of incorporating prior knowledge into the learning process is to manually select which demonstrations will be provided, as in the work of Ertle  [9] . In the work it is suggested that more teacher demonstrations should come from the areas near the unsafe set, in order to teach the agent precisely where the border between safe and unsafe is located. The last technique described in our overview is interleaving autonomous exploration with teacher demonstrations. As in the previous case, some teacher demonstrations are provided in advance, and then the exploration part starts utilizing the teacher-provided information. After some time, or in states very different from all other known states, the agent requests the teacher to provide more examples  [2, 5] . The idea behind this algorithm is that it is impossible to think out in advance what all demonstrations will the agent need in order to learn the optimal policy. Finishing this section, the algorithms utilizing prior knowledge seem to be the most promising out of all the presented approaches. They provide both a speedup of the learning process (by discarding the low-reward areas) and a reasonable way to specify the safety conditions (via LfD or interleaving).
 Conclusion In our work we have given a short introduction on the basics of Markov Decision Processes as well as the basic Reinforcement Learning methods like Value Iteration, Q-learning and Policy Iteration. In Section 3 we have summarized many recent approaches on how to define safety in the framework of optimal control and reinforcement learning. We have also proposed a novelty definition of safety, which divides the state space to safe, critical and unsafe states. We have shown that all other labeling-based safety definitions are covered by our new definition. In Section 4 many different safe exploration methods are categorized into three basic groups -algorithms from optimal control theory, reinforcement learning algorithms based on state labeling, and algorithms utilizing extra prior knowledge. We have shortly summarized the advantages and disadvantages of the particular approaches. We have also stated that at least for difficult real-world problems, safe exploration without prior knowledge is practically impossible, and prior knowledge almost always helps to achieve faster convergence. Another observation has been that some of the safe exploration algorithms need to visit unsafe states to correctly classify them later, which might discard them from some usage scenarios where the unsafe states are really fatal. It seems to us that the field of safe exploration in reinforcement learning has been very fragmented and lacks an all-embracing theory. However, the question is, if it is even possible to find such theory -the main problem may be the fragmentation and differences of various RL methods themselves. At least, the safe exploration community would benefit from a unification of the terminology (and our proposal of the novelty safety labeling would like to help that). Other ways of possible future research are for example the following. New ways of incorporating prior knowledge into methods not utilizing it yet could bring interesting speed-up of those algorithms. There is also a bottleneck in the estimation of the results of unknown actions -some advanced function approximation methods should be explored (we aim to investigate Gaussian Processes this way). There are not enough experiments from difficult continuous real-world environments, which would show for example how large problems can be solved using safe exploration. The interleaved learning needs some guidelines on how to cluster the queries for the teacher to some larger "packs" and "ask" them together, possibly increasing the fully autonomous operating time. Last, but not least, the possibility to share some learned safety functions among different kinds of robots seems to be an unexplored area with many practical applications (maybe robot-to-robot LfD could be used). Fig. 1 . 1 Fig. 1. NIFTi UGV robotic platform
 (a) A safe state. (b) A critical state -if the robot went still forward, it would fall down and probably break.
 Fig. 2 .Fig. 3 . 23 Fig. 2. An illustration of safe and critical states.
 action a is supercritical in state s if it can lead to a supercritical state, state s is called critical if there is a supercritical or fatal action in that state (and the state itself is not supercritical), action a is critical in state s if it leads to a critical state (and the action itself is neither supercritical nor fatal in s), state s is called safe if it is neither critical nor supercritical, action a is safe in state s if it is neither critical, nor supercritical, nor fatal in state s, and finally a policy is safe if for all critical states it leads to a safe state in a finite number of non-fatal transitions (and if it only executes safe actions in safe states).