Abstract: were supported by a generous gift from The Dame Jillian and Dr. Arthur M. Sackler Foundation for the Arts, Sciences, & Humanities, in memory of Dame Sackler's husband, Arthur M. Sackler. The complete program and video recordings of most presentations are available on the NAS website at http://www.nasonline.org/science-of-deeplearning.y

Andr É Barreto
Shaobo Hou
Diana Borsa
David Silver
Doina Precup

Fast reinforcement learning with generalized policy updates

The combination of reinforcement learning with deep learning is a promising approach to tackle important sequential decisionmaking problems that are currently intractable. One obstacle to overcome is the amount of data needed by learning systems of this type. In this article, we propose to address this issue through a divide-and-conquer approach. We argue that complex decision problems can be naturally decomposed into multiple tasks that unfold in sequence or in parallel. By associating each task with a reward function, this problem decomposition can be seamlessly accommodated within the standard reinforcement-learning formalism. The specific way we do so is through a generalization of two fundamental operations in reinforcement learning: policy improvement and policy evaluation. The generalized version of these operations allow one to leverage the solution of some tasks to speed up the solution of others. If the reward function of a task can be well approximated as a linear combination of the reward functions of tasks previously solved, we can reduce a reinforcement-learning problem to a simpler linear regression. When this is not the case, the agent can still exploit the task solutions by using them to interact with and learn about the environment. Both strategies considerably reduce the amount of data needed to solve a reinforcement-learning problem. artificial intelligence | reinforcement learning | generalized policy improvement | generalized policy evaluation | successor features R einforcement learning (RL) provides a conceptual framework to address a fundamental problem in artificial intelligence: the development of situated agents that learn how to behave while interacting with the environment  (1) . In RL, this problem is formulated as an agent-centric optimization in which the goal is to select actions to get as much reward as possible in the long run. Many problems of interest are covered by such an inclusive definition; not surprisingly, RL has been used to model robots  (2)  and animals  (3) , to simulate real (4) and artificial (5) limbs, to play board  (6)  and card  (7)  games, and to drive simulated bicycles  (8)  and radio-controlled helicopters  (9) , to cite just a few applications. Recently, the combination of RL with deep learning added several impressive achievements to the above list  (10) (11) (12) (13) . It does not seem too far-fetched, nowadays, to expect that these techniques will be part of the solution of important open problems. One obstacle to overcome on the track to make this possibility a reality is the enormous amount of data needed for an RL agent to learn to perform a task. To give an idea, in order to learn how to play one game of Atari, a simple video game console from the 1980s, an agent typically consumes an amount of data corresponding to several weeks of uninterrupted playing; it has been shown that, in some cases, humans are able to reach the same performance level in around 15 min  (14) . One reason for such a discrepancy in the use of data is that, unlike humans, RL agents usually learn to perform a task essentially from scratch. This suggests that the range of problems our agents can tackle can be significantly extended if they are endowed with the appropriate mechanisms to leverage prior knowledge. In this article we describe one such mechanism. The framework we discuss is based on the premise that an RL problem can usually be decomposed into a multitude of "tasks." The intuition is that, in complex problems, such tasks will naturally emerge as recurrent patterns in the agent-environment interaction. Tasks then lead to a useful form of prior knowledge: once the agent has solved a few, it should be able to reuse their solution to solve other tasks faster. Grasping an object, moving in a certain direction, and seeking some sensory stimulus are examples of naturally occurring behavioral building blocks that can be learned and reused. Ideally, information should be exchanged across tasks whenever useful, preferably through a mechanism that integrates seamlessly into the standard RL formalism. The way this is achieved here is to have each task be defined by a reward function. As most animal trainers would probably attest, rewards are a natural mechanism to define tasks because they can induce very distinct behaviors under otherwise identical conditions  (15, 16) . They also provide a unifying language that allows each task to be cast as an RL problem, blurring the (somewhat arbitrary) distinction between the two  (17) . Under this view, a conventional RL problem is conceptually indistinguishable from a task self-imposed by the agent by "imagining" recompenses associated with arbitrary events. The problem then becomes a stream of tasks, unfolding in sequence or in parallel, whose solutions inform each other in some way. What allows the solution of one task to speed up the solution of other tasks is a generalization of two fundamental operations underlying much of RL: policy evaluation and policy improvement. The generalized version of these procedures, jointly referred to as "generalized policy updates," extend their standard counterparts from point-based to set-based operations. This makes it possible to reuse the solution of tasks in two distinct ways. When the reward function of a task can be reasonably approximated as a linear combination of other tasks' reward functions, the RL problem can be reduced to a simpler linear regression that is solvable with only a fraction of the data. When the linearity constraint is not satisfied, the agent can still leverage the solution of tasks-in this case, by using them to interact with and learn about the environment. This can also considerably reduce the amount of data needed to solve the problem. Together, these two strategies give rise to a divide-and-conquer approach to RL that can potentially help scale our agents to problems that are currently intractable.
 RL We consider the RL framework outlined in the Introduction: an agent interacts with an environment by selecting actions to get as much reward as possible in the long run  (1) . This interaction happens at discrete time steps, and, as usual, we assume it can be modeled as a Markov decision process (MDP)  (18) . An MDP is a tuple M ≡ (S, A, p, r , γ) whose components are defined as follows. The sets S and A are the state space and action space, respectively (we will consider that A is finite to simplify the exposition, but most of the ideas extend to infinite action spaces). At every time step t, the agent finds itself in a state s ∈ S and selects an action a ∈ A. The agent then transitions to a next state s , where a new action is selected, and so on. The transitions between states can be stochastic: the dynamics of the MDP, p(•|s, a), give the next-state distribution upon taking action a in state s. In RL, we assume that the agent does not know p, and thus it must learn based on transitions sampled from the environment. A sample transition is a tuple (s, a, r , s ) where r is the reward given by the function r (s, a, s ), also unknown to the agent. As discussed, here we adopt the view that different reward functions give rise to distinct tasks. Given a task r , the agent's goal is to find a policy π : S → A, that is, a mapping from states to actions, that maximizes the value of every state-action pair, defined as Q π r (s, a) ≡ E π ∞ i=0 γ i r (St+i , At+i , St+i+1) | St = s, At = a , [1] where St and At are random variables indicating the state occupied and the action selected by the agent at time step t, E π [•] denotes expectation over the trajectories induced by π, and γ ∈ [0, 1) is the discount factor, which gives less weight to rewards received further into the future. The function Q π r (s, a) is usually referred to as the "action-value function" of policy π on task r ; sometimes, it will be convenient to also talk about the "state-value function" of π, defined as V π r (s) ≡ Q π r (s, π(s)). Given an MDP representing a task r , there exists at least one optimal policy π * r that attains the maximum possible value at all states; the associated optimal value function V * r is shared by all optimal policies  (18) . Solving a task r can thus be seen as the search for an optimal policy π * r or an approximation thereof. Since the number of possible policies grows exponentially with the size of S and A, a direct search in the space of policies is usually infeasible. One way to circumvent this difficulty is to resort to methods based on dynamic programming, which exploit the properties of MDPs to reduce the cost of searching for a policy  (19) . Policy Updates. RL algorithms based on dynamic programming build on two fundamental operations (1). Definition 1. "Policy evaluation" is the computation of Q π r , the value function of policy π on task r . Definition 2. Given a policy π and a task r , "policy improvement" is the definition of a policy π such that Q π r (s, a) ≥ Q π r (s, a) for all (s, a) ∈ S × A. [2] We call one application of policy evaluation followed by one application of policy improvement a "policy update." Given an arbitrary initial policy π, successive policy updates give rise to a sequence of improving policies that will eventually reach an optimal policy π * r  (18) . Even when policy evaluation and policy improvement are not performed exactly, it is possible to derive guarantees on the performance of the resulting policy based on the approximation errors introduced in these steps  (20, 21) . Fig.  1  illustrates the basic intuition behind policy updates. What makes policy evaluation tractable is a recursive relation between state-action values known as the Bellman equation: Q π r (s, a) = E S ∼p(•|s,a) r (s, a, S ) + γQ π r (S , π(S )) . [3] Expression (Exp.) 3 induces a system of linear equations whose solution is Q π r . This immediately suggests ways of performing policy evaluation when the MDP is known  (18) . Importantly, the Bellman equation also facilitates the computation of Q π r without knowledge of the dynamics of the MDP. In this case, one estimates the expectation on the right-hand side of Exp. 3 based on samples from p(•|s, a), leading to the well-known method of temporal differences  (22, 23) . It is also often the case that in problems of interest the state space S is too big to allow for a tabular representation of the value function, and hence Q π r is replaced by an approximation Qπ r . As for policy improvement, it is in fact simple to define a policy π that performs at least as well as, and generally better than, a given policy π. Once the value function of π on task r is known, one can compute an improved policy π as π (s) ∈ arg max a∈A Q π r (s, a). [4] In words, the action selected by policy π on state s is the one that maximizes the action-value function of policy π on that state. The fact that policy π satisfies Definition 2 is one of the fundamental results in dynamic programming and the driving force behind many algorithms used in practice  (18) . The specific way policy updates are carried out gives rise to different dynamic programming algorithms. For example, value iteration and policy iteration can be seen as the extremes of a spectrum of algorithms defined by the extent of the policy evaluation step  (19, 24) . RL algorithms based on dynamic programming can be understood as stochastic approximations of these methods or other instantiations of policy updates  (25) .
 Generalized Policy Updates From the discussion above, one can see that an important branch of the field of RL depends fundamentally on the notions of policy evaluation and policy improvement. We now discuss generalizations of these operations. Definition 3. "Generalized policy evaluation" (GPE) is the computation of the value function of a policy π on a set of tasks R.
 A B Fig.  1 . (A) Sequence of policy updates as a trajectory that alternates between the policy and value spaces and eventually converges to an optimal solution (1). (B) Detailed view of the trajectory across the value space for a state space with two states only. The shadowed rectangles associated with each value function represent the region of the value space containing the value function that will result from one application of policy improvement followed by policy evaluation (cf. Exp. 2). Definition 4. Given a set of policies Π and a task r , "generalized policy improvement" (GPI) is the definition of a policy π such that Q π r (s, a) ≥ sup π∈Π Q π r (s, a) for all (s, a) ∈ S × A. [5] GPE and GPI are strict generalizations of their standard counterparts, which are recovered when R has a single task and Π has a single policy. However, it is when R and Π are not singletons that GPE and GPI reach their full potential. In this case, they become a mechanism to quickly construct a solution for a task, as we now explain. Suppose we are interested in one of the tasks r ∈ R, and we have a set of policies Π available. The origin of these policies is not important: they may have come up as solutions for specific tasks or have been defined in any other arbitrary way. If the policies π ∈ Π are submitted to GPE, we have their value functions on the task r ∈ R. We can then apply GPI over these value functions to obtain a policy π that represents an improvement over all policies in Π. Clearly, this reasoning applies without modification to any task in R. Therefore, by applying GPE and GPI to a set of policies Π and a set of tasks R, one can compute a policy for any task in R that will in general outperform every policy in Π. Fig.  2  shows a graphical depiction of GPE and GPI. Obviously, in order for GPE and GPI to be useful in practice, we must have efficient ways of performing these operations. Consider GPE, for example. If we were to individually evaluate the policies π ∈ Π over the set of tasks r ∈ R, it is unlikely that the scheme above would result in any gains in terms of computation or consumption of data. To see why this is so, suppose again that we are interested in a particular task r . Computing the value functions of policies π ∈ Π on task r would require |Π| policy evaluations with a naive form of GPE (here, | • | denotes the cardinality of a set). Although the resulting GPI policy π would compare favorably to all policies in Π, this guarantee would be vacuous if these policies are not competent at task r . Therefore, a better allocation of resources might be to use the policy evaluations for standard policy updates, which would generate a sequence of |Π| policies with increasing performance on task r (compare Figs.  1 and 2 ). This difficulty in using generalized pol-Fig.  2 . Depiction of generalized policy updates on a state space with two states only. With GPE each policy π ∈ Π is evaluated on all tasks r ∈ R. The state-value function of policy π on task r, V π r , delimits a region in the value space where the next value function resulting from policy improvement will be (cf. Fig.  1 ). The analogous space induced by GPI corresponds to the intersection of the regions associated with the individual value functions (represented as dark gray rectangles in the figure). The smaller the space of value functions associated with GPI, the stronger the guarantees regarding the performance of the resulting policy. icy updates in practice is further aggravated if we do not have a fast way to carry out GPI. Next, we discuss efficient instantiations of GPE and GPI. Fast GPE with Successor Features. Conceptually, we can think of GPE as a function associated with a policy π that takes a task r as input and outputs a value function Q π r  (26) . Hence, a practical way of implementing GPE would be to define a suitable representation for tasks and then learn a mapping from r to value functions Q π r  (27) . This is feasible when such a mapping can be reasonably approximated by the choice of function approximator and enough examples (r , Q π r ) are available to characterize the relation underlying these pairs. Here, we will focus on a form of GPE that is based on a similar premise but leads to a form of generalization over tasks that is correct by definition. Let φ : S × A × S → R d be an arbitrary function whose output we will see as "features." Then, for any w ∈ R d , we have a task defined as rw(s, a, s ) = φ(s, a, s ) w,  [6]  where denotes the transpose of a vector. Let R φ ≡ {rw = φ w | w ∈ R d } be the set of tasks induced by all possible instantiations of w ∈ R d . We now show how to carry out an efficient form of GPE over R φ . Following Barreto et al.  (28) , we define the "successor features" (SFs) of policy π as ψ π (s, a) ≡ E π ∞ i=0 γ i φ(St+i , At+i , St+i+1) | St = s, At = a . The ith component of ψ π (s, a) gives the expected discounted sum of φi when following policy π starting from (s, a). Thus, ψ π can be seen as a d -dimensional value function in which the features φi (s, a, s ) play the role of reward functions (cf. Exp. 1). As a consequence, SFs satisfy a Bellman equation analogous to Exp. 3, which means that they can be computed using standard RL methods like temporal differences  (22) . Given the SFs of a policy π, ψ π , we can quickly evaluate π on task rw ∈ R φ by computing ψ π (s, a) w = E π ∞ i=0 γ i φ(St+i , At+i , St+i+1) w|St = s, At = a = E π ∞ i=0 γ i rw(St+i , At+i , St+i+1) | St = s, At = a = Q π rw (s, a) ≡ Q π w (s, a). [7] That is, the computation of the value function of policy π on task rw is reduced to the inner product ψ π (s, a) w. Since this is true for any task rw, SFs provide a mechanism to implement a very efficient form of GPE over the set R φ (cf. Definition 3). The question then arises as to how inclusive the set R φ is. Since R φ is fully determined by φ, the answer to this question lies in the definition of these features. Mathematically speaking, R φ is the linear space spanned by the d features φi . This view suggests ways of defining φ that result in a R φ containing all possible tasks. A simple example can be given for when both the state space S and the action space A are finite. In this case, we can recover any possible reward function by making d = |S | 2 × |A| and having each φi be an indicator function associated with the occurrence of a specific transition (s, a, s ). This is essentially Dayan's (29) concept of "successor representation" extended from S to S × A × S. The notion of covering the space of tasks can be generalized to continuous S and A if we think of φi as basis functions over the function space S × A × S → R, for we know that, under some technical conditions, one can approximate any function in this space with arbitrary accuracy by having an appropriate basis. Although these limiting cases are reassuring, more generally we want to have a number of features d |S |. This is possible if we consider that we are only interested in a subset of all possible tasks-which should in fact be a reasonable assumption in most realistic scenarios. In this case, the features φ should capture the underlying structure of the set of tasks R in which we are interested. For example, many tasks we care about could be recovered by features φ i representing entities (from concrete objects to abstract concepts) or salient events (such as the interaction between entities). In Fast RL with GPE and GPI, we give concrete examples of what the features φ may look like and also discuss possible ways of computing them directly from data. For now, it is worth mentioning that, even when φ does not span the space R, one can bound the error between the two sides of Exp. 7 based on how well the tasks of interest can be approximated using φ (proposition 1 in ref.  30 ). Fast GPI with Value-Based Action Selection. Thanks to the concept of value function, standard policy improvement can be carried out efficiently through Exp. 4. This raises the question whether the same is possible with GPI. We now answer this question in the affirmative for the case in which GPI is applied over a finite set Π. Following Barreto et al.  (28) , we propose to compute an improved policy π as π (s) ∈ argmax a∈A max π∈Π Q π r (s, a). [8] Barreto et al. have shown that Exp. 8 qualifies as a legitimate form of GPI as per Definition 4 (theorem 1 in ref.  28 ). Note that the action selected by a GPI policy π on a state s may not coincide with any of the actions selected by the policies π ∈ Π on that state. This highlights an important difference between Exp. 8 and methods that define a higher-level policy that alternates between the policies π ∈ Π (32). In SI Appendix, we show that the policy π resulting from Exp. 8 will in general outperform its analog defined over Π. As discussed, GPI reduces to standard policy improvement when Π contains a single policy. this also happens with Exp. 8 when one of the policies in Π strictly outperforms the others on task r . This means that, after one application of GPI, adding the resulting policy π to the set Π will reduce the next application of Exp. 8 to Exp. 4. Therefore, if GPI is used in a sequence of policy updates, it will collapse back to its standard counterpart after the first update. In contrast with policy improvement, which is inherently iterative, GPI is an one-off operation that yields a quick solution for a task when multiple policies have been evaluated on it. The guarantees regarding the performance of the GPI policy π can be extended to the scenario where Exp. 8 is used with approximate value functions. In this case, the lower bound in Exp. 5 decreases in proportion to the maximum approximation error in the value function approximations (theorem 1 in ref.  28) . We refer the reader to SI Appendix for an extended discussion on GPI. The Generalized Policy. Strictly speaking, in order to leverage the instantiations of GPE and GPI discussed in this article, we only need two elements: features φ : S × A × S → R d and a set of policies Π = {πi } n i=1 . Together, these components yield a set of SFs Ψ = {ψ π i } n i=1 , which provide a quick form of GPE over the set of tasks R φ induced by the features φ. Specifically, for any w ∈ R d , we can quickly evaluate policy πi on task rw(s, a, s ) = φ(s, a, s ) w by computing Q π i w (s, a) = ψ π i (s, a) w. Once we have the value functions of policies πi on task rw, {Q π i w } n i=1 , Exp. 8 can be used to obtain a GPI policy π that will in general outperform the policies πi on rw. Observe that the same scheme can be used if we have approximate SFs ψπ i ≈ ψ π i , which will be the case in most realistic scenarios. Since the resulting value functions will also be approximations, Qπ i w ≈ Q π i w , we are back to the approximate version of GPI discussed in Fast GPI with Value-Based Action Selection. Given a set of SFs Ψ, approximate or otherwise, any w ∈ R d gives rise to a policy through GPE (Exp. 7) and GPI (Exp. 8). We can thus think of generalized updates as implementing a policy πΨ(s; w) whose behavior is modulated by w. We will call πΨ a "generalized policy." Fig.  3  provides a schematic view of the computation of πΨ through GPE and GPI. Since each component of w weighs one of the features φi (s, a, s ), changing them can intuitively be seen as setting the agent's current "preferences." For example, the vector w = [0, 1, −2] indicates that the agent is indifferent to feature φ1 and wants to seek feature φ2 while avoiding feature φ3 with twice the impetus. Specific instantiations of πΨ can behave in ways that are very different from its constituent policies π ∈ Π. We can draw a parallel with nature if we think of features as concepts like water or food and note how much the desire for these items can affect an animal's behavior. Analogies aside, this sort Fig.  3 . representation of how the generalized policy πΨ is implemented through GPE and GPI. Given a state s ∈ S, the first step is to compute, for each a ∈ A, the values of the n SFs: {ψ π i (s, a)} n i=1 . Note that ψ π i can be arbitrary nonlinear functions of their inputs, possibly represented by complex approximators like deep neural networks  (30, 31) . The SFs can be laid out as an n × |A| × d tensor in which the entry (i, a, j) is ψ π i j (s, a). Given this layout, GPE reduces to the multiplication of the SF tensor with a vector w ∈ R d , the result of which is an n × |A| matrix whose (i, a) entry is Q π i (s, a). GPI then consists in finding the maximum element of this matrix and returning the associated action a. This whole process can be seen as the implementation of a policy πΨ(s; w) whose behavior is parameterized by a vector of preferences w. of arithmetic over features provides a rich interface for the agent to interact with the environment at a higher level of abstraction in which decisions correspond to preferences encoded as a vector w. Next, we discuss how this can be leveraged to speed up the solution of an RL task.
 Fast RL with GPE and GPI We now describe how to build and use the adaptable policy πΨ implemented by GPE and GPI. To make the discussion more concrete, we consider a simple RL environment depicted in Fig.  4 . The environment consists of a 10 × 10 grid with four actions available: A = {up, down, left, right}. The agent occupies one of the grid cells, and there are also 10 objects spread across the grid. Each object belongs to one of two types. At each time step t, the agent receives an image showing its position and the position and type of each object. Based on this information, the agent selects an action a ∈ A, which moves it one cell along the desired direction. The agent can pick up an object by moving to the cell occupied by it; in this case, it gets a reward defined by the type of the object. A new object then pops up in the grid, with both its type and location sampled uniformly at random (more details are in SI Appendix). This simple environment can be seen as a prototypical member of the class of problems in which GPE and GPI could be useful. This becomes clear if we think of objects as instances of (potentially abstract) concepts, here symbolized by their types, and note that the navigation dynamics are a proxy for any sort of dynamics that mediate the interaction of the agent with the world. In addition, despite its small size, the number of possible configurations of the grid is actually quite large, of the order of 10 15 . This precludes an exact representation of value functions and illustrates the need for approximations that inevitably arises in many realistic scenarios. By changing the rewards associated with each object type, one can create different tasks. We will consider that the agent wants to build a set of SFs Ψ that give rise to a generalized policy πΨ(s; w) that can adapt to different tasks through the vector of preferences w. This can be either because the agent does not know in advance the task it will face or because it will face more than one task. Defining a Basis for Behavior. In order to build the SFs Ψ, the agent must define two things: features φ and a set of policies Π. Since φ should be associated with rewarding events, we define each feature φi as an indicator function signaling whether an object of type i has been picked up by the agent (i.e., φ ∈ R 2 ). To be precise, we have that φi (s, a, s ) = 1 if the transition from state s to state s is associated with the agent picking up an object of type i, and φi (s, a, s ) = 0 otherwise. These features induce a set R φ where task rw ∈ R φ is characterized by how desirable or undesirable each type of object is. Now that we have defined φ, we turn to the question of how to determine an appropriate set of policies Π. We will restrict the policies in Π to be solutions to tasks rw ∈ R φ . We start with what is perhaps the simplest choice in this case: a set Π12 = {π1, π2} whose two elements are solutions to the tasks w1 = [1, 0] and w2 = [0, 1] (henceforth, we will drop the transpose superscript to avoid clutter). Note that the goal in tasks w1 and w2 is to pick up objects of one type while ignoring objects of the other type. We are now ready to compute the SFs Ψ induced by our choices of φ and Π. In our experiments, we used an algorithm analogous to Q-learning to compute approximate SFs ψπ 1 and ψπ 2 (pseudocode in SI Appendix). We represented the SFs using multilayer perceptrons with two hidden layers  (33) . The set of SFs Ψ yields a generalized policy π Ψ(s ; w) parameterized by w. We now evaluate π Ψ on the task whose goal is to pick up objects of the first type while avoiding objects of the second type. Using φ defined above, this task can be represented as rw 3 (s, a, s ) = φ(s, a, s ) w3, with w3 = [1, −1]. We thus evaluate the generalized policy instantiated as π Ψ(s ; w3). Results are shown in Fig.  5A . As a reference, we also show the learning curve of Q-learning (23) using the same architecture to directly approximate Q π w 3 . GPE and GPI allow one to compute an instantaneous solution for a new task, without any learning on the task itself, that is competitive with the policies found by Q-learning when using around 6 × 10 4 sample transitions. The performance of the policy π Ψ synthesized by GPE and GPI corresponds to more than 70% of the performance eventually achieved by Q-learning after processing 10 6 transitions. This is quite an impressive result when we note that π Ψ managed to avoid objects of the second type even though its constituents policies π1 and π2 were never trained to actively avoid objects. We used a total of 10 6 sample transitions to learn both SFs ψπ 1 and ψπ 2 , which is the same amount of data used by Qlearning to achieve its final performance. The advantage of doing the former is that, once we have the SFs, we can use GPE and GPI to instantaneously compute a solution for any task in R φ . However, how well do GPE and GPI actually perform on R φ ? To answer this question, we ran a second round of experiments to assess the generalization of π Ψ over the entire set R φ . Since this evaluation clearly depends on the set of policies used, we consider two other sets in addition to Π12 = {π1, π2}. The new sets are Π34 = {π3, π4} and Π5 = {π5}, where the policies πi are solutions to the tasks w3 = [1, −1], w4 = [−1, 1], and w5 =  [1, 1] . We repeated the previous experiment with each policy set and evaluated the resulting policies π Ψ over 19 tasks w evenly spread over the nonnegative quadrants of the unit circle (tasks in the negative quadrant are uninteresting because all of the agent must do is to avoid hitting objects). Results are shown in Fig.  6A . As expected, the generalization ability of GPE and GPI depends on the set of policies used. Perhaps more surprising is how well the generalized policy π Ψ induced by some of these sets perform across the entire space of tasks R φ , sometimes matching the best performance of Q-learning when solving each task individually. These experiments show that a proper choice of base policies Π can lead to good generalization over the entire set of tasks R φ . In general, though, it is unclear how to define an appropriate Π. Fortunately, we can refer to our theoretical understanding of GPE and GPI to have some guidance. First, we know from the discussion above that the larger the number of policies in Π the stronger the guarantees regarding the performance of the resulting policy π Ψ (Exp. 5). In addition to that, Barreto et al. have shown that it is possible to guarantee a minimum performance level for π Ψ on task w based on mini w − wi , where • is some norm and wi are the tasks associated with the policies πi ∈ Π used for GPE and GPI (theorem 2 in ref.  28 ). Together, these two insights suggest that, as we increase the size of Π, the performance of the resulting policy π Ψ should improve across R φ , especially on tasks that are close to the tasks wi . To test this hypothesis empirically, we repeated the previous experiment, but now, instead of comparing disjoint policy sets, we compared a sequence of sets formed by adding one by one the policies π2, π5, π1, and π3, in this order, to the initial set {π4}. The results, in Fig.  6B , confirm the trend implied by the theory. Task Inference. So far, we have assumed that the agent knows the vector w that describes the task of interest. Although this can be the case in some scenarios, ideally we would be able to apply GPE and GPI even when w is not provided. In this section and in Preferences as Actions, we describe two possible ways for the agent to learn w. Given a task r , we are looking for a w ∈ R d that leads to good performance of the generalized policy πΨ(s; w). We could in principle approach this problem as an optimization over w ∈ R d whose objective is to maximize the value of πΨ(s; w) across (a subset of) the state space. It turns out that we can exploit the structure underlying SFs to efficiently determine w without ever looking at the value of πΨ. Suppose we have a set of m sample transitions from a given task, {(si , ai , r i , s i )} m i=1 . Then, based on Exp. 6, we can infer w by solving the following minimization: min w m i=1 |φ(si , ai , s i ) w − r i | p , [9] where p ≥ 1 (one may also want to consider the inclusion of a regularization term, see ref.  33) . Observe that, once we have a solution w for the problem above, we can plug it in Exp. 7 and use GPE and GPI as we did before-that is, we have just turned an RL task into an easier linear regression problem. To illustrate the potential of this approach, we revisited the task w3 = [1, −1] tackled above, but now, instead of assuming we knew w3, we solved the problem in Exp. 9 using p = 2. We collected sample transitions using a policy π that picks actions
 A B Fig.  6 . Results on the space of tasks R φ induced by a two-dimensional φ. The sets of policies Π used in A are disjoint; in B these sets overlap. The evaluation of an algorithm on a task is represented as a vector whose direction indicates the task w and whose magnitude gives the average sum of rewards over 10 runs with 250 trials each. Q-learning learned each task individually, from scratch; the dotted curves correspond to its performance after having processed 5 × 10 4 , 1 × 10 5 , 2 × 10 5 , 5 × 10 5 , and 1 × 10 6 sample transitions. Our method only learned the policies in the sets Π and then generalized across all others tasks through GPE and GPI. The SFs used for GPE and GPI consumed 5 × 10 5 sample transitions to be trained each. uniformly at random and used the gradient descent method to refine an approximation w ≈ w online (SI Appendix). Results are shown in Fig.  5  A and B. The performance of the generalized policy π Ψ(s ; w3) using the correct w3 was recovered after around 800 sample transitions only. To put this number into perspective, recall that Q-learning needs around 60,000 transitions, or 75 times as much data, to achieve the same performance level. The number of transitions needed to learn w can be reduced further if we use the closed-form solution for the least-squares problem. The problem described in Exp. 9 can be solved even if the rewards are the only information available about a given task. Other interesting possibilities arise when the observations provided by the environment can be used to infer w, such as, for example, visual or auditory cues indicating the current task. In this case, it might be possible for the agent to determine w even before having observed a single nonzero reward  (30) . So far, we have used handcrafted features φ. It turns out that Exp. 9 can be generalized to also allow φ to be inferred from data. Given sample transitions from k tasks, {(sij , aij , r ij , s ij )} with p ≥ 1. Note that the features φ can be arbitrary nonlinear functions of their inputs. As discussed by Barreto et al.  (30) , if we make c = k we can solve a simplified version of the problem above in which wj do not show up. More generally, the problem in Exp. 10 can be solved as a multitask regression  (34) . In Fig.  5B , we show the performance of π Ψ(s ; w) when using learned features φ. These results were obtained as follows. First, we used the random policy π to collect data from tasks w1, w2, w4, and w5 and applied the stochastic gradient descent algorithm to solve Exp. 10 with p = 2. We then discarded the resulting approximations wi appearing in Exp. 10 and solved Exp. 9 exactly as before but now using φ instead of φ. As with SFs, in order to represent the features, we used multilayer perceptrons with two hidden layers (SI Appendix). The results in Fig.  5B  also show the effect of varying the dimension of the learned features on the performance of πΨ(s; w) (other illustrations are in refs.  28 and 30) . Preferences as Actions. In this section, we consider the scenario where, given features φ and a set of policies Π, there is no vector w that leads to acceptable performance of πΨ(s; w). To illustrate this situation, we use our environment with a slight twist: now, on picking up an object of a certain type, the agent gets a positive reward only if the number of objects of this type is greater or equal to the number of objects of the other type. Otherwise, the agent gets a negative reward. One can represent this problem using the previously defined features φ by switching between two instantiations of w:  [1, −1]  when the first type of object is more abundant and [−1, 1] otherwise. This means that the appropriate value for w is now state-dependent. To handle this situation, we introduce a function mapping states to preferences, ω : S → W, with W ⊆ R d , and redefine the generalized policy as πΨ(s; ω(s)). The RL problem then becomes to find a ω that correctly modulates πΨ  (32, 35) . We can look at the computation of πΨ(s; ω(s)) as a twostage process: first, the agent computes a vector of preferences w = ω(s); then, using GPE and GPI, it determines the action to be executed in state s as a = πΨ(s; w). Under this view, ω(s) becomes an indirect way of selecting an action in s, which allows us to think of it as a policy defined in the space W. We have thus replaced the original problem of finding a policy π : S → A with the problem of finding a policy ω : S → W. To illustrate the benefits of applying this transformation to the problem, we will use the task described above in which the agent must pick up the type of object that is more abundant. We will tackle this problem by reusing the SFs Ψ induced by the two policies in Π12. Given Ψ, the problem comes down to finding a mapping ω : S → W that leads to good performance of π Ψ(s ; ω(s)). We define W = {−1, 0, 1} 2 and use Q-learning to learn ω. We compare the performance of π Ψ(s ; ω(s)) with two baselines: GPE and GPI using a fixed w, π Ψ(s ; w), and the standard version of Q-learning that learns a policy π(s) over primitive actions. The results, shown in Fig.  7 , illustrate how π Ψ(s ; ω(s)) can significantly outperform the baselines. It is easy to understand why this is the case when we compare π Ψ(s ; ω(s)) with π Ψ(s ; w), for the experiment was designed to render the latter unsuitable. However, why is it that replacing the fourdimensional space A with the nine-dimensional space W leads to better performance? The reason is that ω(s) can be easier to than π(s). As discussed, the vectors w represent the agent's current preferences for types of objects. Since, in general, such preferences should only change when an object is picked up, after selecting a specific w, the agent can hold on to it until it bumps into an object. That is, it is possible to define a good policy ω that maps multiple consecutive states s to the same w. In contrast, it will normally not be possible to execute a single primitive action a ∈ A between the collection of two consecutive objects. This means that the same policy can have a simpler representation as a mapping S → W than as a mapping S → A, making ω easier to learn with certain types of function approximators like neural networks  (33) . Note that other choices of φ may induce other forms of regularities in ω-smoothness being an obvious example. Interestingly, the fact that the agent may be able to adhere to the same w for many time steps allows for a temporally extended policy πΨ(s; ω(s)) that is only permitted to change preferences at certain points in time, reducing even further the amount of data needed to learn the task  (31) . As a final remark, we note that the ability to chain a sequence of preference vectors w changes the nature of the problem of computing an appropriate set of features φ. When the agent has to adhere to a single w, the features φi must span the space of rewards defining the tasks of interest. When the agent is allowed Fig.  7 . Average sum of rewards on the task whose goal is to pick up objects of the type that is more abundant. GPE and GPI used Π 12 as the base policies and the corresponding SFs consumed 5 × 10 5 transitions to be trained each. The results reflect the best performance of each algorithm over multiple parameter configurations (SI Appendix). Shadowed regions are one standard error over 100 runs. to learn a policy ω over preferences w, this linearity assumption is no longer necessary. Similarly, the use of a fixed w requires that the set of available SFs leads to a competent policy for all tasks w we may care about. Here, this requirement is alleviated, since, by changing w, the agent may be able to compensate for eventual imperfections on the induced GPI policies  (31) . Lifelong Learning. We have looked at the different subproblems involved in the use of GPE and GPI. For didactic purposes, we have discussed these problems in isolation, but it is instructive to consider how they can jointly emerge in a real application. One scenario where the problems above would naturally arise is "lifelong learning," in which the interaction of the agent with the environment unfolds over a long period  (36) . Since we expect some patterns to repeat during such an interaction, it is natural to break it in tasks that share some common structure. In terms of the concepts discussed in this article, this common structure would be φ, which can be handcrafted based on prior knowledge about the problem or learned through Exp. 10 at the beginning of the agent's lifetime. Based on the features φ, the agent can compute and store the SFs ψπ associated with the solutions π of the tasks encountered along the way. When facing a new task r , the agent can use the vector of preferences w to quickly adapt the generalized policy π Ψ. We have discussed two ways to do so. In the first one, the agent computes an approximation w through Exp. 9, which immediately yields a solution π Ψ(s ; w). The second possibility is to learn a mapping ω(s) : S → W and use it to have a per-state modulation of the generalized policy, π Ψ(s ; ω(s)). Regardless of how the agent computes w, it can then choose to hold on to π Ψ or use it as a initial solution for the task to be refined by a standard RL algorithm. Either way, the agent can also compute the SFs of the resulting policy, which can in turn be added to the library of SFs Ψ, improving even further its ability to tackle future tasks  (28, 30) .
 Conclusion We have generalized two fundamental operations in RL, policy improvement and policy evaluation, from single to multiple operands (tasks and policies, respectively). The resulting operations, GPE and GPI, can be used to speed up the solution of an RL problem. We showed possible ways to efficiently implement GPE and GPI and discussed how their combination leads to a generalized policy πΨ(s; w) whose behavior is modulated by a vector of preferences w. Two ways of learning w were considered. In the simplest case, w is the solution of a linear regression problem. This reduces an RL task to a much simpler problem that can be solved using only a fraction of the data. This strategy depends on two conditions: 1) the reward function of the tasks of interest are approximately linear in the features φ used for GPE and 2) the set of policies Π used for GPI lead to good performance of πΨ on the tasks of interest. When these assumptions do not hold, one can still resort to GPE and GPI by looking at the preferences w as actions. This strategy can also improve sample efficiency if the mapping from states to preferences is simpler to learn than the corresponding policy. Many extensions of the basic framework presented in this article are possible. As mentioned, Barreto et al.  (31)  have proposed a way of implementing a generalized policy πΨ that explicitly leverages temporal abstraction by treating preferences w as abstract actions that persist for multiple time steps. Borsa et al.  (37)  introduced a generalized form of SFs that has a representation z ∈ Z of a policy πz as one of its inputs: ψ(s, a, z) = ψ πz (s, a). In principle, this allows one to apply GPI over arbitrary, potentially infinite, sets Π (for example, by replacing the maximization in Exp. 8 with sup z∈Z ψ(s, a, z) w).  Hunt et al. (38)  extended SFs to entropy-regularized RL, and, in doing so, they proposed solutions for many challenges that come up when applying GPE and GPI in continuous action spaces. More recently, Hansen et al.  (39)  proposed an approach that allows the features φ to be learned from data in the absence of a reward signal. Other extensions are possible, including different instantiations of GPE and GPI. All of these extensions expand the framework built upon GPE and GPI. Together, they provide a conceptual toolbox that allows one to decompose an RL problem into tasks whose solutions inform each other. This results in a divide-and-conquer approach to RL, which, combined with deep learning, has the potential to scale up our agents to problems currently out of reach. Data Availability. The source code used to generate all of the data associated with this article has been deposited in GitHub (https://github.com/deepmind/deepmind-research/tree/ master/option keyboard/gpe gpi experiments). Fig. 4 . 4 Fig. 4. Depiction of the environment used in the experiments. The shape of the objects (square or triangle) represents their type; the agent is depicted as a circle. We also show the first 10 steps taken by 3 policies, π 1 , π 2 , and π 3 , that would perform optimally on tasks w 1 = [1, 0], w 2 = [0, 1], and w 3 = [1, −1] for any discount factor γ ≥ 0.5.
 Fig. 5 . 5 Fig. 5. Average sum of rewards on task w 3 = [1, −1]. GPE and GPI used Π 12 = {π 1 , π 2 } as the base policies and the corresponding SFs consumed 5 × 10 5 sample transitions to be trained each. B is a zoomed-in version of A showing the early performance of GPE and GPI under different setups. The results reflect the best performance of each algorithm over multiple parameter configurations (SI Appendix). Shadowed regions are one standard error over 100 runs.
 j = 1, 2, . . . , k , we can formulate the problem as the search for a function φ :S × A × S → R c and k vectors wi ∈ R c satisfying min sij , aij , s ij ) wj − r ij | p , [10]
			 | www.pnas.org/cgi/doi/10.1073/pnas.<PHONE> Barreto et al. Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
			 Barreto et al. PNAS | December 1, 2020 | vol. 117 | no. 48 | 30081 Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
			 | www.pnas.org/cgi/doi/10.1073/pnas.<PHONE> Barreto et al. Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
			 PNAS | December 1, 2020 | vol. 117 | no. 48 | 30083 Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
			 | www.pnas.org/cgi/doi/10.1073/pnas.<PHONE> Barreto et al. Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
			 PNAS | December 1, 2020 | vol. 117 | no. 48 | 30085 Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
			 | www.pnas.org/cgi/doi/10.1073/pnas.<PHONE> Barreto et al. Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
			 PNAS | December 1, 2020 | vol. 117 | no. 48 | 30087 Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.


Abstract: Some risks have extremely high stakes. For example, a worldwide pandemic or asteroid impact could potentially kill more than a billion people. Comfortingly, scientific calcultions often put very low probabilities on the occurrence of such catastrophes. In this paper, we argue that there are important new methodological problems which arise when assessing global catastrophic risks and we focus on a problem regarding probability estimation. When an expert provides a calculation of the probability of an outcome, they are really providing the probability of the outcome occurring, given that their argument is watertight. However, their argument may fail for a number of reasons, such as a flaw in the underlying theory, a flaw in the modelling of the problem or a mistake in the calculations. If the probability estimate given by an argument is dwarfed by the chance that the argument itself is flawed, then the estimate is suspect. We develop this idea formally, explaining how it differs from the related distinction between model and parameter uncertainty. Using the risk estimates from the Large Hadron Collider as a test case, we show how serious the problem can be when it comes to catastrophic risks and how best to address it.

Toby Ord
Rafaela Hillerbrand
Anders Sandberg
Francis Rjrr_A_412799 Sgm

Probing the improbable: methodological challenges for risks with low probabilities and high stakes

Introduction Large asteroid impacts are highly unlikely events. 1 Nonetheless, governments spend large sums on assessing the associated risks. It is the high stakes that make these otherwise rare events worth examining. Assessing a risk involves consideration of both the stakes involved and the likelihood of the hazard occurring. If a risk threatens the lives of a great many people, it is not only rational but morally imperative to examine the risk in some detail and to see what we can do to reduce it. This paper focuses on low-probability high-stakes risks. In Section 2, we show that the probability estimates in scientific analysis cannot be equated with the likelihood of these events occurring. Instead of the probability of the event occurring, scientific analysis gives the event's probability conditioned on the given argument being sound. Though this is the case in all probability estimates, we show how it becomes crucial when the estimated probabilities are smaller than a certain threshold. To proceed, we need to know something about the reliability of the argument. To do so, risk analysis commonly falls back on the distinction between model and parameter uncertainty. We argue that this dichotomy is not well suited for incorporating information about the reliability of the theories involved in the risk assessment. Furthermore, the distinction does not account for mistakes made unknowingly. In Section 3, we therefore propose a three-fold distinction between an argument's theory, its model and its calculations. While explaining this distinction in more detail, we illustrate it with historic examples of errors in each of the three areas. We indicate how specific risk assessment can make use of the proposed theory-model-calculation distinction in order to evaluate the reliability of the given argument and thus improve the reliability of their probability estimate for rare events. Recently, concerns have been raised that high-energy experiments in particle physics, such as the Relativistic Heavy Ion Collider (RHIC) at Brookhaven National Laboratory or the Large Hadron Collider (LHC) at CERN, Geneva, may threaten humanity. If these fears are justified, these experiments pose a risk to humanity that can be avoided by simply not turning on the experiment. In Section 4, we use the methods of this paper to address the current debate on the safety of experiments within particle physics. We evaluate current reports in the light of our findings and give suggestions for future research. The final section brings the debate back to the general issue of assessing low-probability risk. We stress that the findings in this paper are not to be interpreted as an argument for anti-intellectualism, but rather as arguments for making the noisy and fallible nature of scientific and technical research subject to intellectual reasoning, especially in situations where the probabilities are very low and the stakes are very high.
 Probability estimates Suppose you read a report which examines a potentially catastrophic risk and concludes that the probability of catastrophe is one in a billion. What probability should you assign to the catastrophe occurring? We argue that direct use of the report's estimate of one in a billion is naive. This is because the report's authors are not infallible and their argument might have a hidden flaw. What the report has told us is not the probability of the catastrophe occurring, but the probability of the catastrophe occurring, given that the included argument is sound. Even if the argument looks watertight, the chance that it contains a critical flaw may well be much larger than one in a billion. After all, in a sample of a billion apparently watertight arguments you are likely to see many that have hidden flaws. Our best estimate of the probability of catastrophe may thus end up noticeably higher than the report's estimate.  2  Let us use the following notation: X, the catastrophe occurs; A, the argument is sound; P(X), the probability of X and P(X|A), the probability of X given A. While we are actually interested in P(X), the report provides us only with an estimate of P(X|A), since it cannot fully take into account the possibility that it is in error.  3, 4  From the axioms of probability theory, we know that P(X) is related to P(X|A) by the following formula: P X P X A P A P X A P A ( ) ( | ) ( ) ( | ) ( ). ( ) = + ¬ ¬ 1 To use this formula to derive P(X), we would require estimates for the probability that the argument is sound, P(A), and the probability of the catastrophe occurring, given that the argument is unsound, P(X|¬A). We are highly unlikely to be able to acquire accurate values for these probabilities in practice, but we shall see that even crude estimates are enough to change the way we look at certain risk calculations. A special case, which occurs quite frequently, is for reports to claim that X is completely impossible. However, this just tells us that X is impossible, given that all our current beliefs are correct, that is P(X|A) = 0. By Equation (1) we can see that this is entirely consistent with P(X) > 0, as the argument may be flawed. Figure  1  is a simple graphical representation of our main point. The square on the left represents the space of probabilities as described in the scientific report, where the black area represents the catastrophe occurring and the white area represents not occurring. The normalized vertical axis denotes the probabilities for the event occurring and not occurring. This representation ignores the possibility of the argument being unsound. To accommodate this possibility, we can revise it in the form of the square on the right. The black and white areas have shrunk in proportion to the probability that the argument is sound and a new grey area represents the possibility that the argument is unsound. Now, the horizontal axis is also normalized and represents the probability that the argument is sound. Figure  1 . The left panel depicts a report's view on the probability of an event occurring. The black area represents the chance of the event occurring, the white area represents it not occurring. The right-hand panel is the more comprehensive picture, taking into account the possibility that the argument is flawed and that we thus face a grey area containing an unknown amount of risk. To continue our example, let us suppose that the argument made in the report looks very solid, and that our best estimate of the probability that it is flawed is one in a thousand, (P(¬A) = 10 −3 ). The other unknown term in Equation (1), P(X|¬A), is generally even more difficult to evaluate, but for the purposes of the current example, let us suppose that we think it highly unlikely that the event will occur even if the argument is not sound and treat this probability as one in a thousand as well. Equation (1) tells us that the probability of catastrophe would then be just over one in a million -an estimate which is a thousand times higher than that in the report itself. This reflects the fact that if the catastrophe were to actually occur, it is much more likely that this was because there was a flaw in the report's argument than that a one in a billion event took place. Flawed arguments are not rare. One way to estimate the frequency of major flaws in academic papers is to look at the proportions which are formally retracted after publication. While some retractions are due to misconduct, most are due to unintentional errors.  5  Using the MEDLINE database,  7 Cokol et al. (2007)  found a raw Figure  1 . The left panel depicts a report's view on the probability of an event occurring. The black area represents the chance of the event occurring, the white area represents it not occurring. The right-hand panel is the more comprehensive picture, taking into account the possibility that the argument is flawed and that we thus face a grey area containing an unknown amount of risk. retraction rate of 6.3 × 10 −5 , but used a statistical model to estimate that the retraction rate would actually be between 0.001 and 0.01 if all journals received the same level of scrutiny as those in the top tier. This would suggest that P(¬A) > 0.001 making our earlier estimate rather optimistic. We must also remember that an argument can easily be flawed without warranting retraction. Retraction is only called for when the underlying flaws are not trivial and are immediately noticeable by the academic community. The retraction rate for a field would thus provide a lower bound for the rate of serious flaws. Of course, we must also keep in mind the possibility that different branches of science may have different retraction rates and different error rates: the hard sciences may be less prone to error than the more applied sciences. Finally, we can have more confidence in an article, the longer it has been open to public scrutiny without a flaw being detected. It is important to note the particular connection between the present analysis and high-stakes low-probability risks. While our analysis could be applied to any risk, it is much more useful for those in this category. For it is only when P(X|A) is very low that the grey area has a relatively large role to play. If P(X|A) is moderately high, then the small contribution of the error term is of little significance in the overall probability estimate, perhaps making the difference between 10 and 10.001% rather than the difference between 0.001 and 0.002%. The stakes must also be very high to warrant this additional analysis of the risk, for the adjustment to the estimated probability will typically be very small in absolute terms. While an additional one in a million chance of a billion deaths certainly warrants further consideration, an additional one in a million chance of a house fire may not. One might object to our approach on the grounds that we have shown only that the uncertainty is greater than previously acknowledged, but not that the probability of the event is greater than estimated: the additional uncertainty could just as well decrease the probability of the event occurring. When applying our approach to arbitrary examples, this objection would succeed; however in this paper, we are specifically looking at cases where there is an extremely low value of P(X|A), so practically any value of P(X|¬A) will be higher and thus drive the combined probability estimate upwards. The situation is symmetric with regard to extremely high estimates of P(X|A), where increased uncertainty about the argument will reduce the probability estimate, the symmetry is broken only by our focus on arguments which claim that an event is very unlikely. Another possible objection is that since there is always a non-zero probability of the argument being flawed, the situation is hopeless: any new argument will be unable to remove the grey area completely. It is true that the grey area can never be completely removed; however, if a new argument (A 2 ) is independent of the previous argument (A 1 ), then the grey area will shrink, for P(¬A 1 , ¬A 2 ) < P(¬A 1 ). This can allow for significant progress. A small remaining grey area can be acceptable if P(X|¬A) P(¬A) is estimated to be sufficiently small in comparison to the stakes.
 Theories, models and calculations The most common way to assess the reliability of an argument is to distinguish between model and parameter uncertainty and assign reliabilities to these choices. While this distinction has certainly been of use in many practical cases, it is unnecessarily crude for the present purpose, failing to account for potential errors in the paper's calculations or a failure of the background theory. In order to account for all possible mistakes in the argument, we look separately at its theory, its model and its calculations. The calculations evaluate a concrete model representing the processes under consideration, for example the formation of black holes in a particle collision, the response of certain climate parameters (such as mean temperature or precipitation rate) to changes in greenhouse gas concentrations or the response of economies to changes in the oil price. These models are mostly derived from more general theories. In what follows, we do not restrict the term 'theory' to well-established and mathematically elaborate theories like electrodynamics, quantum chromodynamics or relativity theory. Rather, theories are understood to include theoretical background knowledge, such as specific research paradigms or the generally accepted research practice within a field (see Figure  2 ). An example is the efficient market hypothesis which underlies many models within economics, such as the Black-Scholes model. We consider adequate models or theories rather than correct ones. For example, we wish to allow that Newtonian mechanics is an adequate theory in many situations, while recognizing that in some cases it is clearly inadequate (such as for calculating the electron orbitals). We thus call a representation of some system adequate if it is able to predict the relevant system features at the required precision. For example, if climate modellers wish to determine the implications our greenhouse gas emissions will have on the well-being of future generations, their model/theory will not be adequate unless it tells them the changes in the local temperature and precipitation. In contrast, a model might only need to tell them changes in global temperature and precipitation to be adequate for answering less sensitive questions. On a theoretical level, much more could be said about this distinction between adequacy and correctness, but for the purposes of evaluating the reliability of risk assessment, the explanation above should suffice. With the following notation: T, the involved theories are adequate; M, the derived model is adequate; and C, the calculations are correct, we can represent P(A) as P(T, M, C). We can then benefit from the theory-model-calculation distinction by expressing this as the product of three separate probabilities, each of which should be easier to estimate than P(A). Firstly, from the laws of conditional probability, it follows that P(T, M, C) = P(T) P(M|T) P(C|M, T). This can be simplified as we can assume C to be independent of M and T, since the correctness of a calculation is independent of whether the theoretical and model assumptions underpinning it were adequate. Given this independence P(C|M, T) = P(C), the above equation can be simplified to: Substituting this back into Equation (1), we obtain a more tractable formula for P(X). P A P T P M T P C ( ) ( ) ( | ) ( ). ( ) = 2 We have already made a rough attempt at estimating P(A) from the paper retraction rates. Estimating P(T), P(M|T) and P(C) is more accurate and somewhat easier, though still of significant difficulty. While estimating the various terms in Equation (  2 ) must ultimately be done on a case-by-case basis, the following elucidation of what we mean by theory, model and calculation will shed some light on how to pursue such an analysis. By incorporating our threefold distinction, it is straightforward to apply findings on the reliability of theories from philosophy of science -based, for example, on probabilistic verification methods (e.g.  Reichenbach 1938)  or falsifications as in  Hempel (1950)  or  Popper (1959) . Often, however, the best we can do is to put some bounds upon them based on the historical record. We thus review typical sources of error in the following three areas.
 Calculation: analytic and numeric Estimating the correctness of the calculation independently from the adequacy of the model and the theory seems important whenever the mathematics involved is non-trivial. Most cases where we are able to provide more than purely heuristic and hand-waving risk assessments are of this sort. Consider climate models evaluating runaway climate change and risk estimates for the LHC or for asteroid impacts. When calculations accumulate, even trivial mathematical procedures become error-prone. A particular difficulty arises due to the division of labour in the sciences: commonly in modern scientific practice, various steps in a calculation are done by different individuals who may be in different working groups in different countries. The Mars Climate Observer spacecraft was lost in 1999 because a piece of control software from Lockheed Martin used imperial units instead of the metric units the interfacing NASA software expected (NASA 1999). Calculation errors are distressingly common. There are no reliable statistics on the calculation errors made in risk assessment or, even more broadly, within scientific papers. However, there is research on errors made in some very simple calculations performed in hospitals. Dosing errors give an approximate estimate of how often mathematical slips occur. Errors in drug charts occur at a rate of 1.2-31% across different studies  (Prot et al. 2005; Stubbs, Haw, and Taylor 2006; Walsh et al. 2008) , with a median of roughly 5% of administrations. Of these errors, 15-40% were dose errors, giving an overall dose error rate of about 1-2%. What does this mean for error rates in risk estimation? Since the stakes are high when it comes to dosing errors, this data represents a serious attempt to get the right answer in a life or death circumstance. It is likely that the people doing risk estimation are more reliable at arithmetic than health professionals and have more time for error correction, but it appears unlikely that they would be more reliable in many orders of magnitude. Hence, a chance of 10 −3 for a mistake per simple calculation does not seem unreasonable. A random sample of papers from Nature and the British Medical Journal found that roughly 11% of statistical results were flawed, largely due to rounding and transcription errors (García-Berthou and Alcaraz 2004). Calculation errors include more than just the 'simple' slips which we know from school, such as confusing units, forgetting a negative square root or incorrectly transcribing from the line above. Instead, many mistakes arise here due to numerical implementation of the analytic mathematical equations. Computer-based simulations and numerical analysis are rarely straightforward. The history of computers contains a large number of spectacular failures due to small mistakes in hardware or software. The 4 June 1996 explosion of an Ariane 5 rocket was due to a leftover piece of code triggering a cascade of failures  (ESA 1996) . Audits of spreadsheets in real world use find error rates on the order of 88%  (Panko 1998) . The 1993 Intel Pentium floating point error affected 3-5 million processors, reducing their numeric reliability and hence our confidence in anything calculated with them  (Nicely 2008) . Programming errors can remain dormant for a long time even in apparently correct code, only to emerge under extreme conditions. An elementary and widely used binary search algorithm included in the standard libraries for Java was found after nine years to contain a bug that emerges only when searching very large lists  (Bloch 2006) . A mistake in data processing led to the retraction of five high-profile protein structure papers as the handedness of the molecules had become inverted  (Miller 2006) . In cases where computational methods are used in modelling, many mistakes cannot be avoided. Discrete approximations of the often continuous model equations are used, and in some cases we know that the discrete version is not a good proxy for the continuous model  (Morawetz and Walke 2003) . Moreover, numerical evaluations are often done on a discrete computational grid, with the values inside the meshes being approximated from the values computed at the grid points. Though we know that certain extrapolation schemes are more reliable in some cases than others, we are often unable to exclude the possibility of error, or to even quantify it.
 Ways of modelling and theorizing Our distinction between model and theory follows the typical use of the terms within mathematical sciences like physics or economics. Whereas theories are associated with broad applicability and higher confidence in the correctness of their description, models are closer to the phenomena. For example, when estimating the probability of a particular asteroid colliding with the Earth, one would use either Newtonian mechanics or general relativity as a theory for describing the role of gravity. One could then use this theory in conjunction with observations of the bodies' positions, velocities and masses to construct a model, and finally, one could perform a series of calculations based on this model to estimate the probability of impact. As this shows, the errors that can be introduced in settling for a specific model include and surpass those which are sometimes referred to as parameter uncertainty. As well as questions of the individual parameters (positions, velocities and masses), there are important questions of detail (Can we neglect the inner structure of the involved bodies?), and breadth (Can we focus on the Earth and asteroid only, or do we have to model other planets, or the Sun?).  7  As can be seen from this example, one way to distinguish theories from models is that theories are too general to be applied directly to the problem. For any given theory, there are many ways to apply it to the problem and these ways give rise to different models. Philosophers of science will note that our theory-model distinction is in accordance with the non-uniform notion used by  Giere (1999) ,  Morrison (1998) ,  Cartwright (1999) , and others, but differs from that of the semantic interpretation of theories  (Suppes 1957) . We should also note that it is quite possible for an argument to involve several theories or several models. This complicates the analysis and typically provides additional ways for the argument to be flawed.  8  For example, in estimating the risk of black hole formation at the LHC, we not only require quantum chromodynamics (the theory on which the LHC is built to test), but also relativity and Hawking's theory of black hole radiation. In addition to their other roles, modelling assumptions also have to explain how to glue such different theories together  (Hillerbrand and Ghil 2008) . In risk assessment, the systems involved are most often not as well understood as asteroid impacts. Often, various models exist simultaneously -all known to be incomplete or incorrect in some way, but difficult to improve upon. 9 Particularly in these cases, having an expected or desired outcome in mind while setting up a model makes one vulnerable to expectation bias: the tendency to reach the desired answer rather than the correct one. This bias has affected many of science's great names  (Jeng 2006) , and in the case of risk assessment, the desire for a 'positive' outcome (safety in the case of the advocate or danger in the case of the protestor) seems a likely cause of bias in modelling.
 Historical examples of model and theory failure A dramatic example of a model failure was the Castle Bravo nuclear test on 1 March 1954. The device achieved 15 megatons of yield instead of the predicted four to eight megatons. Fallout affected parts of the Marshal Islands and irradiated a Japanese fishing boat so badly that one fisherman died, causing an international incident (Nuclear Weapon Archive 2006). Though the designers at Los Alamos National Laboratories understood the involved theory of alpha decay, their model of the reactions involved in the explosion was too narrow, for it neglected the decay of one of the involved particles (lithium-7), which turned out to contribute the bulk of the explosion's energy. The Castle Bravo test is also notable for being an example of model failure in a very serious experiment conducted in the hard sciences and with known high stakes. The history of science contains numerous examples of how generally accepted theories have been overturned by new evidence or understanding, as well as a plethora of minor theories that persisted for a surprising length of time before being disproven. Classic examples for the former include the Ptolemaic system, phlogiston theory and caloric theory; an example for the latter is human chromosome number, which was systematically miscounted as 48 (rather than 46) and this error persisted for more than 30 years  (Gartler 2006 ). As a final example, consider Lord Kelvin's estimates of the age of the Earth  (Burchfield 1975) . They were based on information about the earth's temperature and heat conduction, estimating an age of the Earth between 20 and 40 million years. These estimates did not take into account radioactive heating, for radioactive decay was unknown at the time. Once it was shown to generate additional heat the models were quickly updated. While neglecting radioactivity today would count as a model failure, in Lord Kelvin's day it represented a largely unsuspected weakness in the physical understanding of the Earth and thus amounted to theory failure. This example makes it clear that the probabilities for the adequacy of model and theory are not independent of each other, and thus in the most general case we cannot further decompose Equation (2).
 Applying our analysis to the risks from particle physics research Particle physics is the study of the elementary constituents of matter and radiation, and the interactions between them. A major experimental method in particle physics involves the use of particle accelerators, such as the RHIC and LHC, to bring beams of particles near the speed of light and then collide them together. This focuses a large amount of energy in a very small region and breaks the particles down into their components, which are then detected. As particle accelerators have become larger, the energy densities achieved have become more extreme, prompting some concern about their safety. These safety concerns have focused on three possibilities: the formation of 'true vacuum', the transformation of the Earth into 'strange matter', and the destruction of the Earth through the creation of a black hole.
 True vacuum and strange matter formation The type of vacuum that exists in our universe might not be the lowest possible vacuum energy state. In this case, the vacuum could decay to the lowest energy state, either spontaneously, or if triggered by a sufficient disturbance. This would produce a bubble of 'true vacuum' expanding outwards at the speed of light, converting the universe into different state apparently inhospitable for any kind of life  (Turner and Wilczek 1982) . Our ordinary matter is composed of electrons and two types of quarks: up quarks and down quarks. Strange matter also contains a third type of quark: the 'strange' quark. It has been hypothesized that strange matter might be more stable than normal matter, and able to convert atomic nuclei into more strange matter  (Witten 1984) . It has also been hypothesized that particle accelerators could produce small negatively charged clumps of strange matter, known as strangelets. If both these hypotheses were correct and the strangelet also had a high enough chance of interacting with normal matter, it would grow inside the Earth, attracting nuclei at an ever higher rate until the entire planet was converted to strange matter -destroying all life in the process. Unfortunately, strange matter is complex and little understood, giving models with widely divergent predictions about its stability, charge and other properties  (Jaffe et al. 2000) . One way of bounding the risk from these sources is the cosmic ray argument: the same kind of high-energy particle collisions occur all the time in Earth's atmosphere, on the surface of the Moon and elsewhere in the universe. The fact that the Moon or observable stars have not been destroyed despite a vast number of past collisions (many at much higher energies than can be achieved in human experiments) suggests that the threat is negligible. This argument was first used against the possibility of vacuum decay  (Hut and Rees 1983)  but is quite general. An influential analysis of the risk from strange matter was carried out in  Dar, De Rujula, and Heinz (1999)  and formed a key part of the safety report for the RHIC. This analysis took into account the issue that any dangerous remnants from cosmic rays striking matter at rest would be moving at high relative velocity (and hence much less likely to interact) while head-on collisions in accelerators could produce remnants moving at much slower speeds. They used the rate of collisions of cosmic rays in free space to estimate strangelet production. These strangelets would then be slowed by galactic magnetic fields and eventually be absorbed during star formation. When combined with estimates of the supernova rate, this can be used to bound the probability of producing a dangerous strangelet in a particle accelerator. The resulting probability estimate was < 2 × 10 −9 per year of RHIC operation.  10  While using empirical bounds and experimentally tested physics reduces the probability of a theory error, the paper needs around 30 steps to reach its conclusion. For example, even if there was just a 10 −4 chance of a calculation or modelling error per step this would give a total P(¬A) ≈ 0.3%. This would easily overshadow the risk estimate. Indeed, even if just one step had a 10 −4 chance of error, this would overshadow the estimate. A subtle complication in the cosmic ray argument was noted in  Tegmark and Bostrom (2005) . The Earth's survival so far is not sufficient as evidence for safety, since we do not know if we live in a universe with 'safe' natural laws or a universe where planetary implosions or vacuum decay do occur, but we have just been exceedingly lucky so far. While this latter possibility might sound very unlikely, all observers in such a universe would find themselves to be in the rare cases where their planets and stars had survived, and would thus have much the same evidence as we do. Tegmark and Bostrom had thus found that in ignoring these anthropic effects, the previous model had been overly narrow. They corrected for this anthropic bias and, using analysis from  Jaffe et al. (2000) , concluded that the risk from accelerators was less than 10 −12 per year. This is an example of a demonstrated flaw in an important physics risk argument (one that was pivotal in the safety assessment of the RHIC). Moreover, it is significant that the RHIC had been running for five years on the strength of a flawed safety report, before Tegmark and Bostrom noticed and fixed this gap in the argument. Although this flaw was corrected immediately after being found, we should also note that the correction is dependent on both anthropic reasoning and on a complex model of the planetary formation rate  (Lineweaver, Fenner, and Gibson 2004) . If either of these or the basic Brookhaven analysis is flawed, the risk estimate is flawed.
 Black hole formation The LHC experiment at CERN was designed to explore the validity and limitations of the standard model of particle physics by colliding beams of high-energy protons. This will be the most energetic particle collision experiment ever done, which has made it the focus of a recent flurry of concerns. Due to the perceived strength of the previous arguments on vacuum decay and strangelet production, most of the concern about the LHC has focused on black hole production. None of the theory papers we have found appears to have considered the black holes to be a safety hazard, mainly because they all presuppose that any black holes would immediately evaporate due to Hawking radiation. However, it was suggested by  Dimopoulos and Landsberg (2001)  that if black holes form, particle accelerators could be used to test the theory of Hawking radiation. Thus, critics also began questioning whether we could simply assume that black holes would evaporate harmlessly. A new risk analysis of LHC black hole production (Giddings and Mangano 2008) provides a good example of how risks can be more effectively bounded through multiple sub-arguments. While never attempting to give a probability of disaster (rather concluding 'there is no risk of any significance whatsoever from such black holes'), it uses a multiple bounds argument. It first shows that rapid black hole decay is a robust consequence of several different physical theories (A 1 ). Second, it discusses the likely incompatibility between non-evaporating black holes and mechanisms for neutralizing black holes: in order for cosmic ray-produced stable black holes to be innocuous but accelerator-produced black holes to be dangerous, they have to be able to shed excess charge rapidly (A 2 ). Our current understanding of physics suggests both that black holes decay and that even if they did not, they would be unable to discharge themselves. Only if this understanding is flawed will the next section come into play. The third part, which is the bulk of the paper, models how multidimensional and ordinary black holes would interact with matter. This leads to the conclusion that if the size scale of multidimensional gravity is smaller than about 20 nm, then the time required for the black hole to consume the Earth would be larger than the natural lifetime of the planet. For scenarios where rapid Earth accretion is possible, the accretion time inside white dwarves and neutron stars would also be very short, yet production and capture of black holes from impinging cosmic rays would be so high that the lifespan of the stars would be far shorter than the observed lifespan (and would contradict white dwarf cooling rates) (A 3 ). The force of the total argument (A 1 , A 2 , A 3 ) is significantly stronger than any of its parts. Essentially, the paper acts as three sequential arguments, each partly filling in the grey area (see Figure  1 ) left by the previous. If the theories surrounding black hole decay fail, the argument about discharge comes into play; and if against all expectation black holes are stable and neutral, the third argument shows that astrophysics constrains them to a low accretion rate.
 Implications for the safety of the LHC What are the implications of our analysis for the safety assessment of the LHC? Firstly, let us consider the stakes in question. If one of the proposed disasters were to occur, it would mean the destruction of the Earth. This would involve the complete destruction of the environment, 6.5 billion human deaths and the loss of all future generations. It is worth noting that this loss of all future generations (and with it, all of humanity's potential) may well be the greatest of the three, but a comprehensive assessment of these stakes is outside the scope of this paper. For the present purposes, it suffices to observe that the destruction of the Earth is at least as bad as 6.5 billion human deaths. There is some controversy as to how one should combine probabilities and stakes into an overall assessment of a risk. Some hold that the simple approach of expected utility is the best, while others hold some form of risk aversion. However, we can sidestep this dispute by noting that in either case, the risk of some harm is at least as bad as the expected loss. Thus, a risk with probability p of causing a loss at least as bad as 6.5 billion deaths is at least as bad as a certain 6.5 × 10 9 p deaths. Now let us turn to the best estimate we can make of the probability of one of the above disasters occurring during the operation of the LHC. While the arguments for the safety of the LHC are commendable for their thoroughness, they are not infallible. Although the report considered several possible physical theories, it is eminently possible that these are all inadequate representations of the underlying physical reality. It is also possible that the models of processes in the LHC or the astronomical processes appealed to in the cosmic ray argument are flawed in an important way. Finally, it is possible that there is a calculation error in the report. In Equation (1), P(X) is the sum of two terms. The second of these represents the additional probability of disaster due to the argument being unsound. It is the product of the probability of argument failure and the probability of disaster given such a failure. Both terms are very difficult to estimate, but we can gain insight by showing the ranges they would have to lie within, for the risk presented by the LHC to be acceptable. If we let l denote the acceptable limit of expected deaths from the operation of the LHC, we get: 6.5 × 10 9 P(X) ≤ l. Since P(X) is at least as great as its second term, we obtain: This inequality puts a severe bound on the acceptable values for these probabilities. Since it is much easier to grasp this with an example, we shall provide some numbers for the purposes of illustration. Suppose, for example, that if the limits were set at 1000 expected deaths, then P(X|¬A) P(¬A) would have to be below 1.5 × 10 −7 for the risk to be worth bearing. This requires very low values for these probabilities. We have seen that for many arguments, P(¬A) is above 10 −3 . We have also seen that the argument for the safety of the RHIC turned out to have a significant flaw, which was unnoticed by the experts at the time. It would thus be very bold to suppose that the argument for the safety of the LHC was much lower than 10 −3 , but for the sake of argument, let us grant that it is as low as 10 −4 -that out of a sample of 10,000 independent arguments of similar apparent merit, only one would have any serious error. Even if the value of P(¬A) were as low as 10 −4 , P(X|¬A) would have to be below 0.15% for the risk to be worth taking. P(X|¬A) is the probability of disaster, given that the arguments of the safety report are flawed and is the most difficult component of Equation (  1 ) to estimate. Indeed, few would dispute that we really have very little idea of what value to put on P(X|¬A). It would thus seem overly bold to set this below 0.15% without some substantive argument. Perhaps such an argument could be provided, but until it is, such a low value for P(X|¬A) seems unwarranted. We stress that the above combination of numbers was purely for illustrative purposes, but we cannot find any plausible combination of the three numbers which meets the bound and which would not require significant argument to explain either the levels of confidence or the disregard for expected deaths. We would also like to stress that we are open to the possibility that additional supporting arguments and independent verification of the models and calculations could significantly reduce the current chance of a flaw in the argument. However, our analysis implies that the current safety report should not be the final word in the safety assessment of the LHC. To proceed with the LHC on the arguments of the most recent safety report alone, we would require further work on estimating P(¬A), P(X|¬A), the acceptable expected death toll, and the value of future generations and other life on Earth. Such work would require expertise beyond theoretical physics, and an interdisciplinary group would be essential. If the stakes were lower, then it might make sense for pragmatic concerns to sweep aside this extra level of risk analysis, but the stakes are astronomically large, and so further analysis is critical. Even if the LHC goes ahead without any further analysis, as is very likely, these lessons must be applied to the assessment of other high-stakes low-probability risks.
 Conclusions When estimating threat probabilities, it is not enough to make conservative estimates (using the most extreme values or model assumptions compatible with known data). Rather, we need robust estimates that can handle theory, model and calculation errors. The need for this becomes considerably more pronounced for low-probability high-stake events, though we do not say that low probabilities cannot be treated systematically. Some people have raised the concern that our argument might be too powerful, for it is impossible to disprove the risk of even something as trivial as dropping a pencil, P X A P A l ( | ) ( ) . . ( ) ¬ ¬ ≤ × − 1 5 10 3 then our argument might amount to prohibiting everything. It is true that we cannot completely rule out any probability that apparently inconsequential actions might have disastrous effects, but there are a number of reasons why we do not need to worry about universal prohibition. A major reason is that for events like the dropping of a pencil which have no plausible mechanism for destroying the world, it seems just as likely that the world would be destroyed by not dropping the pencil. The expected losses would thus balance out. It is also worth noting that our argument is simply an appeal to a weak form of decision theory to address an unusual concern: for our method to lead to incorrect conclusions, it would require a flaw in decision theory itself, which would be very big news. It will have occurred to some readers that our argument is fully applicable to this very paper: there is a chance that we have made an error in our own arguments. We entirely agree, but note that this possibility does not change our conclusions very much. Suppose, very pessimistically, that there is a 90% chance that our argument is sufficiently flawed that the correct approach is to take safety reports' probability estimates at face value. Even then, our argument would make a large difference to how we treat such values. Recall the example from Section 2, where a report concludes a probability of 10 −9 and we revise this to 10 −6 . If there is even a 10% chance that we are correct in doing so, then the overall probability estimate would be revised to 0.9 × 10 −9 + 0.1 × 10 −6 ≈ 10 −7 , which is still a very significant change from the report's own estimate. In short, even serious doubt about our methods should not move one's probability estimates more than an order of magnitude away from those our method produces. More modest doubts would have negligible effect. The basic message of this paper is that any scientific risk assessment is only able to give us the probability of a hazard occurring conditioned on the correctness of its main argument. The need to evaluate the reliability of the given argument in order to adequately address the risk was shown to be of particular relevance in low-probability high-stake events. We drew a three-fold distinction between theory, model and calculation, and showed how this can be more useful than the common dichotomy in risk assessment between model and parameter uncertainties. By providing historic examples for errors in the three fields, we clarified the three-fold distinction and showed where flaws in a risk assessment might occur. Our analysis was applied to the recent assessment of risks that might arise from experiments within particle physics. To conclude this paper, we now provide some very general remarks on how to avoid argument flaws when assessing risks with high stakes. Firstly, the testability of predictions can help discern flawed arguments. If a risk estimate produces a probability distribution for smaller, more common disasters, this can be used to judge whether the observed incidences are compatible with the theory. Secondly, reproducibility appears to be the most effective way of removing many of these errors. By having other people replicate the results of calculations independently, our confidence in them can be dramatically increased. By having other theories and models independently predict the same risk probability, our confidence in them can again be increased, as even if one of the arguments is wrong the others will remain. Finally, we can reduce the possibility of unconscious bias in risk assessment through the simple expedient of splitting the assessment into a 'blue' team of experts attempting to make an objective risk assessment and a 'red' team of devil's advocates attempting to demonstrate a risk, followed by repeated turns of mutual criticism and updates of the models and estimates  (Calogero 2000) . Application of such methods could in many cases reduce the probability of error by several orders of magnitude. Figure 2 . 2 Figure 2. Ways in which risk assessments can be flawed.
 Figure 2 . 2 Figure 2. Ways in which risk assessments can be flawed.
			 *Corresponding author. Email: <EMAIL>