ML Agent Safety Mechanisms based on Counterfactual Planning
Year: 2021

Abstract:
We present counterfactual planning as a design approach for creating a range of 1 safety mechanisms for machine learning agents. We specifically target the safety 2 problem of keeping control over hypothetical future AGI agents. The key step 3 in counterfactual planning is to use the agent’s machine learning system to con4 struct a counterfactual world model, designed to be different from the real world 5 the agent is in. A counterfactual planning agent determines the action that best 6 maximizes expected utility in this counterfactual planning world, and then per7 forms the same action in the real world. The design approach is built around a 8 two-diagram graphical notation that provides a specific vantage point on the con9 struction of online machine learning agents, a vantage point designed to make the 10 problem of control more tractable. We show two examples where the construction 11 of a counterfactual planning world acts to suppress certain unsafe agent incentives, 12 incentives for the agent to take control over its own safety mechanisms. 13

We present counterfactual planning as a design approach for creating a range of1 safety mechanisms for machine learning agents. We specifically target the safety2 problem of keeping control over hypothetical future AGI agents. The key step3 in counterfactual planning is to use the agent’s machine learning system to con-4 struct a counterfactual world model, designed to be different from the real world5 the agent is in. A counterfactual planning agent determines the action that best6 maximizes expected utility in this counterfactual planning world, and then per-7 forms the same action in the real world. The design approach is built around a8 two-diagram graphical notation that provides a specific vantage point on the con-9 struction of online machine learning agents, a vantage point designed to make the10 problem of control more tractable. We show two examples where the construction11 of a counterfactual planning world acts to suppress certain unsafe agent incentives,12 incentives for the agent to take control over its own safety mechanisms.13
1 Introduction14
Artificial General Intelligence (AGI) systems are hypothetical future machine reasoning systems15 that match or exceed the capabilities of humans in general problem solving. While it is still unclear16 if AGI systems could ever be built, we can already study AGI related risks and potential safety17 mechanisms [2, 6, 19]. At this time, the still-young field of AGI safety engineering is considering18 a multidisciplinary multitude of problems and methodological ideas. The main problem considered19 in this paper is the problem of control [19].20
In the most well-known example of the problem of control, we equip an autonomous AI agent21 both with a seemingly innocent goal like fetching the coffee [19], and with an emergency stop22 button. If the agent is perceptive enough, it may see that it cannot fulfill its goal if someone presses23 the stop button first. So it might disable its stop button, or seek to disable any human who might24 conceivably want to press it. The stop button problem is not unique to AGI-level agents: for example25 [16] constructs a toy grid-world where a very basic ML agent will consistently learn to disable its26 own stop button. We feel that it is theoretically interesting to consider the problem of designing27 an emergency stop button that is robust in the general case, even though such general solutions28 are not needed for the safety engineering of current ML agents. But if powerful AGI agents are29 ever developed, robust and general designs may turn out to have important practical applications.30 Several partial solutions to the general stop button problem have been identified and proposed in31 [1, 9, 10, 20].32
The main contribution of counterfactual planning is to offer a new vantage point for modeling and33 specifying machine learning agents, developed to make the problem of control more tractable. This34 vantage point is offered by a compact and readable graphical language for depicting the complex35
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
types of self-referencing and indirect representation which are typically present inside machine36 learning agents.37
The aim of this conference paper is to present the core elements of counterfactual planning to a38 technical audience of readers who are already somewhat familiar with Pearl causal models [17]39 and the use of mathematical models to specify cyber-physical systems. A more extensive 39-page40 presentation of counterfactual planning is available as an arXiv preprint [4]. The preprint devotes41 significant space to presenting the material in section 2 in a way that is accessible to a broader42 audience. It also develops the methodology of counterfactual planning in more detail, and includes43 a broader range of examples, examples of safety mechanism design and of failure mode analysis.44
1.1 Related work45
Relation to counterfactual fairness. While this was not originally intended or expected, the46 methodological vantage point of counterfactual planning ended up being very similar to that of47 counterfactual fairness [14]. Both methodologies to some extent invert a default goal of machine48 learning, the goal to improve the machine’s predictive accuracy. Instead, they seek to improve out-49 comes by introducing a calibrated form of machine ignorance or machine indifference.50
Use of counterfactuals in machine reasoning. In the general AI/ML literature which is con-51 cerned with improving system performance, counterfactual planning and projection have been used52 to improve performance in several application domains, see for example [23] and [3].53
In the AI safety/alignment literature, there are several system designs which add counterfactual54 terms to the agent’s reward function. Examples are [1, 10] in the AGI control specific literature, and55 [12, 13, 21, 22], which look at reducing unwanted side effects caused by both current and potential56 future agents. In the literature on encoding specific human values into machine reasoning systems,57 counterfactuals have also been used to encode values beyond non-discriminatory fairness [14], for58 example in [18].59
Graphical models of agents and decision making. Influence diagrams [11] provide a graphical60 notation for depicting utility-maximizing decision making processes. [8] combines Pearl causal61 models [17] with influence diagrams to define the Causal influence diagram (CID) notation. [7]62 proposes the agenda of using CIDs to model and compare AGI safety frameworks. This work was63 in part inspired by that CID agenda, and models some safety frameworks that were not modeled64 earlier. In a departure from previous work, we use two CIDs to model a single reward-maximizing65 agent. By doing so, we more clearly foreground the details of the agent’s machine learning system.66
Graphical models to clarify reward tampering. [5] develops several single-diagram models pro-67 vide more insight into the problem of reward tampering. The two-diagram model in section 5 also68 aims to provide more insight.69
2 Two-diagram graphical notation for agent definitions70
This section defines the two-diagram graphical notation which is central to counterfactual planning.71 Later sections will use the notation to define agents with specific built-in safety mechanisms.72
Figure 1 shows an example causal world model as used in our graphical notation. The model is73 an annotated directed acyclic graph, where the nodes represent observables in a real or imaginary74 world, and the arrows (the directed edges) denote causal relations between these observables. While75 this is not the case for all Pearl causal models [17], in this paper the causal arrows also always denote76 the arrow of time. Color is used in the graph to highlight structure only. The dots on the right hand77 side denote that the graph depicted has an infinitely repeating structure. The label (l) in the upper78 left corner names the causal world model and its corresponding world: we say that figure 1 depicts79 the (world model of) the world l.80
We depart from the definitional conventions used by Pearl and many other authors by treating the81 annotated graph in figure 1 as the sole definition of a causal model l, as opposed to being merely a82 convenient depiction of some of the information in a tuple which defines l. To make this work, we83 annotate each graph node by writing the name of the corresponding structural function above it. We84 write [F ] to name a nondeterministic structural function and F for a deterministic one. We interpret85 the five structural function names π, s0, S, o0 and O written above the nodes in figure 1 as model86 parameters. The two model parameters s0 and o0 are functions taking zero arguments.87
Figure 1 models a world containing an online machine learning agent. In each time step t ≥ 0,88 the agent takes an action at = π(ot, st) selected by the deterministic policy function π. This π is89 informed not only by current state of the agent environment st, but also by an observational record90 ot, a record that captures earlier interactions between the agent and its environment. The initial state91 of the agent environment is given by s0, and the state transitions of the environment are driven by the92 probability density function S, where S(st+1, st, at) is the probability that the agent environment93 transitions from state st to st+1 when the agent performs action at.94
In its mapping to probability theory, as formally defined further below, the world model l defines95 three time series of random variables named At,l, St,l, and Ot,l with t ≥ 0. The ,l subscript in these96 random variable names is used for disambiguation with other world models using the same node97 names. P (At,l = at) is the probability that the agent in world l will take the exact action at at98 time t. The world state of l at time t is given by the combined values of the three random variables99 At,l, St,l, and Ot,l. This means that we break with the convention commonly used in MDP models,100 where the term world state refers to St,l only. We call St,l the agent environment state.101
We use the modeling convention that the physical realizations of the agent’s sensors and actuators102 are modeled inside the environment states St,l. This means that we interpret the arrows from the St103 nodes into the At nodes as digital sensor signals which flow into the agent’s compute core, and the104 arrows out of the At nodes as digital actuator command signals which flow out. The observational105 record nodes Ot are also inside the agent’s compute core.106
2.1 Online machine learning107
Any work that aims to model hypothetical future AGI agents faces the problem of modeling hypo-108 thetical future machine learning systems. Here, we choose to model all machine learning as a type109 of function approximation. We call the world l with the agent above a learning world, and define110 that the objective of its machine learning system is to approximate the function S which determines111 the behavior of the learning world agent environment. We model this machine learning system as a112 function L which takes an observational record o to produce a learned function L = L(o), where113 we intend that L ≈ S.114 We model observational record keeping as a generic process that simply builds a list of all past115 observations. With ++ being the operator which adds an extra record to the end of a list, we define116 the model parameter O as117
O(ot−1, st−1, at−1, st) = ot−1 ++ (st, st−1, at−1)
The initial observational record o 0 may be the empty list, but it might also be a long list of observa-118 tions from earlier agent training runs, in the same environment or in a simulator. The training runs119 may also have used a different policy π.120
2.2 Formal semantics of a graphical world model121
We now formally define the meaning of the graphical notation used in diagrams like figure 1, by122 providing a mapping to probability theory.123
Definition 1 (Random variables defined by a diagram). A diagram with the label (l) written next to124 the graph and the labels X0, · · · , Xn in the graph nodes defines a set of random variables named125 X0,l, · · · , Xn,l. We treat these random variables as representing observables in a (hypothetical or126 real) world, the world called l.127 Definition 2 (Parent, Pa, pa). We call a random variable Xp,l a parent of the random variable Xc,l128 if and only if there is an arrow from the node Xp to node Xc in the diagram l. PaXc,l is the list of129 all parent random variables of Xc,l, with the order of appearance determined by considering each130 incoming arrow of Xc in a clockwise order, starting from the 6-o-clock position. paXc,l is the list of131 lowercase variable names we get by converting PaXc,l to lowercase and removing the , l parts from132 the subscripts.133
For example, with figure 1 above, PaA1,l is the list O1,l, S1,l, and paA1,l is the list o1, s1.134 Definition 3 (Constraints on the random variables defined by the arrows). The arrows in a diagram l135 depict causal relations between observables in the world l: they constrain the values of the associated136 random variables. Take a finite set S of random of random variables X1,l, · · · , Xm,l defined by l, a137 set with the property that if an Xc,l is in S, all variables PaXc,l are also in S. We have that138
P (X1,l = x1, · · · , Xm,l = xm) = P (X1,l = x1|PaX1,l = paX1,l) · . . . · P (Xm,l = xm|PaXm,l = paXm,l).
Definition 4 (Constraints on the random variables defined by the model parameters). Take a node139 Xi in a diagram l. Then:140 1. If he model parameter above the node is written as [F ], we have that141
P (Xi,l = x|PaXi,l = paXi,l) = F (x,paXi,l)142 where we require that the function F satisfies ∀paXi,l ( ∑ x F (x,paXi,l) = 1).143
2. If the model parameter above the node is written as F , we have that144
P (Xi,l = x|PaXi,l = paXi,l) = (ifx = F (paXi,l) then 1 else 0).145
For example, with figure 1 above, we have that P (A1,l = π(o, s)|A1,l = o, S1,l = s) = 1.146
2.3 Reward-maximizing decision making147
To define reward-maximizing agent policies, we draw graphical world models we call planning148 worlds. The purpose of a planning world diagram is to define a reward-maximizing policy that can149 be computed, or at least approximated, by a real-life agent compute core. This approximation may150 again use machine learning techniques.151
Two example planning world diagrams are shown in figure 2. These are graphical world models152 where some graph nodes have special shapes. The square decision nodes denote actions which can153 be freely picked by a decision making process, and the diamond-shaped utility nodes denote values154 that the decision making process will try to maximize. In both examples, the utility node values are155 computed by a reward function R, the function which encodes the agent’s goals.156
The two diagrams in figure 2 act as specifications of two policy functions π∗p and π ∗ q . We now define157 how this specification process works. First, in any diagram a with diamond-shaped utility nodes,158 these nodes define a metric Ua:159
Definition 5 (Expected utility Ua of a diagram a). We define Ua for two cases:160 1. If there is only one utility node R0 in a, then Ua = E(R0,a).161 2. If there are multiple utility nodes Rt in a, with integer subscripts running from l to h, then1623
Ua = E( h∑ t=l γtRt,a )
where γ is a time discount factor, 0 < γ ≤ 1, which can be read as an extra model164 parameter.165
With the infinitely repeating graph of diagram p, we have h = ∞, so we generally need a γ < 1 to166 ensure that Up is a well-defined and computable value.167 Definition 6 (Policy function defined by a diagram). A diagram a with some utility and decision168 modes, where a function π∗a is written above all decision nodes, defines this π ∗ a in two steps.169 1. First, draw a helper diagram b by drawing a copy of diagram a, except that every decision170 node has been drawn as a round node, and every π∗a has been replaced by a fresh function171 name, say π′.172
2. Then, π∗a is defined by π ∗ a = argmaxπ′ Ub, where the argmaxπ′ operator always de-173 terministically returns the same function if there are several candidates that maximize its174 argument.175
2.4 Agent specifications176
The learning world model l in figure 1 can model any possible machine learning agent, as it admits177 every possible value for its model parameter π. An agent specification is a statement which defines178 the exact details of the π in such a model. By doing so, it specifies the exact properties we want an179 agent compute core to have. Our agent specifications are statements that combine a learning world180 with a planning world, for example:181
FP The factual planning agent has the learning world l, where π(o, s) = π∗p(s), with π∗p defined by the planning world p, where L = L(o). 182 183
To make agent specifications stand out, we always typeset them as shown above. When we talk184 about the safety properties of the FP agent, we refer to the outcomes which the defined agent policy185 π will produce in the learning world l.186
We call the FP agent a factual planning agent because its planning world p was designed to project187 the agent’s future in the learning world l as well as possible. A counterfactual planning world188 purposefully mis-projects a learning world agent’s future. The planning world q in figure 2 projects189 a counterfactual future lasting only a single time step. If we were to use q and π∗q in the above agent190 specification, the resulting agent would be a counterfactual planner.191
Our two-diagram notation for defining agents visually shows the availability of certain agent design192 options, options which are relevant to the problem of control.193
3 Using counterfactual planning to protect agent safety interlocks194
We now construct an agent which is equipped with two safety interlocks that will stop the agent195 if certain conditions are met. We use counterfactual planning to add a specific safety feature to196 the design: we use it to suppress certain unsafe incentives which a perceptive factual planning197 agent might otherwise develop, incentives to remove its built-in safety interlocks or to prevent their198 activation.199
As a first step, we construct a learning world si, shown on the left in figure 3, by taking the earlier200 l and adding an extra mode bit to the agent’s compute core, represented by the nodes Mt. The201 bit starts in the go mode, and permanently transitions to the stop mode when a safety interlock202 activates. In the stop mode, the agent policy π will always perform Null actions. The µ function203 specifies in detail when a safety interlock will activate.204
In the next step, we pair si with a counterfactual planning world p, shown on the right in figure 3.205 This planning world projects counterfactual futures where the safety interlock mechanism is absent206 from the agent’s compute core.207
The last step is to write the agent definition:208
SI The safety interlock agent has the learning world si, where
π(o, s,m) = { π∗p(s) if m = go Null if m = stop
with π∗p defined by the planning world p, where L = L(o) and
µ(o, s,m) = { stop if m = stop or stop pressed(s) or Up > Umax go otherwise
with Up defined by the planning world p, where L = L(o).
209
210
The µ above incorporates two safety interlocks. The first interlock, defined by the term211 stop pressed(s), is a basic emergency stop button. It can be pressed by the humans performing212 agent oversight to immediately stop the agent, for example if they observe it performing an act of213 unwanted and surprising creativity that subverts the agent’s design intentions [15]. To complete the214 design of this first interlock, we must also specify the following. In the agent environment modeled215 by s0 and S, the environment which includes the details of the agent’s sensors, the agent is equipped216 with an emergency stop button that will deliver an unambiguous signal directly to an input pin on217 the agent’s compute core. If the pin is high in a world state s, then stop pressed(s) yields true.218
The second interlock, defined by the term Up > Umax, protects against the risk that human oversight219 may be incapacitated or overwhelmed by rapid new developments. The latter might happen in220 the (arguably very speculative) case of the agent undergoing a rapid and unexpected intelligence221 explosion [2]. To protect against this risk, the interlock stops the agent when the metric Up, which222 measures the agent’s ability to achieve goals, gets too high. We can treat Up as a measure for both223 raw (expected) agent power and raw agent intelligence,224
In the field of AGI safety, stop button designs have often been discussed [1, 9, 10, 20], but the above225 power-based interlock seems to be new. What is interesting from a methodological standpoint is that226 the power-based interlock design inverts the default goal of reinforcement learning. We no longer227 consider a very high expected utility score to be an absolute good: instead we treat it as a predictor228 of unwanted outcomes.229
4 Safety analysis using the diagrams230
Our mostly-graphical specification of the SI agent was designed to make certain safety properties231 immediately visible. We can observe that, almost tautologically, the SI agent will not care about232 the safety interlocks constructed in the Mt nodes, because these nodes are absent from its planning233 world.234
The two diagrams in figure 3 also serve as a visual map which draws the eye to certain limitations235 and boundary conditions. For example:236
1. The [S] in the left hand diagram of figure 3 is replaced by an [L] on the right. If the237 approximation L ≈ S is not good enough, the agent may end up behaving in unexpected238 and potentially unsafe ways.239
2. In the world model si, the compute core internals are represented by immutable mathemat-240 ical functions like π and µ. In the real world, these functions will have to be realized by241 software running on physical compute core hardware. This hardware could be damaged or242 corrupted, by actions of the agent or by other actors.243
While the diagrams draw the eye to them, the above two problems are not specific to counterfactual244 planning agents. They are present in almost all agent designs, and the implication is that we will245 have to do further failure mode analysis to quantify and manage the residual risks involved. A more246 detailed discussion of these two problems in the context of counterfactual planning is available in247 [4].248
4.1 The planning world agent249
We now consider how the two-diagram model of counterfactual planning offers us two distinct250 narrative vantage points for safety analysis: we can consider the learning world SI agent and the251 planning world SI agent separately. A safety analysis can start with considering the motivations that252 the planning world agent would develop to take certain actions, given the mechanics and reward253 function of the planning world. Once these are determined, we can shift the perspective and discuss254 how these same actions would affect our own safety as occupants of the agent’s learning world.255
We now provide an example of narrative safety analysis which uses an economic style of reasoning.256 If we consider the planning world diagram p with L ≈ S, we can note that the function L will257 project a somewhat accurate copy of the learning world stop button hardware into the mechanics258 of the planning world. However, to the planning world agent, this hardware will appear to have no259 particularly special function. Say that R rewards paperclip production, and that the stop button has260 metal parts. Then the planning world agent might be motivated to disassemble the stop button, using261 it as a convenient source of metal for making paperclips. If the most efficient planning world policy262 involves disassembling the planning world stop button, it will likely be disassembled in the learning263 world too, which is not what we want.264
We can manage this failure mode by managing the economic cost/benefit equation with respect to265 the planning world stop button. We can consider putting physical barriers around the button, which266 would make the button unattractive as a source of metal, compared to using other near-by sources.267 To create these barriers in the planning world, we can physically construct them in the learning268 world, and then count on them being projected by L, with sufficient accuracy, into the planning269 world.270
4.2 The people in the planning world271
We now provide an example of narrative safety analysis which uses a game theoretical style of272 reasoning. We can distinguish between two groups of people: the people in the learning world and273 the people in the planning world. If the learning world is our real world, then the learning world274 people are real people. The planning world people are always models of people. But even with a275 non-AGI level machine learning system, the people in the planning world may already be modeled276 accurately enough to create very complex dynamics. Though the above SI agent construction has277 taken away a strong direct incentive for the agent to control the stop button, these complex dynamics278 may produce indirect incentives. Consider the following example.279
The learning world people will act differently towards the agent if they lose access to the agent’s280 emergency stop button. If the planning world people are accurate-enough projections, this will affect281 how the planning world agent will play certain games of coercion. The planning world agent may282 consider a policy of where it threatens to harm all people in the planning world, unless they start283 helping the agent to produce as many paperclips as possible. If L projects the mental makeup of the284 people in the learning world correctly into the planning world, then the planning world projections285 will show that a block-then-coerce policy, in which the agent first blocks access to the emergency286
stop button before it starts making threats, will work surprisingly well. If this policy is very success-287 ful in optimizing planning world paperclip production, the learning world people may end up being288 coerced in the same way.289
We would of course hope that any policy of planning world coercion will be effectively suppressed290 in all cases by certain penalty terms present in R. The above failure mode analysis applies mainly to291 the case where the agent finds creative loopholes in these penalty terms. To reduce the residual risk292 due to possible loopholes, we can take further game theoretical counter measures. We can instruct293 agent oversight to be on the alert for any block-and-X move, so that they can press the stop button294 before the agent succeeds in blocking access. The power-based Up > Umax safety interlock offers295 a further line of defense. A projected massive increase in paperclip production following successful296 coercion will raise Up over Umax, stopping the agent.297
5 Live updating of the agent reward function298
One way to control an agent is to equip it with an input terminal, which can be used by humans299 to apply live updates to the agent’s reward function R [10]. A problem with this setup is that by300 default, it gives a perceptive factual planning agent a huge incentive to immediately take control301 over its own input terminal, and input a new reward function like Rhuge(s, s′) = 1010000. We now302 use counterfactual planning to specify an agent design that overcomes this problem.303
On the left hand side in figure 4, we have taken the basic learning world diagram from figure 1304 and modified it by splitting the agent environment state St into two components. The nodes It305 represent the value of a digital signal from the input terminal, as received by the compute core in306 each time step. This signal encodes a reward function. The nodes Xt model all the rest of the agent307 environment state. On the right of figure 4 we define a planning world which uses a learned L to308 project an agent future where the reward function input signal never changes. The agent definition309 is then:310
IT The input terminal agent has the learning world li where π(i, o, x) = π∗i (x), where π∗i is defined by the planning world pi, where L = L(o), and IR(i, xt, xt+1) = i(xt, xt+1). 311 312
This IT agent constructed here has the same basic behavior as that of the indifference methods based313 agent constructions in [1, 10]. The main difference is in the mathematical language used to perform314 the construction: [1] and [10] construct their agents using a somewhat opaque balancing term. The315 resulting agent construction is difficult to visualize and interpret, and these papers rely on somewhat316 dense mathematical proofs to show that the intended safety properties are indeed being created.317
The two-diagram agent model of counterfactual planning allows the same safety properties to be318 built in a more intuitive way, where the desired indifference to change is created somewhat tauto-319 logically by construction. Mathematically. the IT agent above satisfies the agent safety property320 S1 defined in [10] by construction, as it directly implements the right hand side of this S1. Safety321 property S1 [10] in is mathematically equivalent to the first sentence of theorem 4.1 in [1], and to the322
combination of the corrigibility desiderata 1 and 5 in section 2 in [20]. Again, the IT agent satisfies323 all of these properties somewhat tautologically.324
6 Conclusions325
We have presented counterfactual planning as a general design approach for creating a range of326 AGI safety mechanisms. We have graphically constructed two agents with certain safety proper-327 ties, where these properties are directly visible in the construction. We have also introduced a new328 power-based safety interlock design. Our main contributions are methodological in nature. We have329 constructed a new vantage point that makes certain problems of design and failure mode analysis330 more tractable. By using the narrative framing of learning worlds versus planning worlds, we can331 keep track of certain levels of indirect representation in the agent design, while using a style of anal-332 ysis that borrows from economics and game theory. Further discussion of counterfactual planning333 is available in a 39-page preprint [4].334
6.1 Limitations335
We feel that the techniques and results presented in this paper are theoretically and methodologically336 interesting, but their practical usefulness might end up being limited. Fundamentally, when doing337 speculative safety engineering for hypothetical future AGI systems, there are many unknowns. We338 have modeled current and future machine learning in a very general way, so that even ‘black box’339 machine learning systems that produce largely opaque world models can be used in the agent’s340 design. But it is very possible that some or all future AGI systems, if they are ever developed, will341 use architectures that cannot efficiently incorporate the safety mechanisms developed here.342
The safety mechanisms shown here suppress certain unsafe agent incentives, but they do not remove343 all possible incentives towards unsafe behavior, see for example section 4.2. There is a risk that the344 use of these safety mechanisms may have counter-productive effects, by creating a false sense of345 security. Conceivably, a policy process may lead to unsafe outcomes where the mere inclusion of346 these mechanisms in a deployed system is accepted as an alternative to conducting a more thorough347 safety and impact review.348
6.2 Broader impact349
AI safety is not just a technical problem, but also a policy problem. While technical progress on350 safety can sometimes be made by leveraging a type of mathematics that is only accessible to handful351 of specialists, policy progress typically requires the use of more accessible but still well-defined352 language. One specific aim of this work, especially in the 39-page preprint [4], is to develop a well-353 grounded vocabulary for describing potential safety solutions that is also as accessible as possible.354 Policy discussions can move faster, and produce better and more equitable outcomes, when the355 description of a proposal and its limitations can be made more accessible to all stakeholder groups.356
As a work targeted as AI/ML system designers, this paper intends to draw attention to the design357 approach also used in counterfactual fairness: seeking to improve outcomes by creating a form of358 machine ignorance, by creating a system that mis-predicts the future in certain systematic ways.359 Our graphical agent definitions show that the design goal of perfect machine learning, L = S, is not360 necessarily in conflict with the design goal of systematic mis-prediction.361
References362 [1] Stuart Armstrong. Motivated value selection for artificial agents. In Workshops at the Twenty-363 Ninth AAAI Conference on Artificial Intelligence, 2015.364
[2] Nick Bostrom. Superintelligence: paths, dangers, strategies. Oxford University Press, 2014.365
[3] Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X. Charles, D. Max Chickering,366 Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and367 learning systems: The example of computational advertising. Journal of Machine Learning368 Research, 14(65):3207–3260, 2013.369
[4] Anonymous et al. Counterfactual planning in AGI systems. arXiv, 2021.370
[5] Tom Everitt and Marcus Hutter. Reward tampering problems and solutions in reinforcement371 learning: A causal influence diagram perspective. arXiv:1908.04734, 2019.372
[6] Tom Everitt, Gary Lea, and Marcus Hutter. AGI safety literature review. In Proceedings of the373 27th International Joint Conference on Artificial Intelligence, pages 5441–5449. AAAI Press,374 2018.375
[7] Tom Everitt, Ramana Kumar, Victoria Krakovna, and Shane Legg. Modeling AGI safety376 frameworks with causal influence diagrams. arXiv:1906.08663, 2019.377
[8] Tom Everitt, Ryan Carey, Eric Langlois, Pedro A Ortega, and Shane Legg. Agent incentives:378 A causal perspective. In Proceedings of the 35th International Joint Conference on Artificial379 Intelligence. AAAI Press, 2021.380
[9] Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The off-switch game.381 In Workshops at the Thirty-First AAAI Conference on Artificial Intelligence, 2017.382
[10] Koen Holtman. AGI agent safety by iteratively improving the utility function.383 arXiv:2007.05411, 2020.384
[11] Ronald A Howard and James E Matheson. Influence diagrams. Decision Analysis, 2(3):127–385 143, 2005.386
[12] Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg. Penaliz-387 ing side effects using stepwise relative reachability. arXiv:1806.01186, 2018.388
[13] Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, and Shane Legg. Avoiding389 side effects by considering future tasks. In Advances in Neural Information Processing Systems390 33 (NeurIPS 2020), pages 19064–19074, 2020.391
[14] Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In392 Advances in neural information processing systems (NEURIPS 2017), pages 4066–4076, 2017.393
[15] Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu,394 Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. The surprising395 creativity of digital evolution: A collection of anecdotes from the evolutionary computation396 and artificial life research communities. Artificial life, 26(2):274–306, 2020.397
[16] Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq,398 Laurent Orseau, and Shane Legg. AI safety gridworlds. arXiv:1711.09883, 2017.399
[17] Judea Pearl. Causality. Cambridge university press, 2009.400
[18] Luı́s Moniz Pereira and Ari Saptawijaya. Agent morality via counterfactuals in logic program-401 ming. In Bridging@CogSci2017, pages 39–53, 2017.402
[19] Stuart Russell. Human compatible: Artificial intelligence and the problem of control. Penguin403 Random House, 2019.404
[20] Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. In405 Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.406
[21] Alex Turner, Neale Ratzlaff, and Prasad Tadepalli. Avoiding side effects in complex envi-407 ronments. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages408 21406–21415, 2020.409
[22] Alexander Matt Turner, Dylan Hadfield-Menell, and Prasad Tadepalli. Conservative agency410 via attainable utility preservation. In Proceedings of the AAAI/ACM Conference on AI, Ethics,411 and Society, pages 385–391, 2020.412
[23] Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret mini-413 mization in games with incomplete information. In Advances in neural information processing414 systems (NEURIPS 2008), pages 1729–1736, 2008.415
Checklist416
1. For all authors...417
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s418 contributions and scope? [Yes] A caveat: ’accurately reflect’ depends on the back-419 ground of the reader. This paper was written for readers with a CS/ML background.420 In the broader multidisciplinary AI safety community, it is common to see remarks421 from participants with an activist or philosophical background, remarks which are422 highly critical about the coded language being used in ‘mainstream’ CS/ML paper423 abstracts, and of all claims made in these abstracts.424 Within the broad range of activist opinion, there exists a stance which claims that all425 scientific research is systematically fraudulent. Within the broad range of philosophi-426 cal opinion, there exists a stance which claims that all mainstream scientific research427 is systematically mistaken. We feel that this is a problem that can only be managed,428 never be solved.429
(b) Did you describe the limitations of your work? [Yes] See the limitation section and430 other text which identifies and discusses failure modes. Given space limitations, we431 had to be selective about which failure modes to highlight.432
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Discussed433 possibility of creating ‘false sense of safety’ in limitations section, and of policy pro-434 cesses potentially producing the wrong outcomes.435
(d) Have you read the ethics review guidelines and ensured that your paper conforms to436 them? [Yes]437
2. If you are including theoretical results...438
(a) Did you state the full set of assumptions of all theoretical results? [N/A] At its439 core, the paper makes contributions to theory, but it mainly presents new models and440 methodologies, rather than mathematical proofs that follow from assumptions.441 But the paper also claims that the agent definitions included produce certain safety442 properties ’somewhat tautologically by construction’, to use the phrasing of section443 5. So it is possible to read these definitions as proofs of certain theorems (though444 they are theorems stated only in natural language), which would arguably make them445 theoretical results depending on certain assumptions. If so, the assumptions of these446 proofs are stated fully in mathematically well-defined language. In fact, they are stated447 much more formally than the actual safety theorems being proven. It is possible to448 define the safety properties claimed more fully in mathematical language too, as is449 done for example in e.g. [8, 10] with the subject also covered in the 39-page preprint,450 but we have not included any such material in the paper, mainly for space reasons.451
(b) Did you include complete proofs of all theoretical results? [N/A] In the context of the452 observation that certain safety properties are tautologically present by construction,453 we can read the agent constructions themselves as being complete proofs.454
3. If you ran experiments...455
(a) Did you include the code, data, and instructions needed to reproduce the main experi-456 mental results (either in the supplemental material or as a URL)? [N/A]457
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they458 were chosen)? [N/A]459
(c) Did you report error bars (e.g., with respect to the random seed after running experi-460 ments multiple times)? [N/A]461
(d) Did you include the total amount of compute and the type of resources used (e.g., type462 of GPUs, internal cluster, or cloud provider)? [N/A]463
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...464
(a) If your work uses existing assets, did you cite the creators? [N/A] We believe we did465 not use any assets in the sense meant by the checklist. One might argue that modeling466 tools like causal diagrams and CIDs, mentioned at the start of section 2, are assets. In467 any case, we cited their creators.468
(b) Did you mention the license of the assets? [N/A]469
(c) Did you discuss whether and how consent was obtained from people whose data470 you’re using/curating? [N/A]471
(d) Did you discuss whether the data you are using/curating contains personally identifi-472 able information or offensive content? [N/A]473
5. If you used crowdsourcing or conducted research with human subjects...474 (a) Did you include the full text of instructions given to participants and screenshots, if475 applicable? [N/A]476 (b) Did you describe any potential participant risks, with links to Institutional Review477 Board (IRB) approvals, if applicable? [N/A]478 (c) Did you include the estimated hourly wage paid to participants and the total amount479 spent on participant compensation? [N/A]480

Review summaries:
This paper tackles the problem of designing "robust" safety mechanisms against hypothetical AGI agents that may override such safety mechanisms to complete its goal. The paper proposes using counterfactual planning, where an agent plans in a world designed to be different from the real world. The key idea is to plan in a counterfactual world where the safety mechanism does not exist, and use such planned action in the real world. This can then reduce the likelihood of the agent learning a direct correlation between the safety mechanism and goal completion. The paper also argues such planning offers a new vantage point for modeling AGI agents due to its compact and readable graphical language. Two examples are presented to showcase the proposed design approach where construction of counterfactual worlds suppresses some unsafe agent incentives. 

This paper present counterfactual planning as an approach for creating safety mechanisms for agents.  It is framed in the context of AGI.  The topic is relevant to the NeurIPS community and the paper is interesting and well written.  I do believe there are contributions in this work towards building safer AI systems and the way that it proposes to do so.  Other than those concerns articulated in my main review below, I think the work is thorough and well presented.  The technical correctness and detail seems sufficient.  

This paper presents an approach to AI safety based on graphical models. It argues that it is providing new insights to safety problems which leverage these models. The paper exemplifies this by using the case of a stop button, and demonstrates how alterations to the graphical model (stop button, input terminal) in the learning (i.e real) world but not the planning (i.e. agent training/reasoning) allows safety to be introduced and not circumvented during the learning process modelled in the planning world.