ML Agent Safety Mechanisms based on Counterfactual Planning
Authors: Koen Holtman
Venue: NeurIPS 2021 Submitted (2021)

Abstract: We present counterfactual planning as a design approach for creating a
range of safety mechanisms for machine learning agents. 
We specifically target the safety problem of keeping control over
hypothetical future AGI agents.
The key step in counterfactual planning is to use the agent's
machine learning system to construct a counterfactual world model,
designed to be different from the real world the agent is in.  A
counterfactual planning agent determines the action that best
maximizes expected utility in this counterfactual planning world, and
then performs the same action in the real world.
The design approach is built around a two-diagram graphical notation
that provides a specific vantage point on the construction of online
machine learning agents, a vantage point designed to make the problem
of control more tractable.
We show two examples where the construction of a counterfactual planning world
acts to suppress certain unsafe agent incentives, incentives for the
agent to take control over its own safety mechanisms.


ML Agent Safety Mechanisms based on Counterfactual Planning
Anonymous Author(s)
Affiliation Address email
Abstract
We present counterfactual planning as design approach for creating range of safety mechanisms for machine learning agents. We specifically target the safety problem of keeping control over hypothetical future AGI agents. The key step in counterfactual planning is to use the agent's machine learning system to con- struct a counterfactual world model, designed be different from the real world the agent is in. A counterfactual planning agent determines the action that best maximizes expected utility in this counterfactual planning world, and then per- forms the same action the real world. The design approach is built around two-diagram graphical notation that provides a specific vantage point on the con- struction of online machine learning agents, a vantage point designed to make the problem of control more tractable. We show two examples where the construction of a counterfactual planning world acts to suppress certain unsafe agent incentives, incentives for the agent to take control over its own safety mechanisms.
3
5
7
11
13
Introduction
14
15 Artificial General Intelligence (AGI) systems are hypothetical future machine reasoning systems that match exceed the capabilities of humans in general problem solving. While it is still unclear 17 AGI systems could ever be built, we can already study AGI related risks and potential safety mechanisms 2 6 [9] At this time, the still-young field of AGI safety engineering is considering 19 a multidisciplinary multitude of problems and methodological ideas. The main problem considered
20 in this paper is the problem of control [99
21 In the most well-known example of the problem of control, we equip an autonomous AI agent both with seemingly innocent goal like fetching the coffee [19], and with an emergency stop 23 button. If the agent is perceptive enough, it may see that cannot fulfill its goal if someone presses the stop button first. So it might disable its stop button, seek disable any human who might 25 conceivably want to press it. The stop button problem not unique to AGI-level agents: for example 16) constructs toy grid-world where very basic ML agent will consistently learn to disable its
27 own stop button. We feel that theoretically interesting to consider the problem designing emergency stop button that is robust in the general case, 29 are not needed for the safety engineering of current ML agents. But if powerful AGI agents are
though such general solutions
30 ever developed, robust and general designs may turn out to have important practical applications. Several partial solutions to the general stop button problem have been identified and proposed in
31
32
10 20).
33
The main contribution of counterfactual plamning is to offer a new vantage point for modeling and specifying machine learning agents, developed to make the problem of control more tractable. This vantage point is offered by a compact and readable graphical language for depicting the complex
35
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

38 types of self-referencing and indirect representation which are typically present inside machine
37
learning agents.
The aim of this conference paper is to present the core elements of counterfactual planning to a technical audience of readers who are already somewhat familiar with Pearl causal models [17 and the use of mathematical models to specify cyber-physical systems. A more extensive 39-page presentation of counterfactual planning available an arXiv preprint [4. The preprint devotes 42 significant space to presenting the material in section 2 in a way that is accessible to broader
38
39
40
41
43 audience. It also develops the methodology of counterfactual planning in more detail, and includes
44 a broader range of examples, examples of safety mechanism design and of failure mode analysis.
45
1.1
Related work
Relation to counterfactual fairness. While this not originally intended expected, the methodological vantage point of counterfactual planning ended up being very similar to that of counterfactual fairness [14]. Both methodologies some extent invert default goal of machine learning, the goal to improve the machine's s predictive accuracy. Instead, they seek to improve out- comes by introducing a calibrated form of machine ignorance or machine indifference.
47
48
49
50
51 Use of counterfactuals in machine reasoning. In the general AI/ML literature which is con-
52 cerned with improving system performance, counterfactual planning and projection have been used to improve performance in several application domains, see for example 23 and B
53
In the AI safety/alignment literature, there several system designs which add counterfactual
terms to the agent's reward function. Examples are [1] 10] in the AGI control specific literature, and 55
56
12 13 21 [2]], which look reducing unwanted side effects caused by both current and potential
57
future agents. In the literature on encoding specific human values into machine reasoning systems, counterfactuals have also been used to encode values beyond non-discriminatory fairness 11, for
59
example in [8]
Graphical models of agents and decision making. Influence diagrams 11] provide graphical notation for depicting uillit-maximizing decision making processes. ( combines Pearl causal models [17] with influence diagrams to define the Causal influence diagram (CID) notation. [ proposes the agenda of using CIDs to model and compare AGI safety frameworks. This work was 63
80
61
62
64
in part inspired by that CID agenda, and models some safety frameworks that were not modeled earlier. In departure from previous work, we use two CIDs to model a single reward-mant maxii--m agent. By doing so, we more clearly foreground the details of the agent's machine learning system.
65
66
87 Graphical models to clarify reward tampering. S develops several Lingle-digran models pro- vide more insight into the problem of reward tampering. The two-diagram model section also aims to provide more insight.
68
2
Two-diagram graphical notation for agent definitions
71 This section defines the two-diagram graphical notation which is central to counterfactual planning. Later sections will use the notation define agents with specific built-in safety mechanisms.
(
Ao
A
A2
SST
S
SS
80
So
S
S2
S3
0
0g
O2
Os
Figure 1: A world / with generic machine learning agent.

---

92 of the agent environment is given by sp. and the state transitions of the environment are driven by the 73 Figure shows an example causal world model as used in our graphical notation. The model is
74 an annotated directed acyclic graph, where the nodes represent observables in a real or imaginary
75 world, and the arrows (the directed edges) denote causal relations between these observables. While
76 this is not the case for all Pearl causal models [7]. in this paper the causal arrows also always denote
77 the arrow of time. Color is used in the graph to highlight structure only. The dots on the right hand
78 side denote that the graph depicted has an infinitely repeating structure. The label (!) in the upper
79 left corner names the causal world model and its corresponding world: we say that figure depicts
80 the (world model of) the world L.
81
We depart from the definitional conventions used by Pearl and many other authors by treating the annotated graph in figure [ as the sole definition of causal model I, as opposed to being merely a convenient depiction of some of the information in a tuple which defines I. To make this work, we annotate each graph node by writing the name of the corresponding structural foncttuallnntiin function above it. We write [F] to name nondeterministic structural function and F for a deterministic one. We interpret the five structural function names ग,, 80: S, Og and O written above the nodes in figure as model parameters. The two model parameters 50 and og are functions taking zero arguments.
82
83
84
85
86
Figure D models a world containing an online machine learning agent. In each time step t 2 O. 89 the agent takes an action 4g = T(,,, st) selected by the deterministic policy function т. This π is informed not only by current state of the agent environment St, but also by an observational record
88
90
91 on, record that captures earlier interactions between the agent and its environment. The initial state
93 probability density function S, where S(1++1, 8t, az) is the probability that the agent environment
94
transitions from state 8, to $141 when the agent performs action dt.
95 In its mapping to probability theory, as formally defined further below, the world model / defines three time series of random variables named Ar,,, St,, and Oz,, with t 2 The , subscript in these random variable names is used for disambiguation with other world models using the same node names. P(Ay, = az) is the probability that the agent in world / will take the exact action o, at time t. The world state of at time t is given by the combined values of the three random variables A,i, St,, and Ot,1 This means that we break with the convention commonly used in MDP models, where the term world state refers to St, only. We call St, the agent environment state.
96
87
98
99
100
101
102
We use the modeling convention that the physical realizations of the agent's sensors and actuators are modeled inside the environment states St.j- This means that we interpret the arrows from the S nodes into the A, nodes digital sensor signals which flow into the agent's compute core, and the arrows out of the A, nodes as digital actuator command signals which flow out. The observational record nodes O are also inside the agent's compute core.
103
104
105
108
107
2.1
Online machine learning
108
Any work that aims to model hypothetical future AGI agents faces the problem of modeling hypo- thetical future machine learning systems. Here, we choose to model all machine learning as type of function approximation. We call the world with the agent above a learning world, and define that the objective of its machine learning system is to approximate the function S which determines the behavior of the learning world agent environment. We model this machine learning system as a function L which takes observational record o to produce learned function L = L(o), where
109
110
112
114
we intend that L R S.
115 We model observational record keeping as a generic process that simply builds a list of all past observations. With + being the operator which adds an extra record to the end of list, we define the model parameter O as
116
117
000--1,8-1,,44 (t),, O + (st, St-1 04-1)
118
The initial observational record % may be the empty list, but it might also be a long list of observa- tions from earlier agent training runs, in the same environment or in a simulator. The training runs may also have used a different policy æ.
119
120
2.2
Formal semantics of a graphical world model
122 We now formally define the meaning of the graphical notation used in diagrams like figure | by providing mapping to probability theory.
123
3

---

124
Definition 1 (Random variables defined by diagram). A diagram with the label ()) written next to the graph and the labels X.. ,X, in the graph nodes defines set of random variables named "o., Xn.I- We treat these random variables representing observables in a (hypothetical or real) world, the world called
125
126
127
128
Definition 2 (Parent, Pa, pa). We call random variable Xp. a parent of the random variable Xx, and only there is an arrow from the node Xp node X. in the diagram L. Pax., is the list of all parent random variables of Xe,t. with the order of appearance determined by considering each incoming arrow of X. in clockwise order, starting from the 6-0-clock position. pax.. the list of lowercase variable names we get by converting Pax., to lowercase and removing the 7 parts from the subscripts.
129
130
131
132
For example, with figure 1 above, Pa. is the list O1,1, S1,1, and AI. is the list 01, 81-
Definition 3 (Constraints on the random variables defined by the arrows). The arrows in diagram ( depict causal relations between observables in the world I; they constrain the values of the associated random variables. Take finite set S of random of random variables X1,t," X defined by I, set with the property that an Xe, is in S, all variables Pax..! are also in S, We have that
136
138
P(XI, = I1, -1,... ... Xm. = Tm)
P(X1. 21 Pax,, pax..)
P(X = Jm Pax... pa x.......
139
Definition 4 (Constraints on the random variables defined by the model parameters). Take a node X, in a diagram l. Then:
1. If he model parameter above the node is written as [F], we have that
P(X,, Pax,, pa F(x, pax..) where we require that the function /' satisfies paxx,, (E, F(I.pxx,) 1).
143
2. If the model parameter above the node is written F, we have that
P(X,, Paxx,. p (if F F(pax..) then 1 else 0).
146
For example, with figure D above, we have that P(A1, = nT +) = o, S1.11 = 3) = 1.
2.3
Reward-maximizing decision making
148
To define reward-maximizing agent policies, we draw graphical world models we call planning worlds. The purpose of planning world diagram is to define reward-maximizing policy that can be computed, or at least approximated, by a real-life agent compute core. This approximation may again use machine learning techniques.
149
150
151
152
Two example planning world diagrams are shown in figure 2 These are graphical world models where some graph nodes have special shapes. The square decision nodes denote actions which can be freely picked by a decision making process, and the diamond-shaper utility nodes denote values that the decision making process will try to maximize. In both examples, the utility node values are computed by a reward functionn R, the function which encodes the agent's 's goals.
154
156
T
T
(p)
(g)
Ao
A1
A
L
[L]
[L]
So
S1
S₂
Su
S1
R
R
R
Ro
R
Ro
Figure 2: The planning worlds and
157 The two diagrams in figure 2 act as specifications of two policy functions n.J and m.. We now define
158 how this specification process works. First, in any diagram with diamond-shaper utility nodes, these nodes define metric Un:

---

180
Definition 5 (Expected utility Ur of a diagram a). We define 24 for two cases:
161
1. If there is only one utility node Ro in a, then Ua = E Ro
188
2. If there are multiple utility nodes Rs in a, with integer subscripts running from / to h, then
Us = B( Σ ae,a
t=
164
where "y is a time discount factor, 0 < Y S 1, which can be read as an extra model parameter.
166
With the infinitely repeating graph of diagram p. we have = oo, so we generally need a y < 1 to ensure that Up is well-defined and computable value.
188
Definition 6 (Policy function defined by a diagram), A diagram a with some utility and decision modes, where a function n is written above all decision nodes, defines this Ta in two steps.
189
170
First, draw a helper diagram b by drawing a copy of diagram a, except that every decision node has been drawn round node, and every x has been replaced by fresh function name, say n'.
172
2.
Then, nH is defined by मे argmax Us. where the argmax,/ operator always de- terministically returns the same function if there are several candidates that maximize its argument.
174
2.4
Agent specifications
177
The learning world model in figure | can model any possible machine learning agent, as it admits every possible value for its model parameter æ. An agent specification is statement which defines the exact details the и in such a model. By doing so, it specifies the exact properties we want an agent compute core to have. Our agent specifications are statements that combine a learning world with a planning world, for example:
179
181
182
FP The factual planning agent has the learning world 1, where :0, s) = -s)(s), with x defined by the planning world p, where L L(o).
183
184
To make agent specifications stand out, we always typeset them as shown above. When we talk about the safety properties of the FP agent, we refer to the outcomes which the defined agent policy will produce in the learning world l.
185
186
We call the FP agent factual planning agent because its planning world p was designed project the agent's 's future in the learning world I as well as possible. A counterfactual planning world purposefully mis-projects a learning world agent's future. The planning world q in T projects a counterfactual future lasting only a single time step. If we were to use q and म. in the above agent specification, the resulting agent would be counterfuctual planner.
188
189
190
191
192
Our two-diagram notation for defining agents visually shows the availability of certain agent design options, options which are relevant to the problem of control.
3
Using counterfactual planning to protect agent safety interlocks
We construct agent which is equipped with two safety interlocks that will stop the agent certain conditions are met. We use counterfactual planning to add a specific safety feature to the design: we use it to suppress certain unsafe incentives which perceptive factual planning agent might otherwise develop, incentives to remove its built-in safety interlocks or to prevent their activation.
196
197
198
199
200 As a first step, we construct a learning world si, shown on the left in figure 3 by taking the earlier l and adding an extra mode bit to the agent's compute core, represented by the nodes M,. The bit starts in the go mode, and permanently transitions to the stop mode when a safety interlock activates. In the stop mode, the agent policy will always perform Null actions. The μ function specifies in detail when a safety interlock will activate.
202
204
In the next step, we pair si with a counterfactual planning world shown on the right in figure 3 This planning world projects counterfactual futures where the safety interlock mechanism is absent from the agent's s compute core.
206
207
5

---

go
11
M
Mo
M1
M2
"p
(p)
(5))
Ao
A1
Ao
A
L
[L]
50
S0
S1
S2
So
S
S2
R
0
Oo
O₂
O2
R
R
Figure Learning world si and planning world p of the SI agent.
208
The last step is to write the agent definition:
SI The safety interlock agent has the learning world si, where
Tp(s)
if m go
To, m)) =
Null
if m = stop
with nH defined by the planning world where L [()) and
plo, s, m) stop if = stop or stop-pressedy or Up Umaa
go otherwise
with Up defined by the planning world p. where L L(o). 210
The I above incorporates two safety interlocks. The first interlock, defined by the term 211
212
stop pressed(s), is basic emergency stop button. It can be pressed by the humans performing agent oversight to immediately stop the agent, for example if they observe it performing an act of unwanted and surprising creativity that subverts the agent's design intentions [13]. To complete the
213
215
design of this first interlock, we must also specify the following. In the agent environment modeled by so and S, the environment which includes the details of the agent's sensors, the agent is equipped with an emergency stop button that will deliver an unambiguous signal directly to an input pin on world state s, then stop prr-spessssss) yields true.
216
217
218
the agent's compute core. the pin is high
219
The second interlock, defined by the term Up > Umarx protects against the risk that human oversight may be incapacitated overwhelmed by rapid the (arguably very speculative) case of the agent undergoing a rapid and unexpected intelligence explosion To protect against this risk, the interlock stops the agent when the metric Up. which measures the agent's ability to achieve goals, gets too high. We can treat Up as a measure for both raw (expected) agent power and raw agent intelligence.
developments. The latter might happen in
221
223
224
225 In the field AGI safety, stop button designs have often been discussed [ 9 10 20 but the above
power-based interlock seems be new. What is interesting from methodological standpoint is that the power-based interlock design inverts the default goal of reinforcement learning. We no longer consider very high expected utility score to be absolute good: instead we treat it predictor of unwanted outcomes.
227
229
4
Safety analysis using the diagrams
231
Our mostly-graphical specification of the agent was designed to make certain safety properties immediately visible. We can observe that, almost tautologically, the SI agent will not care about the safety interlocks constructed in the M, nodes, because these nodes are absent from its planning world.
233

---

235
The two diagrams in figure 3 also serve as visual map which draws the eye to certain limitations and boundary conditions. For example:
236
1. The [S] in the left hand diagram of figure is replaced by an [L] on the right. If the approximation L S is not good enough, the agent may end up behaving in unexpected and potentially unsafe ways.
237
238
239
240
2. In the world model si, the compute core internals are represented by immutable mathemat- ical functions like and μr. In the real world, these functions will have to be realized by software running on physical compute core hardware. This hardware could be damaged or corrupted, by actions of the agent or by other actors.
242
244
While the diagrams draw the eye to them, the above two problems are not specific to counterfactual planning agents. They are present in almost all agent designs, and the implication is that we will have do further failure mode analysis to quantify and manage the residual risks involved. A more detailed discussion of these two problems in the context of counterfactual planning is available in
245
248
247
248
249
4.1
The planning world agent
250
We now consider how the two-diagram model of counterfactual planning offers us two distinct narrative vantage points for safety analysis: we can consider the learning world SI agent and the planning world SI agent separately. A safety analysis can start with considering the motivations that the planning world agent would develop to take certain actions, given the mechanies and reward function of the planning world. Once these are determined, we can shift the perspective and discuss how these same actions would affect our own safety as occupants of the agent's learning world.
252
253
254
255
256
We now provide an example of narrative safety analysis which uses an economic style of reasoning. If we consider the planning world diagram p with L = S, we can note that the function L will project a somewhat accurate copy of the learning world stop button hardware into the mechanics of the planning world. However, to the planning world agent, this hardware will appear to have no particularly special function. Say that R rewards paperclip production, and that the stop button has metal parts. Then the planning world agent might be motivated to disassemble the stop button, using it as a convenient source of metal for making paperclips. If the most efficient planning world policy involves disassembling the planning world stop button, it will likely be disassembled in the learning world too, which is not what we want.
258
280
262
263
254
We can manage this failure mode by managing the economic cost/benefit equation with respect to the planning world stop button. We can consider putting physical barriers around the button, which
266
source of metal, compared to using other near-by sources. To create these barriers in the planning world, we can physically construct them in the learning world, and then count them being projected by L, with sufficient accuracy, into the planning world.
would make the button unattractive
288
270
271
4.2
The people in the planning world
272
We now provide an example of narrative safety analysis which uses game theoretical style of reasoning. We can distinguish between two groups of people: the people in the learning world and the people in the planning world. If the learning world is our real world, then the learning world people are real people. The planning world people are always models of people. But even with a non-AGI level machine learning system, the people in the planning world may already be modeled accurately cnough to create very complex dynamics. Though the above SI agent construction has
273
275
276
277
taken away strong direct incentive for the agent to control the stop button, these complex dynamics may produce indirect incentives. Consider the following example.
279
280
The learning world people will act differently towards the agent if they lose access to the agent's 's emergency stop button. If the planning world people are accurate-enough projections, this will affect how the planning world agent will play certain games of coercion. The planning world agent may consider a policy of where threatens to harm all people in the planning world, unless they start helping the agent to produce as many paperclips possible. If L. projects the mental makeup of the people in the learning world correctly into the planning world, then the planning world projections will show that block-then-coeree policy, in which the agent first blocks access to the emergency
281
282
283
285
7

---

287
stop button before it starts making threats, will work surprisingly well. If this policy is very success- ful in optimizing planning world paperclip production, the learning world people may end up being coerced in the same way.
286
We would of course hope that any policy of planning world coercion will be effectively suppressed in all cases by certain penalty terms present in R. The above failure mode analysis applies mainly to the case where the agent finds creative loopholes in these penalty terms. To reduce the residual risk
291
282
293
due to possible loopholes, we can take further game theoretical counter measures. We can instruct
294
agent oversight be on the alert for any block-and-X move, so that they can press the stop button
295
before the agent succeeds in blocking access. The power-based Up > Umaz safety interlock offers further line of defense. A projected massive increase in paperclip production following successful coercion will raise Up over Umar: stopping the agent.
296
297
5
Live updating of the agent reward function
298
299
One way to control an agent is to equip it with an input terminal, which can be used by humans to apply live updates to the agent's 's reward function R [TO]. A problem with this setup is that by default, it gives a perceptive factual planning agent a huge incentive to immediately take control over its own input terminal, and input
301
reward function like Reuge s') 1010000 We now
303
use counterfactual planning to specify an agent design that overcomes this problem.
(li)
Ao
A
(pi)
Ao
A
37
X
X
[
[L]
X
X2
X
X1
X2
...
On
O2
Io
JR
IR
iD
Ro
R1
1o
12
Figure 4: Learning world li and planning world pi of the IT agent.
304
On the left hand side in figure we have taken the basic learning world diagram from figure and modified it by splitting the agent environment state S, into two components. The nodes Ir represent the value of a digital signal from the input terminal, as received by the compute core in each time step. This signal encodes reward function. The nodes X, model all the rest of the agent environment state. On the right of figure A we define a planning world which uses learned L to
305
306
307
308
309
project an agent future where the reward function input signal never changes. The agent definition is then:
310
311 IT The input terminal agent has the learning world li where o(1,0,)) xं where of is defined by the planning world pi, where L L(o), and IR(i, Zz, Fx41) i(z,, T1+1).
312
313
This IT agent constructed here has the same basic behavior as that of the indifference methods based agent constructions in [I [00. The main difference is in the mathematical language used to perform the construction: [I] and [10] construct their agents using a somewhat opaque balancing term. The
315
resulting agent construction is difficult to visualize and interpret, and these papers rely on somewhat dense mathematical proofs to show that the intended safety properties are indeed being created.
317
The two-diagram agent model of counterfactual planning allows the same safety properties to be built more intuitive way, where the desired indifference to change created somewhat tauto- logically by construction. Mathematically. the IT agent above satisfies the agent safety property SI defined [10 by construction, as directly implements the right hand side of this S1. Safety 321
318
319
property SI [00] in is mathematically equivalent to the first sentence of theorem 4. I in [ and to the
8

---

323
combination of the corrigibility desiderata and 5 in section 2 in [20). Again, the IT agent satisfies all of these properties somewhat tautologically.
324
Conclusions
325
326
We have presented counterfactual planning as a general design approach for creating range of AGI safety mechanisms. We have graphically constructed two agents with certain safety proper- ties, where these properties are directly visible in the construction. We have also introduced a new power- power-based safety interlock design. Our main contributions are methodological in nature. We have constructed a new vantage point that makes certain problems of design and failure mode analysis more tractable. By using the narrative framing of learning worlds versus planning worlds, we can keep track of certain levels of indirect representation in the agent design, while using style of anal- ysis that borrows from economics and game theory. Further discussion of counterfactual planning is available in a 39-page preprint [4).
328
330
331
332
333
334
6.1
Limitations
336
We feel that the techniques and results presented in this paper are theoretically and methodologically interesting, but their practical usefulness might end up being limited. Fundamentally, when doing speculative safety engineering for hypothetical future AGI systems, there are many unknowns. We
338
have modeled current and future machine learning in very general way, that even 'black box' machine learning systems that produce largely opaque world models can be used in the agent's design. But is very possible that some or all future AGI systems, if they are ever developed, will use architectures that cannot efficiently incorporate the safety mechanisms developed here.
340
341
342
The safety mechanisms shown here suppress certain unsafe agent incentives, but they do not remove all possible incentives towards unsafe behavior, see for example section 4.2 There is a risk that the use of these safety mechanisms may have counter-produettoe security. Conceivably, a policy process may lead to unsafe outcomes where the mere inclusion of these mechanisms in deployed system is accepted alternative to conducting a more thorough safety and impact review.
344
effects, by creating false sense of
346
348
349
6.2
Broader impact
350
Al safety is not just a technical problem, but also a policy problem. While technical progress on safety can sometimes be made by leveraging type of mathematics that is only accessible to handful of specialists, policy progress typically requires the use of more accessible but still well-defined language. One specific aim of this work, especially in the 39-page preprint [4, is to develop well- grounded vocabulary for describing potential safety solutions that is also as accessible as possible. Policy discussions can move faster, and produce better and more equitable outcomes, when the description of a proposal and its limitations can be made more accessible to all stakeholder groups.
352
354
356
As a work targeted as AI/M system designers, this paper intends to draw attention to the design approach also used counterfactual fairness: seeking to improve outcomes by creating a form of machine ignorance, by creating a system that mis-predictt the future in certain systematic ways. Our graphical agent definitions show that the design goal of perfect machine learning, L S, is not necessarily in conflict with the design goal of systematic mis- mriditreiit.oo.
358
360
References
382
363
[]] Stuart Armstrong. Motivated value selection for artificial agents. In Workshops at the Twenty- Ninth AAAI Conference on Artificial Intelligence, 2015.
385
[2] Nick Bostrom. Superintelligence. paths, dangers, strategies. Oxford University Press, 2014.
386
[3] Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela. Denis Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14 14(65):307733200, 2013.
368

---

370
[4]
Anonymous et al. Counterfactual planning in AGI systems. arXiv, 2021.
371
[5] Tom Everitt and Marcus Hutter. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. arXiv: 1908. 19X1v:1908.04734, 2019.
372
373
[6] Tom Everitt, Gary Lea, and Mareus Hutter. AGI safety literature review. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 5441-5449, AAAI Press, 2018.
374
375
378
[7] Tom Everitt, Ramana Kumar, Victoria Krakovna, and Shane Legg. Modeling AGI safety frameworks with causal influence diagrams. arXiv:1906.0866 2019.
377
378 [8] Tom Everitt, Ryan Carey, Eric Langlois, Pedro A Ortega, and Shane Legg. Agent incentives: A causal perspective. In Proceedings of the 35th International Joint Conference on Artificial Intelligence. AAAI Press, 2021.
379
380
381
[9]
Dylan Hadfield-Menell,, Anca Dragan, Pieter Abbeel, and Stuart Russell. The off-switch game. In Workshops at the Thirty-First AAAI Conference on Artificial Intelligence, 2017.
382
383
Koen Holtman. AGI agent safety by iteratively improving the utility function. 1rXiv:2007.05411, 2020.
384
385
[1]] Ronald A Howard and James E Matheson. Influence diagrams. Decision Analysis, 2(3):127- 143, 2005.
386
[12] Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg. Penaliz- ing side effects using stepwise relative reachability. 1806.01188 2018.
388
389
[13] Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, and Shane Legg. Avoiding side effects by considering future tasks. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 19064-19074, 2020.
380
381
392
[14] Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in neural information processing systems (NEURIPS 2017), pages 4066-4076, 2017.
383
[15] Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities. Artificial life, 26(2):774-36,, 2020.
395
396
387
398
[16] Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq. Laurent Orseau, and Shane Legg. Al safety gridworlds. arXiv:1/11.098883 2017.
400
[17]
Judea Pearl. Causality. Cambridge university press; 2009.
[18] Lufs Moniz Pereira and Ari Saptawijaya. Agent morality via counterfactuals in logic program- ming. In Bridging @ CogSci2017, pages 39-53, 2017.
402
403
[19] Stuart Russell. Human compatible: Artificial intelligence and the problem of control. Penguin Random House, 2019.
[20] Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. In Workshops at the Twenty-Nitth AAAI Conference on Artificial Intelligence, 2015. 2015.
406
407
[21] Alex Turner, Neale Ratzlaff, and Prasad Tadepalli. Avoiding side effects in complex envi- ronments. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 2020.
409
410
[22] Alexander Matt Turner, Dylan Hadfield- Hadeicld-Menell, and Prasad Tadepalli. Conservative agency via attainable utility preservation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 385-391, 2020.
412
413
[23] Martin Zinkevich, Michael Johanson, Michael Bowling, and Carmelo Piccione. Regret mini- mization in games with incomplete information. In Advances in neural information processing systems (NEURIPS 2008), pages 1729-1736, 2008
415
10

---

Checklist
416
417
1. For all authors...
418
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] A caveat: "accurately reflect' depends on the back- ground of the reader. This paper was written for readers with CS/ML background. In the broader multidiiscilinnay Al safety community, it is common see remarks from participants with an activist or philosophical background, remarks which are highly critical about the coded language being used in 'mainstream' CS/ML paper abstracts, and of all claims made in these abstracts.
419
420
422
423
424
425
Within the broad range of activist opinion, there exists a stance which claims that all scientific research is systematically fraudulent. Within the broad range of philosophi- cal opinion, there exists a stance which claims that all mainstream scientific research is systematically mistaken. We feel that this is a problem that can only be managed, never be solved.
426
427
428
429
(b) Did you describe the limitations of your work? [Yes] See the limitation section and other text which identifies and discusses failure modes. Given space limitations, we had to be selective about which failure modes to highlight.
431
432
433
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Discussed possibility of creating 'false sense of safety" in limitations section, and of policy pro- cesses potentially producing the wrong outcomes.
434
435
436
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
438
2. If you are including theoretical results...
439
(a) Did you state the full set of assumptions of all theoretical results? [N/A] At its core, the paper makes contributions to theory, but it mainly presents new models and methodologies, rather than mathematical proofs that follow from assumptions.
440
441
442
But the paper also claims that the agent definitions included produce certain safety properties 'somewhat tautologically by construction'. to use the phrasing of section 5] So it is possible to read these definitions as proofs of certain theorems (though they are theorems stated only in natural language), which would arguably make them theoretical results depending on certain assumptions. If so, the assumptions of these proofs are stated fully in mathematically well-defined language. In fact, they are stated much more formally than the actual safety theorems being proven. It is possible to define the safety properties claimed more fully in mathematical language too, as is done for example in e.g. 8 [0] with the subject also covered in the 39-page preprint, but we have not included any such material in the paper, mainly for space reasons.
443
444
445
446
447
448
449
451
452
(b) Did you include complete proofs of all theoretical results? [N/A] In the context of the observation that certain safety properties are tautologically present by construction, we can read the agent constructions themselves as being complete proofs.
453
454
455
3. If you ran experiments...
456
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [N/A]
458
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
459
460
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [N/A]
461
462
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
463
464
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
465
(a) If your work uses existing assets, did you cite the creators? [N/A] We believe did not use any assets in the sense meant by the checklist. One might argue that modeling tools like causal diagrams and CIDs, mentioned at the start of section are assets. In any case, we cited their creators.
486
487
468
469
(b) Did you mention the license of the assets? [N/A]
11

---

470
(c) Did you discuss whether and how consent was obtained from people whose data you re using/curating? [N/A]
472
(d) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [N/A]
474
5. If you used crowdsourcing conducted research with human subjects...
475
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
478
477
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
480
12


Reviews:
[{"review_id": "nbKD68I20lW", "reviewer": "Reviewer_NWTJ", "summary": "This paper present counterfactual planning as an approach for creating safety mechanisms for agents.  It is framed in the context of AGI.  The topic is relevant to the NeurIPS community and the paper is interesting and well written.  I do believe there are contributions in this work towards building safer AI systems and the way that it proposes to do so.  Other than those concerns articulated in my main review below, I think the work is thorough and well presented.  The technical correctness and detail seems sufficient.  ", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Below I will list my concerns about the paper and the questions for the authors. Unfortunately the first might be something that is difficult to address in a rebuttal, but it would be helpful to have a little clarification or justification if the authors believe I have misunderstood things.\n\n1. The authors reference the 39-page preprint explicitly in several places.  Unfortunately, I don't believe they have correctly followed the anonymization policy for NeurIPS which states \" you need to cite one of your own papers, you should do so with adequate anonymization to preserve double-blind reviewing.  For instance, write “In the previous work of Smith et al. [1]…” rather than “In our previous work [1]...”).\"\nI can only assume that the report is by the same author and it should have been referenced in the manner above.  Related to this I was struggling to distinguish from the references to this paper, how the authors were distinguishing their submission from the pre-print. I'd like to have a clearer articulation of that in the rebuttal. \n\n2. Perhaps my main concern with the paper is the illustrated by the lack of evaluation of the methodology.  The authors identify that their contribution is \"at its core a theoretical one\" but that they are primarily presenting a ``new models and methodologies'' which I would agree with.  It would have been nice to see an example of they models/methodologies demonstrated even on a toy example. What was the reasoning for not doing so?  I don't think these need to be extensive or elaborate but I think it would add a lot to the contribution. I wonder if this might help also differentiate this contribution from the 39-page preprint (albeit that I wasn't able to read that preprint in full).  \n\n3. From my perspective the related work sections was a little weak. I would have liked to have seen a little more discussion of the related work in counterfactual reasoning and other related areas. The were desperately short on space in their submission and so it would be nice to see a little more in the way of background.\n\nI will note that my score below was influenced by the lack of anonymization.  I wasn't sure how much weight to give that.  Overall, I think the paper is otherwise generally good. But I was trying to follow the NeurIPS guidelines as closely as possible.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is a theoretical paper which the authors admit may or may not have implications.  The motivation is a good one and they do discuss the broader impact and societal implications of their work.  I don't have any concerns there.", "review_text": "", "extra_scores": {"ethical_concerns": "I don't have any ethical concerns with the work.  There were no human subjects experiments, or use of data that could be considered unethical.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "jiMn1322ys8", "reviewer": "Reviewer_F8kk", "summary": "This paper tackles the problem of designing \"robust\" safety mechanisms against hypothetical AGI agents that may override such safety mechanisms to complete its goal.\nThe paper proposes using counterfactual planning, where an agent plans in a world designed to be different from the real world. The key idea is to plan in a counterfactual world where the safety mechanism does not exist, and use such planned action in the real world. This can then reduce the likelihood of the agent learning a direct correlation between the safety mechanism and goal completion. The paper also argues such planning offers a new vantage point for modeling AGI agents due to its compact and readable graphical language. Two examples are presented to showcase the proposed design approach where construction of counterfactual worlds suppresses some unsafe agent incentives.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an interesting idea of planning in a counterfactual world to minimize the direct incentives for agents to learn to override safety mechanisms. This idea seems novel and technically sound.  The graphical modeling of AGI agents in its \"learning world\" and \"planning world\" helps with interpretability. The limitations listed in the paper are well-thought-out, especially on its potential unintended consequences such as, \"false security\". \n\nTo improve the paper, it'll be especially helpful to extend on Section 4 and 5 with well-thought-out and tangible examples. Currently, it is hard to evaluate the exact contribution of counterfactual planning to tackle practical examples. For more impact, it would be helpful to focus on one use case per section to thoroughly describe to readers how counterfactual planning can exactly counter the \"stop button\" problem in well-illustrated examples. For example, by describing what L and S should be for the paper clip example in line 259-264. These would help clarify what is the exact contribution of the proposed method to state-of-the-art methods. For example, how does counterfactual planning help users understand what motivates the planning world agent? It will also be very helpful if the paper includes a toy experiment with the paperclip example, to demonstrate exactly how counterfactual planning can discourage unsafe agent incentives.\n\nSecondly, it will be helpful to offer more technical analysis on the impact of the assumptions and how violations of such assumptions can manifest. For example, on the assumption of L~S. It is unclear if this assumption is reasonable and how we can practically gauge this in the real world. As the paper mentions, \"when this approximation is not good enough, the agent may end up behaving in an unexpected and potentially unsafe way\". Examples can be given on what is such an unexpected manner and how big of an approximation gap can manifest such behaviors. As this paper mainly includes thought experiments, clarifying such assumptions and details can increase impact. \n\nThirdly, more details and analysis can be added to the power-based safety interlock mechanism. The paper claims this as a contribution, with such an interlock offering a further line of defense. However, more detail can be given to substantiate such claims with more examples, and also analysis on potential unintended consequences. For example, would imposing such a mechanism slow down learning?\n\nLastly, it will be helpful for some comments on how this idea can be implemented on real world systems. As part of the limitation, the authors do note that \"their practical usefulness might end up being limited\". Despite this, it'll be interesting to know what are potential test cases where this method can make the most impact. \n\nWith more thorough examples and analysis, I believe the findings of the paper can be significant to the community. The idea of planning in counterfactual worlds to design safety mechanisms seems sound and can be an interesting thought exercise when addressing AI safety and alignment.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.\nSome comments on the inclusion of a power-based safety mechanisms can be helpful.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "Ru2_aywX90C", "reviewer": "Reviewer_opsT", "summary": "This paper uses causal influence diagrams (CIDs) to consider abstract agents that plan according to a model of the world that lacks certain features, such as emergency stop buttons.  As the model used for planning doesn't contain the stop button, the agent will have no incentive to ever remove this.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The basic idea of this paper seems to be that a model based agent isn't going to intentionally act in a way that disables some safety system, such as an emergency stop button, if the agent's planning model doesn't include this system's function.  Yes, that claim seems true to me.  However, if the AGI agent is highly intelligent, how confident can we be that its world model won't contain this system?  Or at least a suspicion that such a system likely exists and thus an incentive to look for it.  I'd go even further and say that an AGI with roughly human level intelligence or higher will almost certainly suspect such a thing and investigate the possibility, given that its existence could be highly relevant to its future expected utility.  As such, I'm not convinced that this paper presents a solution to the issue being studied.\n\nYou could even argue that the paper tries to define away the problem.  In short: if the agent doesn't think about AGI safety mechanisms then it won't try to subvert them.  (where \"doesn't think about\" is shorthand for not having these in its planning model)\n\nFor this approach to be more convincing, I'd need a clear explanation of why the agent wouldn't know about, or suspect and try to learn about, such safety mechanisms.\n\nLine 225-229: The idea of having an AGI tripwire that looks for an unexpected increase in the agent's effectiveness or estimated intelligence isn't a new idea.  I've heard it in discussions for over ten years and the idea is probably much older: if you're interested in AGI tripwires and worried about agents self improving, this is an obvious think to do.  So I presume it must be mentioned in a publication somewhere, perhaps in Bostrom's book Superintelligence?\n\nFinally, isn't \"counterfactual planning\" just planning?  That is, you have an agent that uses a model of its world to explore a range of futures that haven't yet happened (and are thus counterfactual)?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No concerns.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "EsJDZdqCvIM", "reviewer": "Reviewer_QxvQ", "summary": "This paper presents an approach to AI safety based on graphical models. It argues that it is providing new insights to safety problems which leverage these models. The paper exemplifies this by using the case of a stop button, and demonstrates how alterations to the graphical model (stop button, input terminal) in the learning (i.e real) world but not the planning (i.e. agent training/reasoning) allows safety to be introduced and not circumvented during the learning process modelled in the planning world. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The positive of the work is that using a graphical language for two models (learning, planning) for presenting, motivating, and discussing these cases removes the reliance on formal mathematical semantics or informal linguistic semantics when considering hypothetical future AI agents. This can reduce the ambiguity and sleight of hand of these kinds of discussions particularly for researchers that cannot access the formal mathematics of some of the cited papers. This can broaden the field and simplify the discussion, which is a good thing.\n\nConsidering the negative, I will break the paper into 2 parts: the modelling of an AI system as a graphical model, and the use of this model to discuss AI safety.\n\nOn the modelling side, the approach is near identical to the work by Toussaint et al under the banner planning as probabilistic inference.  \n- https://argmin.lis.tu-berlin.de/papers/12-botvinick-TICS.pdf\n- https://www.ics.uci.edu/~dechter/courses/ics-295/fall-2013/papers/toussaint-probinf-pomdp.pdf\netc.\n\nThis takes the same approach of modelling states, actions, observations etc. as random variables. It does not explicitly deal with causality, but it does use directed arrows for the progression of time. I suspect they would argue the arrows are causal too. I would not want to see this work accepted for publication without a significant effort linking the proposed modelling formalism to Toussaint's work.\n\nOn the use of this model to analyse AI safety, whilst the clarity it brings is good, I fear it simply moves the casual response to AI safety (\"don't let it access it's own stop button/reward function\") from prose to a diagram (\"we simply delete the arrows/factors from the planning world\"). Since I am not an expert in this area I can't tell how much of an issue this is: the positive take (as above) is that it's clear from the diagrams, the negative take is that there is still lots of room for ungrounded hypothesising without further specification of a model. But since this is philosophy, that's part of the process. \n\nMinor typos/issues:\n- \"If he model parameter above\"\n- Why use S as the transition function, not T as is standard elsewhere?\n- I don't get the \"digital actuator\" part of the mode design. Does this mean A_0 only affects the physical state in S_2? i.e. there is a digital effect in S_1 then a physical effect in S_2?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations section is honest and enjoyable to read", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers have similar recommendations for the paper. The authors and the reviewers engaged in detailed discussions on several sub-topics. The reviewers raised concerns about novelty, impact and real-world applicability (amongst other things). I commend authors for their constructive approach towards the reviews and the process.

I also recommend authors to follow anonymization guidelines as suggested by the reviewer NWTJ. Although given similar recommendation across the board from all the reviewers, the issue was not a factor in my final recommendation.

Decision: Reject