Optimal Policies Tend To Seek Power
Authors: Alexander Matt Turner
Logan Riggs Smith
Rohin Shah
Andrew Critch
Prasad Tadepalli
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: ai alignment
mdp theory
reinforcement learning
optimal policies

Abstract: Some researchers speculate that intelligent reinforcement learning (RL) agents would be incentivized to seek resources and power in pursuit of the objectives we specify for them. Other researchers point out that RL agents need not have human-like power-seeking instincts. To clarify this discussion, we develop the first formal theory of the statistical tendencies of optimal policies. In the context of Markov decision processes, we prove that certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment. These symmetries exist in many environments in which the agent can be shut down or destroyed. We prove that in these environments, most reward functions make it optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states.

Optimal Policies Tend To Seek Power
Alexander Matt Turner Oregon State University curneaale@orggonttate edu
Logan Smith Mississippi State University 1s12540msstate. edu
Rohin Shah
UC Berkeley
rohimmshahhbbrrkeeyy edu
Andrew Critch
Prasad Tadepalli Oregon State University
UC Berkeley
critch@berkelly.com edu
tadepall@eecs. wrw.oessorrgnnntttec.oomm edu
Abstract
Some researchers speculate that intelligent reinforcement learning (RL) agents would be incentivized to seek resources and power pursuit of the objectives specify for them. Other researchers point that RL agents need not have human-like power-seeking instincts. To clarify this discussion, we develop the first formal theory of the statistical tendencies of optimal policies. In the context of Markov decision processes (MDPS), we prove that certain environmental symme- tries sufficient for optimal policies to tend seek power over the environment. These symmetries exist in many environments in which the agent can be shut down destroyed. We prove that these environments, most reward functions make it optimal to seek power by keeping range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states.
Introduction
Omohundro [2008], Bostrom [2014], Russell [2019] hypothesize that highly intelligent agents tend to seek power in pursuit of their goals. Such power-seeking agents might gain power over humans. Marvin Minsky imagined that an agent tasked with proving the Riemann hypothesis might rationally turn the planct-along with everyone it-into computational resources [Russell and Norvig, 2009]. However, another possibility is that such concerns simply arise from the anthoopomorphization of Al systems [LeCun and Zador, 2019, Various, 2019, Pinker and Russell, 2020, Mitchell, 2021].
We clarify this discussion by grounding the claim that highly intelligent agents will tend seek power. In section 4, we identify optimal policies as reasonable formalization of "highly intelligent agents." Optimal policies "tend to" take an action when the action optimal for most reward functions. We expect future work to translate our theory from optimal policies to leamed, real-world policies.
Section 5 defines "power" as the ability achieve a wide range of goals. For example, "money power," and money instrumentally useful for many goals. Conversely, it's harder to pursue most goals when physically restrained, and a physically restrained person has little power. An action "seeks power" if it leads to states where the agent has higher power.
We make claims about when large-scale AI power-secking behavior could become plausible. Instead, we consider the theoretical consequences of optimal action MDPs. Section 6 shows that power-secking tendencies arise from anthropomorphism, but from certain graphical symmetries present in many MDPs. These symmetries automatically occur in many environments where the agent can be shut down or destroyed, yielding broad applicability of main result (theorem 6. 13)..
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

2
Related work
An action instrumental an objective when it helps achieve that objective. Some actions are instrumental to a range of objectives, making them convergently instrumental. The claim that power-seeking is convergently instrumental is an instance of the instrumental convergence thesis:
Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent's s goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely be pursued by broad spectrum of situated intelligent agents [Bostrom, 2012|.
For example, in Atari games, avoiding (virtual) death is instrumental for both completing the game and for optimizing curiosity [Burda et al., 2019]. Many Al alignment researchers hypothesize that most advanced Al agents will have concerning instrumental incentives, such as resisting deactiva- tion [Soares et al., 2015, Milli et al., 2017. Madfield-Mened et al., 2017, Carey, 2018] and acquiring resources [Benson- Tilsen and Soares, 2016].
We formalize power the ability to achieve a wide variety of goals. Appendix A demonstrates that our formalization returns intuitive verdicts in situations where information-theowm empowerment does not [Salge et al., 2014].
Some of our results relate the formal power of states to the structure of the environment. Foster and Dayan [2002], Drummond [1998], Sutton et al. [2011], Schaul al. [2015] note that value functions encode important information about the environment, as they capture the agent's ability to achieve different goals. Turner el al. [2020] speculate that a state's optimal value correlates strongly across reward functions. In particular, Schaul et al. [2015] learn regularities across value functions, suggesting that some stales are valuable for many different reward functions (i.e. powerful). Menache et al. [2002] identify and navigate towards convergently instrumental bottleneck states.
We are not the first to study convergence of behavior, form, function. In economies, tumpike theory studies how certain paths of accumulation tend to be optimal [McKenzie, 1976]. In biology, convergent evolution occurs when similar features (e.g. flight) independently evolve in different time periods [Reece and Campbell, 2011]. Lastly, computer vision networks reliably learn egg. edge detectors, implying that these features are useful for a range of tasks [Olah al., 2020].
3 State visit distribution functions quantify the agent's available options
We clarify the power-seeking discussion by proving what optimal policies usually look like in given environment. We illustrate our results with simple case study, before ex- plaining how to reason about a wide range of MDPs. Appendix D.1 lists MDP theory contri- butions of independent interest, appendix D lists definitions and theorems, and appendix E contains the proofs.
Es
Tb
left
right
Figure 1: & is a 1-cycle, and Ø is terminal state. Arrows represent deterministic transitions induced by taking some action E A. Since the right subgraph contains a copy of the left subgraph, proposition 6.9 will prove that more reward functions have optimal policies which go right than which go left at state *, and that such policies seek power- both intuitively, and a reasonable formal sense.
Definition 3.1 (Rewardless MDP). (S. A,)) is rewardless MDP with finite state and action spaces S and A, and stochastic transition function T: S x A + (SS). We treat the discount rate 7 as a variable with domain [0, 1].
Definition 3.2 (1-cyele states). Let e, E R!Sl be the standard basis vector for state 3, such that there a in the entry for state s and 0 elsewhere. State is I-cycle if 3a E A T(s, a) = e State s a terminal state if Va E A: T(s, a)=a)=== =0)==e. =
Our theorems apply to stochastic environments, but present deterministic study for clarity. The environment of fig. 1 is small, but its structure is rich. For example, the agent has more "options" at than at the terminal state Ø. Formally, has more visit distribution functions than Ø does.
2

---

Definition 3.3 (State visit distribution [Suiton and Barto, 1998)). II := A the set of stationary deterministic policies. The visit distribution induced by following policy π from state & at discount rale E [0. 1) is fT (r) = Li=o E [e,,]. f is visit distribution function; F(s) ==
{fTie T E IT).
In fig. 1, starting from l,, the agent can stay at & or alternate between / and & and so F(( = (1- e/
1-7 (er, + ter, }}. In contrast, at 2, all policies т map to visit distribution
function 0-
Before moving on, we introduce two important concepts used in our main results. First, we sometimes restrict our attention to visit distributions which take certain actions (fig. 2).
Definition 3.4 (F single-state restriction). Considering only visit distribution functions induced by
policies taking action a at state 8', F(s TTs') = a) = {f E F(s) 3r E II T(s') = a, f = f].
Second, some E F(s) are "unimportant." Consider an agent optimizing reward function er_ (1 reward when at r oth- erwise) at e.g. 7 3. Its optimal policies navigate to rs, and stay there. Similarly, for reward function e,--, optimal policies navigate to > and stay there. However, for no reward function is it uniquely optimal to alternate between y and ryy Only dominated visit distribution functions alternate between > and T (definition 3.6).
Tp
right
Figure 2: The subgraph corre- sponding to F(* =(() right). Some trajectories cannot be strictly optimal for any reward function, and so our results can ignore them. Gray dotted actions are only taken by the policies of dominated f E F(() (+).
Definition 3.5 (Value function). Let π E II. For any reward function R E R& over the state space, the on-policy value at state and discount rate E [0,1) is Vž (s,7) = (*) where E R (I is R expressed as column vector (one entry per state). The optimal value is Vū (8,7)= = max Vū (s,7).
Definition 3.6 (Non-domination).
(sa = [f" F(s) Br El E (0, :f
max (FE
PTT (7)
(1)
For any reward function R and discount rate 7, f F(s) is (weakly) dominated by fr' E F(s) if (i 7) s V (s, 7). f Fs)(s) is non-daminated if there exist R and y at which f is not dominated by any other fF'.
Some actions have a greater probability of being optimal
We claim that optimal policies "tend" take certain actions in certain situations. We first consider the probability that certain actions are optimal.
Reconsider the reward function er, optimized at y J. Starting from x, the optimal trajectory goes right to 75 to r',, where the agent remains. The right action is optimal at * under these incentives. Optimal policy sets capture the behavior incentivized by reward function and discount rate.
Definition 4.1 (Optimal policy set function). II- (R, 7) is the optimal policy set for reward function R at 7 E (0, 1). All Rt have at least one optimal policy æ E II [Puterman, 2014]. II" (R, 0) = lim,-0 IIo (R, and IIe (I2, 1) = lim II* (I,,7) exist by lemma E.35 (taking the limits with respect to the diserete topology over policy sets).
We may be unsure which reward function agent will optimize. We may expect to deploy system in known environment, without knowing the exact form of e.g. the reward shaping [Ng et al., 1999] or intrinsic motivation [Pathak et al., 2017). Alternatively, RL agents, whose details are unknown. Our power-seeking results do not hinge on such uncertainty, as they also apply to degenerate distributions (i.e. know what reward function will be optimized).
might attempt reason about future
Definition 4.2 (Reward function distributions). Different results make different distributional as- sumptions. Results with Dany E Dany == A A (si) hold for any probability distribution over R D Dround is the set of bounded-support support probability distributions Dround- For any distribution X over IR,

---

DX-ID == X For example, when Xx, == unif(O, 1), Dx.- is the maximum-entropp distribution. Dj is the degenerate distribution on the state indicator reward function ex, which assigns I reward to and O elsewhere.
With Dany representing our prior beliefs about the agent's reward function, what behavior should we expect from its optimal policies? Perhaps we want reason about the probability that it's optimal go from * to Ø, or to go to r, and then stay at >. In this case, we quantify the optimality probability
of = fe. + - e. + te,, or
Definition 4.3 (Visit distribution optimality probability). Let F S FFs)), (0, F... F.., (F1r) =
PR-D... (3f E F: π E II" ())
Alternatively, perhaps we're re interested in the probability that right is optimal
Definition 4.4 (Action optimality probability). At discount rate Y and at state the optimality probability of action is D.., (s, (,,a,7 == PR-Dan (3x* II (R, x >*()) = a).
Optimality probability may seem hard to reason about. It's hard enough to compute an optimal policy for single reward function, let alone for uncountably many! But consider any Dx-m distributing reward independently and identically across states. When y = O, optimal policies greedily maximize next-state reward. At *, identically distributed reward means &s and rs have equal probability of having maximal next-state reward. Therefore, PTx (+, Melleft 0) = PDxx up (+, right, 0). This not proof, but such statements are provable.
With Dr, being the degenerate distribution on reward function ex,, PDe, (+, (e, 1) = =
ID. (+, right, 1). Similarly, P.., (+, (xleff,)) 2) = 0 < PD. right, 4). Therefore, "what do optimal policies 'tend" to look like?" seems to depend on one's prior beliefs. But in fig. 1, we claimed that left optimal for fewer reward functions than right is. The claim is meaningful and true, but we will return to it in section 6.
5 Some states give the agent more control over the future
The agent has more options at & than at the inescapable terminal state Ø. Furthermore, since has loop, the agent has more options r-y than at & A glance fig. 3 leads to intuit that r affords the agent more power than Ø.
What is power? Philosophers have many answers. One prominent answer is the dispositional view: Power is the ability to achieve a range of goals [Sattarov, 2019]. In an MDP, the optimal value function V (s, 7) captures the agent's s ability to "achieve the goal" R. Therefore, average optimal value captures the agent's s ability to achieve a range of goals D
Definition 5.1 (Average optimal value). The average optimal value² at state s and discount rate E (0, 1) is Vin. (8,7) = ERoDra [VA (s, 7)] EpwDrsnn (mxxFeF()) fry)Tr.
Figure 3 shows the plcasing result that for the max-entropy distribution, T. has greater average optimal value than Ø. However, average optimal value has a few problems as mcasure of power. The agent is rewarded for its initial presence at state s (over which it has no control), and be- cause [ ||] = T (proposition E.3) diverges as + 1, lim,--1 V (s, 7) tends to diverge. Definition 5.2 fixes these issues in order to better measure the agent's control over the future.
Definition 5.2 (POWER). Let E 1).
POWER Deama (s.7): =
E Dround
max (E)
1-7 (f(n) es)"T
T
1-
E R Droan
(s, - R(s)]
7
(2)
POWER has nice formal properties.
'Drand''s 's bounded support ensures that ER-D... [VA (8)7) is dell-defined.
AAppendix C relaxes the optimality assumption.
4

---

left
right
Figure 3: Intuitively, state ry affords the agent more power than state Ø. Our POWER formal- ism captures that intuition by computing a function of the agent's 's average optimal value across
range of reward functions. For X, unif(0, 1), Vxx (0,7)
2 1- V Vx (C/,7)
+ Tー + 77), and Vm = 2 and 3 are the expected maxima of one
and two draws from the uniform distribution, respectively, For all 7 E (0.1), VDx cm (2,7)
ax (kx+7) < Dx (r 7). POWERDx,-.. (0,;) = 2. ..WERDx,.. (6/:7)
14, (g +
77), and POWERDx... ()) 2. The POWER of l reflects the fact that when greater reward assigned to & the agent only visits & every other time step.
Lemma 5.3 (Continuity of POWER). POWER (8,7) Lipschitz continuous on y E [0,1].
Proposition 5.4 (Maximal POWER). POWERD (8 S RR-Daas [maxxes R(s)], with equality if s can deterministically reach all states in one step and all states are I-cycles.
Proposition 5.5 (POWER is smooth across reversible dynamics). Let Dround be bounded [b, c). Sup- pose s and s' can both reach each other in one step with probability 1.
POWER (s, 7) POWER Denuet (5',7) (c-b)(1-3)))
We consider power-seeking be relative. Intuitively, "live and keep some options open" seeks more power than "die and keep no options open." Similarly, "maximize open options" seeks more power than "don't 't maximize open options."
Definition 5.6 (PowER-seeking actions). At state s and discount rate 7 E [0, 11, action a seeks more DowER than a' when (s,~T(aa) [POWERD,...s, 7) 2 (.,.a)(saa)) Din.u 7)]-
POWER sensitive to choice of distribution. Di, gives maximal POWERT, to l.. Dr,, assigns maximal POWERD, to r-. Do even gives maximal POWERD, to Ø! In what sense does Ø have "less POWER" than n' and in what sense does right "tend to seek POWER" compared to left?
6 Certain environmental symmetries produce power-seeking tendencies
Proposition 6.6 proves that for all E [0, 1] and for most distributions D, POWERD (l S POWERp(">,,)) 7). But first, we explore why this must be true.
F(L)) (
(er, )ex and F(r.) f (ep + ye, + Ze, These two sets look awfully similar. F(e/) is "subset" of F(rr), only with "different states." Figure demonstrates a state permutation 6 which embeds F(ez) into F(r.).
Involution
left
right
Figure 4: Intuitively, the agent can do more starting from rx, than from &, By definition 6.1, F(r, contains a copy of F(e,): o F((())= = Paes 1- 1 P (ele + yex )} ( (1 e er (e + 5)

---

Definition 6.1 (Similarity of vector sets), Consider state permotation & E Siss inducing [S] x |8] permutation matrix Po in row representation: (Pa).s 1 i = o(i) and 0 otherwise. For X C gls! 6 == [Pox x E x]. X' S T is similar to X when 3Φ X' = X. & is an involution if o = 6- (it either transposes states, or fixes them in place). X contains a copy of X' when X' is similar to subset of X via an involution Φ.
Definition 6.2 (Similarity of vector function sets), Let / R. FF F² are sets of functions + R!S!, F is (pointwise) similar to F' when 3Ø Vay E I (Pof(7) f F) (f( f' E F*}.
Consider a reward function R' assigning reward 2/6 and & and 0 elsewhere. R' assigns more optimal value 6 than to ry: Vir. (ir- = - > 0= Viz (r) Considering from fig. 4, 6 R' assigns reward to > and r and elsewhere. Therefore, Φ- R'' assigns more optimal value w ry, than to / V (e = < T- = (r Remarkably, this @ has the property that for any R which assigns & greater optimal value than rs (i.e. VR(E > VR( 7)), the opposite holds for the permuted ø R: (l <
We can permute reward functions, but we can also permute reward function distributions. Permuted distributions simply permute which states get which rewards.
Definition 6.3 (Pushforward distribu- tion of a permutation). Let & E S s. é Dany is the pushforward distribu- tion induced by applying the random vector f(r)= == Par to Dany.
'D
Dawap
g
Definition 6.4 (Orbit of probabil- ity distribution). The orbit of Dany under the symmetric group S is {ő- anyy 1 S|ss
D
Siss Dany
For example, the orbit of a degener- ate state indicator distribution D, is Sig) D. = (Der 2.. 8 E S). and fig. 5 shows the orbit of 2D Gaussian dis- tribution.
Figure 5: A permutation of a reward function swaps which states get which rewards. Wc will show that in certain situa- tions, for any reward function R. power-seeking is optimal for most of the permutations of R. The orbit of a reward function the set of its permutations. We can also consider the orbit of a distribution over reward functions. This figure shows the probability density plots of the Gaussian distribu- tions DD and D' over R². The symmetric group S2 contains the identity permutation Φia and the reflection permutation Oswap (switching the y and x values). The orbit of D consists of dhs D D and @swap" D D'.
Consider again the involution & of fig. 4. For every Doand for which & has more POWER than ry.. E has less POWER o Drant than r This fact is not obvious- iisuuus--iiinn is shown by the proof of lemma E.24.
Imagine Droand's ' orbit elements "voting" whether & or r has strictly more POWER, Proposition 6.6 will show that ry can't lose the "vote" for the orbit of any bounded reward function distribution. Definition 6.5 formalizes this "voting" notion.3
Definition 6.5 (Incqualitics which hold for most probability distributions). Let f1, f2 A(RIS)) + E be functions from reward function distributions to real numbers and let D C A (R))) We write
(D) > most: D f2(D) when, for all D E D, the following cardinality inequality holds:
|ID' E S DD -
> f2(D')) Z
(D' E Sisi D DI PI((D)) < S(D))
(4)
We write (D( 2most (2( when D is clear from context. Proposition 6.6 (States with "more options" have more POWER). F(s) contains a copy of Fna(s") via é. then Vy E [0, 1] POWERD 7) 2 most POWERDwodd (8),7). If Fna(s) \ 6 Fna(s') non-empty, then for all E (0, 1), the converse Smost statement does not hold.
"The voting analogy and the "most" descriptor imply that we have endowed each orbit with the counting measure. However, a priori, we might expect that some orbit elements are more empirically likely to be specified than other orbit elements. See section 7 for more this point.
6

---

Proposition 6.6 proves that for all E [0. 1/, POWERDool (r,,)) 2 most POWER Droan (C()7) via 8' = 1 ry, and the involution & shown in fig. 4. In fact, because (I (n Fna(e/), r has "strictly more options" and therefore fulfills proposition 6.6's stronger condition.
Proposition 6.6 is shown using the fact that @ injectively maps under which r> has less POWER BRw, to distributions & D which agree with the intuition that rs offers more control. Therefore, at least half of each orbit must agree, and ry never "loses the POWER vote" against t.
6.1 Keeping options open tends POWER-seeking and tends to be optimal
Certain symmetrics in the MDP structure ensure that, compared to left, going right tends to be optimal and to be POWER-seeking. Intuitively, by going r right, the agent has "strictly more choices." Proposition 6.9 will formalize this tendency.
Definition 6.7 (Equivalent actions). Actions ai and ag are equivalent at state (written ai =, 02) if they induce the same transition probabilities: ((s, a1) T(s, Q2).
The agent can reach states in T' T,, by taking actions equivalent to right at state *.
Definition 6.8 (States reachable after taking an action). REACH (s, a) is the set of states reachable with positive probability after taking the action in state
Proposition 6.9 (Keeping options open tends to be POWER-seeking and tends to be optimal).
Suppose Fg == F(s (s)) a) contains copy of Fa: == F(s) (x)) = a') via Φb
1. If REACH as, then by E [0,1] a(,,0)) [POWER (Sa; (sa 2 zmost. D Deane!
E (e.ia((e.a') [POWER ()]-
2. If can only reach the states of REACH (s, a') UREACH (s, a) by taking actions equivalent
to a' or a at state 8, then Vy E 1] : P... (8, a, 7) > most: Den D (s.a',7)
If Fnd(s)) n (Fa / Fax) non-empty, then Vy E (0, 1), the converse Soos statements do not hold.
We check the conditions of proposition 6.9. == x, a' = left, = right. Figure 6 shows that & REACH (+, left) and that can only reach fla, & U (orr,,,
when the agent imme- diately takes actions equivalent to left right. F(x T(*) = right) contains copy of F(* (*) left) via @. Furthermore, Fad (e.
Involution
To
left right
ye, + 2er, + e- e. + 7e,, - e-
Figure 6: Going right is optimal for most reward functions. This is because when- ever makes left strictly optimal over right, its permutation ø R makes right strictly optimal over left by switching which states get which rewards.
{e. - ye,, - e non-empty, and all condi-
tions are met.
For any E [0. and D such that PP (+, left,r) > Po right, 7), environmental symmetry ensures that P&D (+, left, < POOD (x, r). A similar statement holds for POWER.
6.2 When = 1, optimal policies tend to navigate towards "larger" sets of cycles
Proposition 6.6 and proposition 6.9 are powerful because they apply all 7 [0, 1]. but they can only be applied given hard-to-satisee environmental symmetries. In contrast, proposition 6.12 and theorem 6. 13 apply many structured environments common to RL.
Propposiionn 6.6 also proves that in general, has less POWER than e and However, does not prove that most distributions D satisfy the joint incquality POWERD (2), 7) <. (oWERRp(, (f,,,) < (rw.rp(r>..)) 7)). This only proves that these inequalities hold pairwise for most D. The orbit elements D which agree that Ø has less POWER than & need not be the same elements D' which agree that & has less POwERp than rsy.

---

Starting from x. consider the cycles which the agent reach. Recurrent state distributions (RSDs) generalize deterministic graphical cycles to potentially stochastic environments. RSDS simply record how often the agent tends to visit state in the limit of infinitely many time steps.
Definition 6. 10 (Recurrent state distributions [Puterman, 2014)). The recurrent state distributions which can be induced from state are RSD (s) == {limy (1 - ()) π E ETI.. RSD (8) the set of RSDS which strictly maximize average reward for some reward function.
cr (e + er (, As dis- cussed in section 3, ( er is dominated: Alternating between y and rs. is never strictly better than choosing one or the other.
As suggested by fig. 3, RSD (+) =
+
reward function's is optimal policies can vary with the discount rate. When = 1, optimal policies ignore transient reward because average reward is the dominant consideration.
Definition 6.11 (Average-optimal policies). The average-optimad policy set for reward function R is I1 (R) = E II Vs E S d E arg maX d d'r) rr} (the policies which induce optimal
RSDS at all states). For D S RSD (s), the average optimality probability is P (D, average) =
IP R- Pay (3d D क E II (R)).
Average-optimal policies maximize average reward. Average reward is governed by RSD access. For example, x has "more" RSDs than Ø; therefore, rs, usually has greater POWER when ? 1.
Proposition 6.12 (When 7 1, RSDS control POWER). IfRSD (s) contains copy of RfDDd (s') via é. then Po (s. 1) 2mast POWERD (s'.1). IfRSD (s) (-RSDna(s') non-empty, then the converse Smast statement does not hold.
We check that both conditions of proposition 6.12 are satisfied when = Ø. = ry,, and the involution o swaps Ø and r-. Formally, o RSDnaa (Ø) (S C [r]. The conditions are satisfied.
{ee] (er,,, S fer )
Informally, states with more RSDS generally have more POWER at 7 1, no matter their transient dynamics. Furthermore, average-optimal policies are more likely to end up in larger sets of RSDs than in smaller ones. Thus, average-optimal policies tend navigate towards parts of the state space which contain more RSDs.
left
right
Figure 7: The cycles in RSD (+). Most reward functions make it average-optimal to avoid Ø, because is only a single inescapable terminal state, while other parts of the state space offer more 1-cycles.
Theorem 6.13 (Average-optimal policies tend to end up "larger" sets of RSDS). Let D, C RSD (s). Suppose that D contains a copy of D' via é, and that the sets DUD' and RSD (s) (D' D) have pairwise orthogonal vector elements (i.e pairwise disjoint vector support). Then P (veaaergee) Zmmst P D.... (D', average)) DSD (s)n(D) o D') is non-empty, the converse Smost statement does not hold.
Corollary 6.14 (Average-optimal policies tend not end up in any given 1-cycle). Sup- pose eoy: eyy E RSD (s) are distinct. Then FD.... (RSD (s) \ (e,, }. average) Zmost D... ((e,,) average). If there is third e.". RSD (s), the converse Smost statement does not hold.
Figure 7 illustrates that eø. Cr, Cr, E RSD (+). corollary 6.14 hold: PD.., (RSD f average) 2mssi
Thus, both conclusions of
P PD (fee), average) and
8

---

D... (RSD (+) \ {ez), average) most PD.., (feø), J, average). In other words, average-optimal policies tend to end up in RSDS besides 2. Since ø is terminal state, it cannot reach other RSDs. Since average-optimal policies tend to end up other RSDs, average-optimal policies tend to avoid Ø
This section's ss results prove the ? = case. Lemma 5.3 shows that POWER is continuous a 1. Therefore, if an action is strictly -oweRi-sceking when 1, it is strietly PowERp-scckingg at discount rates sufficiently close to 1. Future work may connect average optimality probability to optimality probability at ~ 1.
I Lastly, our key results apply to all degenerate reward function distributions. Therefore, these results apply not just to distributions over reward functions, but to individual reward functions.
6.3 How to reason about other environments
Consider an embodied navigation lask through a room with a vase. Proposition 6.9 suggests that optimal policies tend to avoid immediately breaking the vase, since doing so would strictly decrease available options.
Theorem 6. 13 dictates where average-optimal agents tend to end up, but not what actions they tend to take in order reach their RSDS. Therefore, care is needed. In appendix B, fig. 10 demonstrates an environment in which seeking POWER a detour for most reward functions (since optimality probability measures "median" optimal value, while POWER is function of mean optimal value). However, suppose the agent confronts a fork in the road: Actions a and a' lead to two disjoint sets of RSDS D2 and Da', such that Da contains a copy of Dez Theorem 6. 6.13 shows that will tend be average-optimal over a', and proposition 6.12 shows that a will lend to be POWER wee-seinnng compared to a'. Such forks seem reasonably common in environments with irreversible actions.
Theorem 6.13 applies to many structured RL environments, which tend to be spatially regular and to factorize along several dimensions. Therefore, different sets of RSDs will be similar, requiring only modification of factor values. For example, an embodied agent can deterministically navigate set of three similar rooms (spatial regularity), then the agent's position factors via (room number] x (position in room). Therefore, the RSDS can be divided into three similar subsets, depending on the agent's room number.
Corollary 6. dictates where average-optimal agents tend to end up, but not how they get there. Corollar 6.14 says that such agents tend not to stay in any given I-cycle. It does not say that such agents will avoid entering such states. For example, in an embodied navigation task, a robot may enter 1 -cyclle by idling in the center of room. Corollary 6. 14 implies that average-optimal robots tend not to idle in that particular spot, but not that they tend to avoid that spot entirely.
However, average- opttimal robots do tend to avoid getting shut down. The agent's 's task MDP often represents agent shutdown with terminal states. A terminal state is, by definition 3.2, unable to access other 1-cycles. Since corollary 6.14 shows that average-optimal agents tend to end up in other 1-cycles, average-optimal policies must tend to completely avoid the terminal state. Therefore, we conclude that in many such situations, average-optimal policies tend to avoid shutdown. Intuitively, survival is power-seeking relative to dying. and so shutdown-avoidann
is power-seeking behavior.
In fig. 8, the player dies by going left, but can reach thousands of RSDS by heading in other directions. Even if some average-optimal policies go left in order to reach fig. 8's "game over" terminal state, all other RSDS cannot be reached by going left. There are many 1 1-cycles besides the immediate terminal state. Therefore, corollary 6.14 proves that average-optimal policies tend not go left in this situation. Average-optimal policies tend to avoid immediately dying in Pac-Man, even though most reward functions do not resemble Pac-Man's 's original score function.
left
right
Figure 8: Consider the dynamics of the Pac-Man video game. Ghosts kill the player, at which point consider the player enter a "game over" terminal state which shows the final configuration. This rewardless MDP has Pac-Man dynamics, but not its usual score function. Fixing the dynamics, as the reward function varies, right tends to be average-optimal over teft. Roughly, this is because the agent can do more by staying alive.

---

7 Discussion
Reconsider the case of hypothetical intelligent real-world agent which optimizes average reward for some objective. Suppose the designers initially have control over the agent. If the agent began to misbehave, perhaps they could just deactivate it. Unfortunately, our results suggest that this strategy might not work. Average-optimal agents would generally stop from deactivating them, if physically possible. Extrapolating from our results, we conjecture that when ς R 1, optimal policies tend to seek power by accumulating resources- the detriment of any other agents in the environment.
Future work. Most real-world tasks are partially obscrvable, and in high- ligh-dimensional environments, even superhuman learned policies are rarely optimal. However, the field of RL aims to improve learned policies toward optimality. Although our results only apply to optimal policies in finite MDPs, we expect the key conclusions to generalize. Furthermore, irregular stochasticity in environmental dynamics can make it hard to satisfy theorem 1.1sss similarity requirement. We look forward to future work which addresses partially observable environments, suboptimal policies, or "almost similar" RSD sets.
Past work shows that it would be bad for an agent to disempower humans in its environment, In a two-playcr agent / human game, minimizing the human's 's information -etoorrticcctcccccco empowerment [Salge et al., 2014] produces adversarial agent behavior [Guckelsberger et al., 2018]. In contrast, maximizing human empowerment produces helpful agent behavior [Salge and Polani, 2017, Guckelsberger et al., 2016, Du al., 2020]. We do not yet formally understand when, why PoWER-seeking policies tend to disempower other agents in the environment.
More complex environments probably have more pronounced power-seeking incentives. Intuitively, there are often many ways for power-seeking to be optimal, and relatively few ways for power-seeking not to be optimal. For example, suppose that in some environment, theorem 6.13 holds for one million involutions é. Does this guarantee more pronounced incentives than if theorem 6.13 only held for one involution?
We proved sufficient conditions for when reward functions tend to incentivize power-seeking. In the absence of prior information, one should expect that arbitrary reward function incentivizes power-seeking behavior under these conditions. However, we have prior information: Al designers usually try to specify good reward function. Even so, it may be hard to specify orbit elements which do not incentivize bad power-seeking.
Societal impact. We believe that this paper builds toward rigorous understanding of the risks presented by Al power-seeking incentives. Understanding these risks is the first step in addressing them. However, basic theoretical work can have many consequences. For example, this theory could somehow help future researchers build nower-seeking agents which disempower humans. We believe that the beneñt of understanding outweighs the potential societal harm.
Conclusion. We developed the first formal theory of the statistical tendencies of optimal policies in reinforcement learning. In the context of MDPS, we proved sufficient conditions under which optimal policies tend to seek power, both formally (by taking POWER- taking actions which keep the agent's 's options open). Many real-world environments have symmetries which produce power-seeking incentives. In particular, optimal policies tend seek power when the agent can be shut down or destroyed. Seeking control over the environment will often involve resisting shutdown, and perhaps monopolizing resources.
actions) and intuitively (by
We caution that many real-world tasks are partially observable and that learned policies are rarely optimal. Our results do not mathematically prove that hypothetical superintelligent AI agents will seek power However, we hope that this work will foster thoughtful, serious, and rigorous discussion of this possibility.
Acknowledgments
Alexander Turner supported by the Berkeley Existential Risk Initiative and the Long- Long-Term Future Fund. Alexander Turner, Rohin Shah, and Andrew Critch were supported by the Center for Hluman-Compatte AI. Prasad Tadepalli supported by the National Science Foundation.
10

---

Yousif Almulla, John E. Ball, Daniel Blank, Steve Byrnes, Ryan Carey, Michael Dennis, Scott Emmons, Alan Fern, Daniel Filan, Ben Garlinkel, Adam Gleave, Edouard Harris, Evan Hubinger, DNL Kok, Vanessa Kosoy, Victoria Krakovna, Cassidy Laidlaw, Joel Lehman, David Lindner, Dylan Mudfield-Menche Richard Möhn, Alexandra Nolan, Matt Olson, Neale Ratzlaff, Adam Shimi, Sam Toyer, Joshua Turner, Cody Wild, Davide Zagami, and our anonymous reviewers provided valuable feedback.
References
Tsvi Benson-Tilsed and Nate Soares. Formalizing convergent instrumental goals. Workshops at the Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Nick Bostrom. The superintelligent will: Motivation and instrumental rationality in advanced artificial agents. Minds and Machines, 22(2):11-85, 2012.
Nick Bostrom. Superintelligence. Oxford University Press, 2014.
Yuri Burda, Harri Edwards. Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driver learning. International Conference on Learning Representations, 2019.
Ryan Carey, Incorigibility in the CTRL framework. AI, Ethics, and Society, 2018.
Chris Drummond. Composing functions to speed up reinforcement learning in a changing world. In Machine Learning: ECML-98, volume 1398, pages 370-381, 381. Springer, 1998.
Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan, AvE: Assistance via empowerment. Advances in Neural Information Processing Systems, 33, 2020.
David Foster and Peter Dayan. Structure the space value functions. Machine Learning, pages 325-346. 2002.
Christian Guckelsberger, Christoph Salge, and Simon Colton. Intrinsically motivated general companion NPCs coupled empowerment maximisation. IEEE Conference on Computational Intelligence and Games, pages 1-8, 2016.
Christian Guckelsberger, Christoph Salge, and Julian Togelius, New and surprising ways to be mean. IEEE Conference on Computationa Intelligence and Games, pages 1-8, 2018.
Dylan HadfilddMMnnll Anca Dragan, Pieter Abbeel, and Stuart Russell. The off-switch game. Proceedings of the Twenty-Sin Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 220-227, 2017.
Yann LeCun and Anthony Zador. Don't fear the Terminator, September 2019. URL https://bloss. scientificammricee com/ wwwpo/aatioss/ee///oooooooo wwweeiiiiinnnntttieeeeeeeeeee fear- the-terminator/.
Steven A Lippman. On the set of optimal policies in discrete dynamic programming. Journal of Mathematical Analysis and Applications, 24(2)440-445, 1968.
Lionel W McKenzie. Turnpike theory. Econometrica: Journal of the Econometric Society, pages 841-865, 1976.
Ishai Menache, Shie Mannor, and Nahum Shimkin. Docutddynad discovery of sub-goals in reinforcement learning. In European Conference Machine Learning, pages 295-306. Springer, 2002.
Smitha Milli, Dylan Hadfield- Menell, Anca Dragan, and Stuart Russell. Should robots be obedient? In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 4754 2017.
Melanie Mitchell. Why Al harder than we think. arXiv preprint
Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformation:: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, pages 278-287. Morgan Kaufmann, 1999.
Chris Olah, Nick Cammarata. Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom An introduction to circuits. Distill, 2020.
Stephen Omohundro. The basic AT drives, 2008.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. In JCML,
11

---

Steven Pinker and Stuart Russell. The foundations, benefits, and possible existential threat of AI, June 2020. URL httpps:/ httpiiiee.rggg/////////////// arg//220/06/15//teeen http////////////////////// wiweeeen--iiee-aaeeeee and- stuart-ruseell the founoatinnnnnssss -eneeeiiooneeeeeeteeeeeeeee and podd-ossebe-- existential riiaalr-r------- risk-of-ai. of-ai/
Martin I. Puterman, Markov decision processes: Discrete stochastic dynamic progranmiigg. John Wiley & Sons, 2014.
J.B. Reece and N.A. Campbell. Campbell Biology. Pearson Australia, 2011.
Kevin Regan and Craig Boutilier. Robust policy computation in reward-uncertall MDPs using nondominated policies. Twenty-Fourty Fourth AAAI Conference Artificial Intelligence, 2010.
Stuart Russell. Human compatible: Artificial intelligence and the problem control. Viking, 2019.
Stuart Russell and Peter Norvig. Artificial intelligence: modern approach. Pearson Education Limited, 2009.
Christoph Salge and Daniel Polani. Empowerment as replacement for the three laws of robotics. Frontiers Robotics and AI, 4:25, 2017.
Christoph Salge, Comellus Glackin, and Daniel Polani. Enppowemennt-nn introduction. In Guided Self- Organization: Inception, pages 67-114. Springer, 2014.
Faridun Sattarov. Power and technology: a philosophical and ethical analysis. Rowman & Littlefield Interna- tional, Lid, 2019.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning, pages 1312-1320, 2015.
Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. AAAI Workshops, 2015.
Richard S Sutton and Andrew G Barto. Reinforcement learning: an introduction. MIT Press, 1998.
Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In International Conference on Autonomous Agents and Multiagent Systems, pages 761-768,
2011.
Alexander Matt Tumner, Dylan HadfieldMMmmell, and Prasad Tadepalli, Conservative agency via attainable utility preservation. In Proceddings of the AAAI/ACM Conference on A1, Ethics, and Society, pages 385-391, 2020.
Various. Debate on instrumental convergence between LeCun, Russell, Bengio, Zador, and more, 2019. URL battpa://wwtt al www.llignmntttuuo for.. www.orgggg...o.eeeeo... warrg/oots///iie /aats////iiG.eeff222//m//a////// wetpn///debbbetebeeeeeee on- -at.raatteeo-nnnttmmetttaaa instrumental- convergence--eeteeen- 1ecun-ruseellel
Tao Wang, Michael Bowling, and Dale Schuurmans. Dual representations for dynamic programming and rein- forcement learning. International Symposium Approximate Dynamic Programming and Reinforcement Learning, pages 44-51, IEEE, 2007.
Tao Wang. Michael Bowling, Dale Schuummans, and Daniel J Lizotte. Stable dual dynamic programming. In Advances Neural Information Processing Systems, pages 1569-1576, 2008
12


Reviews:
[{"review_id": "nTwfF8PavN", "reviewer": "Reviewer_xxyk", "summary": "The paper formalizes a notion of power-seeking in MDPs and shows that many reward functions lead to optimal policies that achieve powerful states.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a significant step towards settling a long-standing debate that most AI researchers will have considered or even participated in but only in an informal context. It is also an important debate as it affects the field’s priorities. The result is perhaps not surprising to everyone but nonetheless important because it contributes to this ongoing debate. Not only the results but also the formalizations will be useful for future research and discussion. Taken together, the paper is likely to be among the most high-impact ones at Neurips. \n\nAlthough the community has expected that results like the ones in this paper can be proven, my impression is that, it has been difficult to do so with any generality and therefore nothing is published yet. It is good to see results with some generality now.\n\nPerhaps as a side effect of trying to achieve more general results, the derivations with substantial amounts of notation are a bit tedious to follow at times. Although the prose is well-written, mathematical presentation could be further improved.\n\n-------\n\n\nMinor comments:\n\n- Can we use these insights to engineer a system that avoids power seeking?\n\n- First paragraph of discussion: not clear what is the point of this paragraph\n\n- Can you discuss more clearly why and to what extent your results apply to many MDPs?\n\n- It’s obvious but maybe mention why an analysis of optimal policies is useful. I.e. because our community is constantly improving agents to produce closer to optimal policies\n\n-------\n\nComments on presentation/writing:\n\n- The paper could benefit from an illustrative example. The MDP graphs help but they are not self-explanatory. One way to address this would be to cast the MDP in terms of concrete objects and explain the intuitions in the caption. Additionally, section 6 could also benefit from an intuitive characterization both in section 6 itself and in other parts of the paper that refer to it. Otherwise I fear you’ve lost many readers by this point. This should also explain why many MDPs should have the proposed symmetries.\n\n- The paper could state more clearly throughout if the goal is to show that agents seek influence (e.g. by earning money), or seek survival, or both. The last sentence of the discussion highlights this issue - can you discuss the evidence that your results can be extrapolated to accumulating resources? The concept of ‘keeping options open’ sounds more like postponing commitment than e.g. accumulating resources or influence. A more appropriate term might be ‘maximizing open options’?\n\n- The notation is overall a cluttered which can make the derivations tedious to follow. \n\n- “into computational resources” - while this is related to power seeking it seems more of a central example for side effects\n\n- “Section 5 defines “power” as the ability to achieve a wide range of goals: after all, “money is power”, and money is instrumentally useful for many goals.”: the first part of the sentence (“power” as the ability to achieve a wide range of goals) does not follow from the second.\n\n- I’d recommend using less scare quotes \n\n- “Optimal policies for e.g. Pac-Man do not seek power in the real world”: I don’t follow, why would someone deploy a pacman policy in the real world?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are adequately addressed in the discussion.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "X7YA4kMpLHm", "reviewer": "Reviewer_un2T", "summary": "The paper proposes a formal definition for a measure of power in reinforcement learning. It is expressed as a scaled expectation of the difference of the optimal value function in a state and the immediate reward in that state, where the expectation is taken over possible reward functions and it is scaled by (1-\\gamma)/\\gamma. Using this measure, the authors show that any state or action that leaves more options open has more power. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally very well-written and propose an intriguing framework to discuss about whether intelligent agent may seek more power. I also appreciate the small example of Markov decision process, which is provided to illustrate all the results.\n\nHowever, I am not completely convinced that the proposed measure of power helps answer the initially-raised question of whether an intelligent agent would seek power, a question that I understand as focusing on one given agent, which would in the course of solving a given task seek power. It seems to me that the authors rather provide an answer to the question of whether given many intelligent agents with potentially different reward functions would they on average favor some states or actions for their optimal policies. For this latter question, the obtained results can be understood quite intuitively since a state or an action that leaves option open can potentially satisfy more various reward functions.\nAlthough the authors mention that their results apply to the case with a degenerate distribution, it is not clear to me how the proposed measure is useful to answer the former question about one given agent. I think more discussion on that point would be warranted.\n\nBesides, could the introduced notion of power be extended when the reward function depends on the action and/or future state?\n\nPOST-REBUTTAL\nThe authors' responses clarify the first point I raised and address my concern. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the societal impact of their work in a satisfying way.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "JRBlVjR6rEs", "reviewer": "Reviewer_TsSV", "summary": "The paper presents a new way to think about the power of an MDP state with a view to understanding when and why autonomous agents seek power in a given environment. Similar to other treatments \"power\" captures something about the freedom to act towards diverse objectives, and the authors relate this appropriately to that literature, e.g. empowerment.\n\nThe authors begin with motivation to develop a notion of power that is free from anthropomorphic a priori assumption. However, this is a mostly theoretical/mathematical contribution with a series of definitions, e.g. the power-function of a state, and theoretical results, e.g. smoothness properties of their power-function and gives a general perspective of why autonomous agents might act to seek power (i.e. to navigate towards states of high power). The later results exploit ways to think about symmetries in MDPs based on permutations of states, and allows for quite general results pertaining to agents navigating to states of high power under weak assumptions.\n\nThe paper ends with a discussion of how the concrete mathematical arguments can be applied to more general domains including those of a general agent acting in an open environment. One conclusion is a warning about how weak assumptions about how a an autonomous learner is motivates can lead to impulsions to \"seek power\" and explain that this may have relevance for the safe development of autonomous agents operating in society.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors develop their mathematical ideas for an MDP with infinite horizon (i.e. with absorbing/terminal states represented as those with only reflexive outgoing edges of the graph) and then define states in terms of their set of available \"visit distribution functions\" and various restrictions of this notion. This is a very neat way to abstract away from many of the complexities arising from such things as multiple actions having the same outcome, and lays the foundations for simliar neat abstractions on top of this, such as the non-dominated set of visit functions.\n\nSection 5 gives a result that states something along the lines of \"this state has a higher expected optimal value\" (as an expectation over reward functions drawn from a known distribution) and consequently defines the power of a state as a related function with various goodness properties. I had some concerns reading the opening sections that there were going to be (strong) implicit assumptions about the distribution of tasks/reward functions, and these concerned were alleviated here, as the assumptions are explicitly stated and in many cases only weak constraining (e.g. that the functions/distributions over functions are bounded). \n\nSection 6 defines the orbit of a distribution as a set of distributions resulting from permutations of the original distribution's dimensions, and another novel (to me) idea that similarity up to permutation between a one states visit distribution (functions) and a subset of another's visit distribution (functions). This gives them a way to explore how fairly general symmetries in the environment can be used to score states and actions within states with very weak assumptions over reward functions. Finally, the authors give results relating to Blackwell optimal policies (essentially average reward optimisers) again reasoning in a general way about the states that \"tend\" to be navigated towards by \"most\" policies. The descriptive language here lets them down a little, as I think they could be clearer in explaining what they mean by such things as \"tend\", \"usually\" and \"most\" but the mathematical definitions are clear.\n\n\nI find the arguments here compelling and novel and expect them to have relevance to the RL community as well as more widely (especially the final few comments relating to autonomous agents acting in society). The arguments are clearly made with formal mathematical definitions. I spotted a few minor things that were either unclear or ambiguious, but on the whole it was exceptionally clear mathematically and well described in natural language.\n\nI have a few specific points, e.g. typos etc, these are:\n\n* [line 39 onwards] The definition of robustly instrumental is making some assumption about distribution of tasks. That could be clearer.\n* [line 64] The comment about computer vision systems reliably learning is less obviously tied to the line of reasoning than it could be.\n* [line 111] The set of \"non-dominated\" visit distribution functions seems to comprise of visit functions that strictly dominate for at least one reward function. Would a better name be \"dominating\" visit distribution functions? It's a minor point but one that confused me for a while.\n* [lines 129] For $D_{bound}$ are you assuming a distribution whose support is bounded, or a distribution with bounded variance. I assume the former, but this could be clearer, and the former would exclude a gaussian distribution.\n* [lines 144-148] This was a bit difficult for me to parse.\n* [line 166] By \"count the agent's initial presence\" I think you mean \"reward\" the agent for its presence in state s at the beginning of any trajectory from s. \n* [Figure 3 caption] The last sentence should end with something like \"...when greater reward is assigned to $l_{\\nwarrow}$.\n* [line 194] drop the word \"instead\"\n* [lines 201-206] I get a bit lost here in the example.\n* Definition 6.7 uses a very similar notation for a subset of possible states as it does for the full set of states (only the font style changes).\n* [lines 333-334] It isn't really clear what you mean by randomly generated MDPs nor why they would be \"unlikely\" to have any particular property. Nor is it entirely clear what sufficient conditions you are referring to.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is a clear route to societal impact outlined by the authors, and while it isn't conclusive, the authors are asking for a debate and I think that is valid.\n\nMy concerns are relatively minor and tend to relate to the justifications for a implications of the theoretical work rather than the work itself. These are:\n\n* A comment that different results make different distributional assumptions over rewards appears on line 129 in section 4, but it would be good to have this stated very early on e.g. abstract/introduction.\n* That at times (towards the end of the paper), I am not sure whether the descriptive language appropriately captures the mathematical concepts being developed. For instance, the use of the words \"tends\", \"usually\" and \"most\". While there are distributions applied to reward functions, the orbits are sets of distributions and there is no measure defined over this set. The use of adverbs of degree to describe the high frequency outcomes in a subset of distributions should be made a little clearer I think.\n* In fact, it could be clearer what the motivation is for defining distributions of plausible reward functions in the way that is done. It may be the case that certain topological properties of sub-parts of a naturally arising decision problem correlate with reward function over that sub-part. In fact, it is extremely uncommon in games/human activities for task success to be associated with hard to reach states. That isn't to say these states aren't of high power, just that the assumptions about IID rewards and orbits based on permutations of rewards, may not always be applicable.\n* On lines 320-323, you state that your result applies to all degenerate distributions. However, we should remember that the orbit of a degnenerate distributions is any permutation of a degenerate distribution. Your earlier use of \"tends\" and \"most\" etc , seems to suggest that you are treating this orbit in some kind of measurable way, e.g. as a distribution of its own, and so the message here isn't as clear as it could be.\n\nThese issues could be addressed with a couple of comments relating to the validity/scope of assumptions and how to interpret particular choices of words.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "FLqGCE72T4t", "reviewer": "Reviewer_U4CB", "summary": "The paper formalises the notion of an agent's power in a state $s$ as the average optimal value of that state, which the authors claim can be measured using a related power function. The paper posits that most optimal policies tend to seek states that achieve a higher value of power according to the power function under the set of bounded reward distributions. This is explained by optimal policies keeping options open, which naturally leads to a greater probability of optimal value for at least one option under the set of bounded reward distributions, and hence greater power. The paper then gives conditions in terms of symmetries in the MDP where such behaviour can be proved. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Taking a dispositional view of power 'as the ability to achieve a wide variety of goals' is certainly valid, however I have concerns about the limitations of its formalisation here. \n\n1. PATHOLOGIES IN THE FORMALISATION OF POWER\n\nThe paper posits that 'in an MDP, optimal value functions capture the agent’s ability to “achieve the goal” R. So average optimal value captures the agent’s ability to achieve a range of goals'. In the current definition of the MDP, I'm not sure that this is true as this does not account for preference over goals in terms of their optimal value. It would seem that using the average optimal reward would be vunerable to certain pathologies. As a counter example, consider the following MDP:\n\nConsider a starting state $s_0$ and two actions that lead the agent deterministically to $s_l$ and $s_r$. In $s_l$, the agent recieves a reward of $R_l$ and then transitions to a terminal state $s_t$ with reward of 0. In $s_r$, the agent has a choice of 9 actions, each deterministically transitioning to a state $s_{r_i}$, $i\\in\\{1:9\\}$. The reward for each state is $R_{r_i}$. The agent then transitions to the terminal state $s_t$.\n\nIf $R_l >> \\max\\_{i\\in\\{1:9\\}} R_{r_i}$, then the average optimal value for being in $s_l$ is much higher than $s_r$. However being in state $s_l$ only allows the achievement of one goal, not a wide variety of goals. One could avoid such situations by introducing a restriction on rewards to normalise optimal value functions, however I believe this would severely limit the scope of the work as it seems natural that some goals should always have more value to agents than other: As an example of where this sort of MDP could be encountered, consider the option of somebody training for a very specific but very powerful job represented by entering in state $s_l$ vs obtaining a more general qualification that would allow that person to do a wide variety of jobs, but none of which would have as much power (or earn as much money) as the job in $s_l$. It seems natural that we would want to account for these situations if we were to formalise power. Perhaps a better solution would be to re-define or at least clarify what is meant by power to include a notion that powerful states can also lead agents to goals that a achieve a greater affect on the world in terms of optimal value at the expense of being able to achieve fewer goals overall.\n\n2. DIFFERENCE BETWEEN $\\textrm{Power}$ FUNCTION AND AVERAGE OPTIMAL VALUE\n\nIgnoring this pathology, there is also a disconnect between the measure of power in terms of average optimal value and the $\\textrm{Power}$ function that isn't explained. This is revealed by manipulating its definition:\n$$\n\\begin{aligned}\\textrm{Power}_\\mathcal{D}(s,\\gamma):=&\\frac{1-\\gamma}{\\gamma} \\mathbb{E}_\\mathcal{D}\\left[V_R^*(s)-R(s)\\right],\\\\\\\\\n=&\\frac{1-\\gamma}{\\gamma} \\mathbb{E}_\\mathcal{D}\\left[ \\mathbb{E}\\_{P\\^\\star(\\tau\\vert s)}\\left[R(s) +\\sum\\_{i=1}^\\infty\\gamma^i R(s_i)\\right]- R(s)\\right],\\\\\\ \n=&(1-\\gamma)\\mathbb{E}_\\mathcal{D}\\left[ \\mathbb{E}\\_{P\\^\\star(\\tau\\vert s)}\\left[ \\sum\\_{i=1}^\\infty\\gamma\\^{i-1} R(s_i)\\right]\\right],\\\\\\\\\n=&(1-\\gamma) \\mathbb{E}_\\mathcal{D}\\left[\\mathbb{E}\\_{P\\^\\star(s_1\\vert s)}\\left[V^*_R(s_1)\\right]\\right],\n\\end{aligned}\n$$\nwhich is the average optimal value of the next state given state $s$. This leads to a different definition of power that ignores the reward of the current state (notice how it cancels between lines 2 and 3 above). It is easy to construct counterexample MDPs that would lead to opposing notions of which state is more powerful under the average optimal value definition and the $\\textrm{Power}$ function in the same vein as the pathological example above. In addition, the $\\textrm{Power}$ function is not motivated as a replacement for average optimal value beyond it not diverging in the limit $\\gamma\\rightarrow 1$. If this is a critical property, why not just use the optimal average value scaled by $(1-\\gamma)$? As we see in the above derivation, this is essentially what you are using but averaged over the transition dynamics for the next state, and so by using optimal average value scaled by $(1-\\gamma)$ you will obtain the same desired properties without changing its meaning. \n\n3. UNCLEAR SIGNIFICANCE OF CONTRIBUTION\n\nTo me, it seems like the paper proves an obvious observation that can be elucidated by considering the MDP in 1 again. It seems somewhat trivial that for most bounded reward distributions over the reward function, being in state $s_r$ would lead to greater average optimal value as $9$ out of $10$ times the reward function would give the highest reward to one of $s_{r_i}$. Hence having more options would clearly lead to greater power under your definition. Although the paper seems to formalise and generalise this observation, which I think may be a valuable contribution as an abstract peice of theory, I'm not sure that it makes sense in the context of trying to formalise power. As discussed in 1, characterising power in this way is very limited, precisely because in many situations it doesn't make sense to consider all bounded distributions of rewards: we want to preserve some notion that highly specific states can lead to more power because they tend to give greater reward, even if we are uncertain about precisely what that reward is. \n\n4. VALID CONTRIBUTION IN TERMS OF AI SAFETLY\n\nI do, however, believe there is a valid contribution in the context of analysing the behaviour of a agent trying to be shut down in an MDP. If this was the sole focus of the work, rather than attempting to formalise a notion of power, I think it would improve its clarity and contribution significantly. \n\nEDIT:\n\nFollowing discussion with authors, most of my concerns have been addressed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "On a philosophical level, I think that the authors' formalisation that power is being in a state that can lead to many rewarding goals being achieved in terms of average optimal value is limited and doesn't fully account for power under the working definition given in the paper, i.e. that 'power is the ability to achieve a range of goals [Sattarov, 2019]'. For example, the agent might not be able to learn the optimal value to act optimally, they might be uncertain about the MDP or even have incorrect knowledge of the state they are in - in the context of an AI agent avoiding being shut down, the agent is clearly in a lesser state of power if we can withold information about it ever being able to be shut down. Even the MDP the agent is in must be an essential part of understanding power. All of these other aspects are necessary for an agent to have power in terms of being able to actually achieve a goal. However these apsects are ignored in this work. As a real-life example, imagine that somebody has inherited a car. Unfortunately, they have no idea how to drive it. Having a car clearly increases the agent's power under the authors' definition of average optimal value as they are in a state where their actions have the potential to acheive a wider ranger of goals. Indeed, it would be easy to show that their average optimal value in this state is higher than a state without a car.  Crucially though, they don't have access to the optimal value function to be able to actually drive it. This means that they may actually be in state of less power (now being burdened with road tax , storage fees, nobody wanting to buy it etc.). One could posit a way of resolving this by making knowledge part of the agent's state, however this would mean extending state to include the agent's knowledge of the value function, which clearly creates issues in an MDP as any optimal value function would depend on knowledge of itself! \n\nI believe this issue can be resolved by acknowledging that there are many necessary and sufficient aspects of being in a position of power, with one sufficient aspect being that states where more options are available can be more powerful. I think taking this perspective, discussing (or better formalising) other necessary aspects of power as well as fixing the pathologies identified above would significantly improve this paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The discussion with the authors helped clear all the reviewer concerns.  The scores were all updated to clear accept.
Decision: Accept (Spotlight)