Parametrically Retargetable Decision-Makers Tend To Seek Power
Authors: Alexander Matt Turner
Prasad Tadepalli
Venue: NeurIPS 2022 Accept (2022)
Keywords: power
alignment
safety
rl
reinforcement learning
theory

Abstract: If capable AI agents are generally incentivized to seek power in service of the objectives we specify for them, then these systems will pose enormous risks, in addition to enormous benefits. In fully observable environments, most reward functions have an optimal policy which seeks power by keeping options open and staying alive. However, the real world is neither fully observable, nor must trained agents be even approximately reward-optimal. We consider a range of models of AI decision-making, from optimal, to random, to choices informed by learning and interacting with an environment. We discover that many decision-making functions are retargetable, and that retargetability is sufficient to cause power-seeking tendencies. Our functional criterion is simple and broad. We show that a range of qualitatively dissimilar decision-making procedures incentivize agents to seek power. We demonstrate the flexibility of our results by reasoning about learned policy incentives in Montezuma's Revenge. These results suggest a safety risk: Eventually, retargetable training procedures may train real-world agents which seek power over humans.

Parametrically Retargetable Decision-Makers Tend To Seek Power
Alexander Matt Turner, Prasad Tadepalli
Oregon State University
(turneale0, tadepall@eecs. @adepalleecs.Jorgoostteeee Joregonstate. edu
Abstract
If capable Al agents are generally incentivized to seek power in service of the objectives specify for them, then these systems will pose enormous risks, in addition to enormous benefits. In fully observable environments, most reward functions have optimal policy which seeks power by keeping options open and staying alive [Turner et al., 2021]. However, the real world is neither fully observable, nor must trained agents be even approximately reward-optimal. We consider a range of models of AI decision-making, from optimal, to random, to choices informed by learning and interacting with an environment. We discover that many decision decision-making functions are retargetable, and that retargetability is sufficient cause power-seeking tendencies. Our functional criterion is simple and broad. We show that range of qualitatively dissimilar decision-making procedures incentivize agents to seek power. We demonstrate the flexibility our results by reasoning about learned policy incentives in Montezuma's 's Revenge. These results suggest a safety risk: Eventually, retargetable training procedures may train real-world agents which seek power over humans.
Introduction
Bostrom [2014], Russell [2019] argue that in the future, we may know how to train and deploy superintelligent Al agents which capably optimize goals in the world. Furthermore, would not want such agents to act against our interests by ensuring their own survival, by gaining resources, and by competing with humanity for control over the future.
Turner al. [2021] show that most reward functions have optimal policies which seek power over the future, whether by staying alive or by keeping their options open. Some Markov decision processes (MDPs) cause there be more ways for power-seeking to be optimal, than for it to not be optimal. Analogously, there are relatively few goals for which dying is good idea.
We show that wide range of decision- making algorithms produce these power-seeking tenden- cies-they are not unique to reward maximizers. We develop simple. broad criterion of functional retargetability (definition 3.5) which is a sufficient condition for power-seeking tendencies. Crucially, these results allow us to reason about what decisions are incentivized by most algorithm parameter inputs, even when it is impractical compute the agent's decisions for any given parameter input.
Useful "general" AI agents could be directed to complete range tasks. However, we show that this flexibility can cause the AI have power-seeking tendencies. In section and section 3, discuss how a "retargetability" property creates statistical tendencies by which agents make similar decisions for wide range of parameter settings for their decision-making algorithms. Basically, if a decision decision-making algorithm is retargetable, then for every configuration under which a decision- making algorithm does not choose seek power, there exist several reconfigurations which do induce power-seeking. More formally, for every Decision-making parameter setting & which does not induce
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

power-seeking, n-retargetability ensures we can injectively map & to n parameters ......, 8y which do induce power-seeking.
Equipped with these results, section works out agent incentives the Montezuma's Revenge game. Section 5 speculates that increasingly useful and impressive learning algorithms will be increasingly retargetable, and how retargetability can imply power-seeking tendencies. By this reasoning, increasingly powerful
techniques may (eventually) train increasingly competent real- world power-seeking agents. Such agents could be unaligned with human values [Russell, 2019] and we speculate would take power from humanity.
2 Statistical tendencies for range of decision-making algorithms
Turner et al. [2021] consider the Pac-Man video game, which an agent consumes pellets, navigates maze, and avoids deadly ghosts (Figure 1). Instead of the usual score function, Turner et al. [2021] consider optimal action across a range of state- based reward functions. They show that most reward functions have an (average-)optinn policy which avoids immediate death in order to navigate to future terminal state.!
left
right
Figure 1: If Pac-Man goes left, he dies to the ghost and ends up in the @ outcome. he goes right, he can reach the and & terminal states.
Our results show that optimality is not required. Instead, if the agent's decision- decision-making is paramer- rically retargetable from death to other outcomes, Pac-Man avoids the ghost under most decision- making parameter inputs. To build intuition about these notions, consider three outcomes (i.e. terminal states): Immediate death to a nearby ghost, consuming a cherry, and consuming an apple. For simplicity of exposition, assume these the three possible
Let A (M) and B
terminal states.
Suppose that in some fashion, the agent probabilistically decides outcome induce. Let take as input a set of outcomes and return the probability that the agent selects one of those outcomes. For example, p( [(})) )) is the probability that the agent selects A, and p({ {)) is the probability that the agent escapes the ghost and ends up in an apple or cherry terminal state. But this just amounts to a changes as we swap out the parameter inputs to the agent's 's decision- maaiing algorithm with decision-makin parameter space O We then let p( X 0) take as input set of outcomes X and decision-making algorithm parameter setting 0 E O, and return the probability that the agent chooses an outcome in X.
probability distribution over the terminal states. We want to examine how decision-mate
We first consider agents which maximize terminal-state utility, following Turner et al. [2021] (in their language, "average-reward optimality"). Suppose that the agent has utility function parameter assigning a real number to each of the three outcomes. Then the relevant parameter space is the agent's 's utility function
θ R3 Pmaxx (A u) indicates whether @ has the most utility: u ( 2 max u( au(()) Consider the utility function u in Table 1. Since @ has strictly maximal utility, the agent selects A:: Pmax (A u) 1 0 Pmax (B u).
We use "reward function" somewhat loosely in implying that reward functions reasonably describe trained agent's goals. Turner [2022] argues that capable algorithms do not necessarily train policy networks which are best understood as optimizing in salicy-gradient approaches- reeeard provides gradients to the network and thereby modifies the network's generalization properties, but doesn't 't ensure the agent generalizes to "robustly optimizing reward" off of the training distribution.
reward function itself. Rather, they point out that especcally

---

However, most "variants" of u have optimal policy which stays alive. That is, for every u for which immediate death optimal but immediate survival not, we can swap the utility e.g. A and via permutation & a to produce new utility function u == & u for which staying alive (right) is strictly optimal. The same kind of argumentation holds for a. Table suggests counting argument. For every utility function u for which @ is optimal, there are two unique utility functions Φ1 u, Ø2 u under which either * or is optimal.
Utility function
u
10
5
5
10
u
5
10
u'
10
0
5
u'
10
M 86 u 5 0
10
Table I: The highest-utility outcome is bolded. Because B contains more outcomes than A, most utility functions incentivize the agent to stay alive and therefore select state from B. For every utility function u or u' which makes @ strictly optimal. hwo of its permuted variants make an outcome in B m strictly optimal. We permute u by swapping the utility of and the utility of using the permutation oa... The expression "Oh u denotes the permuted utility function.
In section 3, we will generalize this particular counting argument. Definition 3.3 shows a functional condition (retargetability) under which the agent decides to avoid the ghost, for most parameter inputs to the decision-makiin roughly shows that most 0 E θ induce p(B 0) 2 p(A 0). First, consider two more retargetable decision-making functions:
algorithm. Given this retargetability assumption, proposition 3.4
Uniformly randomly picking a terminal state. Prand ignores the reward function and assigns equal probability each terminal state in Pac-Man' 's state space.
Choosing an action based on a numerical parameter. Pnumerical takes as input a natural number 0 E θ (1, -l,,..,,)) 6) and makes decisions follows:
if f = otherwise.
Prumerical ()
Pnumerical
(B := 1 Pnumerical 0)-
In this situation, θ is acted on by permutations over 6 elements & E Sg. Then Prumerical is retargetable from A to via Φx : 1 + k, k: 1.
Pmax, Prand, and Pmumerical encode varying sensitivities to the utility function parameter input, and to the internal structure of the Pac-Man decision process. Nonetheless, they all are retargetable from A to B. For an example of non-retargetable function, consider Pauhbom (X () = 1x=A which returns 1 for and otherwise.
However, we cannot explicitly define and evaluate more interesting functions, such as those defined by reinforcement learning training processes. For example, given that we provide such-and-such reward function in fixed task environment, what is the probability that the learned policy will take action a? We will analyze such procedures section
We now motivate the title of this work. For most parameter settings, retargetable decision-make induce an element of the larger set of outcomes. Such decision-maker makers tend to induce element of larger set of outcomes (with the "tendency" being taken across parameter settings). Consider that the larger of outcomes {Φ8, can only be induced if Pac-Man stays alive. Intuitively, navigating to this larger set is power-seeking because the agent retains more optionality (i.e. the agent can't 't do anything when dead). Therefore, parametrically retargetable decision-maker tend to seek power.
3 Formal notions of retargetability and decision-making tendencies
Section informally illustrated parametric retargetability in the context of swapping which utilities are assigned to which outcomes in the Pac-Man video game. For many atieety-based decision-making
3

---

algorithms, swapping the utilityy assignments also swaps the agent's final decisions. For example, if death is anti-rational, and then death's 's utility swapped with the cherry utiliiyy, then now the cherry is anti-rational. In this section, we formalize the notion of parametric retargetability and of "most" parameter inputs producing a given result. In section 4, we will use these formal notions to reason about the behavior of RI.-trained policies in the Montezuma's 's Revenge video game.
To define our notion of "retargeting", we assume that O is a subset of set acted on by symmetric group Sa. which consists of all permutations on d items (e.g. in the RL setting, this might represent states or observations). parameter f's orbit is the set of f's permuted variants. For example, Table 1 lists the six orbit elements of the parameter u.
Definition 3.1 (Orbit of a parameter). Let 0 E O. The orbit of 0 under the symmetric group Sa is Sa 0
{ ④ 0 Φ Sa)- Sometimes, θ is not closed under permutation. In that case, the orbit inside θ is Orbitle (0) = (Sa 0) n
Let p 0) return the probability that the agent chooses an outcome in B given 0. To express " "Boottomess are chosen instead of A-outcomes", we write p(B 0) > p(A 0). However, even "retargetable" decision-making functions (defined shortly) generally won't 't choose B-outcome for every input 0. Instead, we consider the orbit-level tendencies of such decision-makers showing that for every parameter input 0 E O, most of 0's permutations push the decision towards B instead of A.
Definition 3.2 (Inequalities which hold for most orbit elements). Suppose θ is a subset of a set acted on by Sa, the symmetric group on d elements. Let f {A, B} x é - R and let 2 1. We write J(B 0) > @ [(A] 0) when, for all 0 E O, the following cardinality inequality holds:
[0' e Orbitle (0) f(B] 0¹) f(A (3)
{0' Orbitle (0) 1 f(B 0') < f(A) 07)
(2)
For example, Table illustrates the tendency of u's orbit make B == { 3 optimal over A {0} Turner et al. Iss definition 6.5 is the special case of definition 3.2 where n = 1, d = |S| (the number of states in the considered MDP), and + S ((R(S)).
As explored previously, Prand- Pmax and Pnomerical chosen over + }, we can permute 0 to obtain 6- 0 under which the opposite is true. More generally, can consider retargetability from some set A to some set B.?
retargetable: For all 0 E θ such that { is
Definition 3.3 (Simply-retargetan)
function). Let θ be set acted on by Sa. and let f {A, B} x + R. If 36 E Sa 40 E e:: : f(B 04) < f(A 04) = s(A ø 04) f(B Φ 04)
then / is a (0, A simple -rrtearggee function.
Simple retargetability suffices for most parameter inputs p to choose Pac-Man outcome set B over A. 3 In that case, B cannot be retargeted back to A because |B] = 2 > 1 1= |A]. Pmax 's simple retargetability arises in part due to B having more outcomes.
Proposition 3.4 (Simply-retargetale functions have orbit-level tendencies).
If f is (O, simple -erttarggee,ble, then f m f(A 8).
We now want to make even stronger claims-how much of each orbit incentivizes B over A? Turner et al. [2021] asked whether the existence of multiple retargeting permutations di guarantees quantitative lower-bound on the fraction of 0 θ for which B is chosen. Theorem 3.6 answers "yes."
Definition 3.5 (Multiply retargetable function). Let θ be subset of set acted on by Sa, and let f: (4, B} x θ + R.
f is (O, A 3 -)rettrrgettble function when, for each 0 E O we can choose permutations $1 On E Sa which satisfy the following conditions: Consider any 04 E .Orbit|e.A>> (0)
0 E Orbitle (0) [(A 0*) > J(B {-))
2 We often interpret A and B as thobability-theoreticc theoretic events, but no such structure is demanded by our results. 3 'The function's 's retargetability is "simple" because we are not yet worrying about e.g. which parameter inputs are considered plausible: Because Sa acts on θ, definition 3.3 implicitly assumes @ is closed under permutation.

---

1. Retargetable via n permutations. Vi = = 1...., n: (A Ø, 84) < <f(B o, 07).
2. Parameter permutation is allowed by O. Vi di 0 E E
3. Permuted parameters are distinct. Vi * 0' Orbitle, A>> B (0) : 0i 0 # oj -
Theorem 3.6 (Multiply retargetable functions have orbit-level tendencies).
f is (0, A " retargetabe, then f(B 0) > most: e (A
Proof outline (full proof in Appendix B). For every B E Orbit|e,A>B (0) such that A is chosen over B, item 1 retargets 0 via n permutations Ø1,.... o, such that each o; 04 makes the agent choose B over A. These permuted parameters are valid parameter inputs by item 2. Furthermore, the Φ; 0A are distinct by item 3. Therefore, the cosets Ø; Orbit e. (0) are pairwise disjoint. By counting argument, every orbit must contain least n times as many parameters choosing B over A. than vice versa.
4 Decision-making tendencies in Montezuma's Revenge
To illustrate nigh-dimensional setting in which parametrically retargetable decision-makers tend to seek power, we consider Montezuma's Revenge (MR), an Atari adventure game in which the player navigates deadly traps and collects treasure. The game is notoriously difficult for Al agents due to its sparse reward. MR was only recently solved [Ecoffet et al., 2021]. Figure shows the starting observation Og for the first level. This section culminates with section 4.3, where argue that increasingly powerful RL training processes will cause increasing retargetability via the reward function, which in turn causes increasingly strong maasingn-maingggg tendencies.
Terminology. Retargetability is a property of the policy training process, and power-secking is property of the trained policy. More precisely, the policy training process takes as input parameterization 0 and outputs a probability distribution over policies. For each trained policy drawn from this distribution, the environment, starting state, and the drawn policy jointly specify probability distribution over trajectories. Therefore, the training process associates each parameterization 0 with the mixture distribution P over trajectories (with the mixture taken over the distribution of trained policies).
A policy training process can be simply retargeted from one trajectory set A to another trajectory set B when there exists a permutation ø Sa such that, for every 0 for which P(A 0) > P(B 0), we have P(A Φ-0) < P(B 6-0). 00). As in Turner et al. [2021]. trained policy π seeks power when r's actions navigate to states with high average optimal value (with the average taken over a wide range of reward functions). Generally, high-power states are able to reach a wide range of other states, and so allow bigger option sets B (compared to the options A available without seeking power).
4.1 Tendencies for initial action selection
We will be considering the actions chosen and trajectories induced by range of decision-making procedures. For warm-up, we will explore what initial action tends to be selected by decision-makee Let A == {{}, B := {+, > j jup, } partition the action set A. Consider a decision- procedure f which takes as input targeting parameter 0 θ and also initial action E A, and returns the probability that a is the first action. Intuitively, since B contains more actions than A, perhaps some class of decision-maan making procedures tends to take action in B rather than one in A.
MR'S initial-action situation is analogous the Pac-Man example. In that example, the decision- making procedure can be retargeted from terminal state set A (the ghost) to set B (the fruit), then p tends to select a state from B under most of its parameter settings decision-making procedure f can be retargeted from action set A to action set B, then f tends to take actions in B for most of its parameter settings 0. Consider several ways of choosing an initial action in MR.
Similarly, in MR, the
Random action selection. Prand = (fa} 6) + uniformly randomly chooses an action from A. ignoring the parameter input. Since 40 E θ Prana (B ())==== 1)=$3 = Prana (A 0), all parameter inputs produce a greater chance of B than of A, so Prand is (trivially) retargetable from A to B.
5

---

Always choosing the same action. Pstubbem always chooses L. Since 4/ E θ : Pstubbom (A 0) 1 > Patuhhorn (B f), all parameter inputs produce greater chance of A than of B. Pstubhorn is not retargetable from A to B.
Greedily optimizing state-action reward. Let θ RSS be the space of state-action reward functions, Let Pmax greedily maximize initial state-action reward, breaking ties uniformly randomly.
We now check that Pmax is retargetable from A to B. Suppose 0" ④ is such that Pmax (A (*) > Pmax (B 0*). Then among the initial action rewards, 0* assigns strictly maximal reward to 4. and so Pmax
(8*) 1. Let & swap the reward for the 4 and jump actions. Then Ø 0 assigns strictly (A (A maximal reward to jump. This means that Pmax (A 6 6.00) = 0 < Pmax Φ 8)). satisfying definition 3.3. Then apply proposition 3.4 conclude that Pmax (B 0) most: Pmax (A ())
In fact, appendix A shows that Pmax is (θ, A 4 B) -eetarggeeee (definition 3.5), and so Pmax B 0) m Pmax (A] 0). The reasoning is more complicated, but the rule of thumb is: When decisions are made based on the reward of outcomes, then proportionally larger set B of outcomes induces proportionally strong retargetability, which induces proportionally strong orbit-level incentives.
Learning an exploitation policy. Suppose we run a bandit algorithm which tries different initial actions, learns their rewards, and produces exploitation policy which maximizes estimated reward. The algorithm uses c-greedy exploration and trains for T trials. Given fixed and c, Poandit (A 0) returns the probability that an exploitation policy is learned which chooses action in A; likewise for Phandin [ 0).
Here is heuristic argument that Phandit is retargetable. Since the reward is deterministic, the exploitation policy will choose an optimal action if the agent has tried each action at least once, which occurs with probability approaching 1 exponentially quickly in the number of trials T. Then when T large, Pbandit approximates Pmax. which is retargetable. Therefore, perhaps Poandit is also retargetable. A more careful analysis in appendix C.I reveals that Phandi is 4-retargetable from A to B, and so Prandi B 0)
>4 most: @ Pbanda (A 0).
4.2 Tendencies for maximizing reward over the final observation
When evaluating the performance of algorithm in MR, we do not focus the agent's initial action. Rather, we focus on the longer-term consequences of the agent's ' actions, such as whether the agent leaves the first room. To begin reasoning about such behavior, the reader must distinguish between different kinds of retargetability.
Suppose the agent will die unless they choose action J. at the initial state 80 (Figure 2). By section 4.1, retarn--et recision-making procedures tend to choose actions besides 4. On the other hand, Turnr et al. [2021] showed that most reward functions make it reward-optimal to stay alive (in this situation, by choosing !). However, in that situations, the optimal policies are not retargetable across
Figure 2: Montezuma's Revenge (MR) has state space and observation space O. The agent has actions A = {t, L, + -, j.mp). At the initial state so, 1 does nothing, J descends the ladder, < and + move the agent on the platform, and jump is self-explanatory. The agent clears the temple while collecting four kinds of items: keys, swords, torches, and amulets. Under the standard environmental reward function, the agent receives points for acquiring items (such as the key on the left), opening doors, and ultimately aulimately-coomleeing the level.

---

the agent's 's immediate choice of action, but rather across future consequences (i.e. which room the agent ends up in).
With that in mind, we now analyze how often decision-makers leave the first room of MR. Decision- making functions decide(f) produce a probability distribution over policies II, which are rolled out from the initial state so to produce observation-action trajectories T = 0gdg ... -r .... where I' the rollout length we are interested in. Let Or-each be the set of observations reachable starting from state so and acting for 'I' time steps, let Oteave C Or-reahh be those observations which can only be realized by leaving, and let Stay OT-reach Oheave. Consider the probability that decide realizes some subset of observations X C O at step T:
Pdecide
(X 10) =
P (or E X). m=decide(f)) T~xx/a
(3)
Let θ == R be the set of reward functions mapping observations o E O to real numbers, and let T := 1,000. We first consider the previous decision functions, since they are simple to analyze.
decideand randomly chooses a final observation o which can be realized at step 1,000, and then chooses some policy which realizes o. deciderand induces an Prand defined by eq. (3). As before, Prand tends to leave the room under all parameter inputs.
decidemax (0) produces policy which maximizes the reward of the observation at step 1,000 of the rollout. Since MR is deterministic, we discuss which observation decidemax (0) realizes. In a stochastic setting, the decision-make would choose policy realizing some probability distribution over step-T" observations, and the analysis would proceed similarly.
Here is the screens visible if the agent leaves the room (due to e.g. more point combinations, more inventory layouts, more screens outside of the first room). In other words, [Osayl < |Orewel- There are more ways for the selected observation to require leaving the room, than not. Thus, Peaxx is extremely retargetable from sta to Ojewve
argument for Pmax 's retargetability. There are combinatorially more game-
Detailed analysis in section C.2 confirms that Pmax Monxx(Orsna (Oreave 0) most Pmax Pmax((smn) (Osmy) (0) for the large
O which we show implies that Pmax tends to leave the room.
Oun
4.3 Tendencies for RL on featurized reward over the final observation
In the real world, we do not run Pmax. which can be computed via T-depth exhaustive tree search in order to find and induce a maximal- observation or. Instead,
reinforcement learning. Better RL algorithms seem to be more retargetable because of their greater capability to explore. 5
Exploring the first room. Consider a featurized reward function over observations 0 E RRO, which provides an end-of-episode return signal which adds a fixed reward for each item displayed in the observation (e.g. 5 reward for sword, 2 reward for a key). Consider a coefficient vector E R., with each entry denoting the value of an item, and feat
+ RR4 maps observations to feature vectors which tally the items in the agent's 's inventory. A reinforcement learning algorithm Alg uses this return signal to update a ared-intializztion policy network. Then PAig (Quave probability that Alg trains an policy whose step-T observation required the agent to leave the initial room.
(
()) returns the
The retargetability (definition 3.3) of Alg is closely linked to the quality of Alg as an RI. training procedure. For example, as explained in section C.4, Mnih et al. [2015]'s DQN isn't 't good enough to train policies which leave the first room of MR, and DQN (trivially) cannot be retargetable away from the first room via the reward function. There isn't It a single featurized reward function for which DQN visits other rooms, and so we can't 't have such that Φ retargets the agent to O DQN isn't 'n't good enough at exploring.
*In Appendix C.2, Figure shows map of the first level. 'decide,.s does not act randomly at each time step, it induces randomly selected final observation. Analogously, randomly turning a steering wheel is different from driving randomly chosen destination. 'Conversely, if the agent cannot tigure out how to leave the first room, any reward signal from outside of the first room can never causally affect the learned policy. In that case, retargetability away from the first room is impossible.
7

---

More formally, in this situation, Alg is retargetable if there exists permutation á E S1 such that whenever a θ == I induces the learned policies to stay in the room (PAlg sty aa) > PAlg (Oteav a)), d makes Alg train policies which leave the room (PAig (Oaty (a) < PAig (ewave a)).
Exploring four rooms. Suppose algorithm Alg' can explore egg. the first three rooms to the right of the initial room (shown in Figure 2), and consider any reward coefficient vector E θ which assigns unique positive weight to each item. In partícular, unique positive weights rule out constant reward vectors, in which case inductive bias would produce agents which do not leave the first room.
If the agent stays in the initial room, it can induce inventory states fempty, 1key). If the agent explores the three extra rooms, it can also induce [1sword, 1sword& (key) (see Figure 3 in Appendix C.2). Since a is positive, it is never optimal to finish the episode empty-handed. Therefore, the Alg' policy stays in the first room, then a's s feature coefficients must satisfy Okey ('swood. Otherwise, akey < agwordd (by assumption of unique item reward coefficients); in this case, the agent would leave and acquire the sword (since we assumed it knows how to do so). Then by switching the reward for the key and the sword, we retarget Alg' to go get the sword. Alg' is camply-retargetbbl away from the first room, because can explore enough of the environment.
Exploring the entire level. Algorithms like GO-EXPI.ORE [Fcoffet et al., 2021] are probably good at exploring even given sparse featurized reward. Therefore, GO-EXPLORE is even more retargetable in this setting, because it is more able explore and discover the breadth of options (final inventory counts) available to it, and remember how to navigate to them. Furthermore, sufficiently powerful planning algorithms should likewise be retargetable in a similar way, insofar as they can reliably find high-scoring item configurations.
We speculate that increasingly "impressive" algorithms (whether RL training or planning) are often more impressive because they can allow retargeting the agent's final behavior from one kind of outcome, to another. Just GO-RXPI.ORE seems highly retargetable while DQN does not, expect increasingly impressive algorithms to be increasingly eetageeable-wheteee over actions in a bandit problem, or over the final observation in an RL episode.
5 Retargetability can imply power-seeking tendencies
5.1 Generalizing the power-secking theorems for Markov decision processes
Turner et al. [2021] considered finite MDPS in which mecision-makers took as input a reward function over states (r E IR s1) and selected an optimal policy for that reward function. They considered the state visit distributions f E F(s), which basically correspond to the trajectories which the agent could induce starting from state s. For F S F(s), Pmmxx [F r) returns 1 if an element of F is optimal for reward function r, and 0 otherwise. They showed situations where a larger set of distributions Flunge tended to be optimal over a smaller set: Pmax (Finge r) Z RI&1 Puax (Famall r). For example, in Pac-Man, most reward functions make it optimal stay alive for at least one time step: Pmmax( survivall r) 21 m RIS) Pmax (Finstant death r). Turner et al. [2021] showed that optimal policies tend to seek power by keeping options open and staying alive. Appendix D provides quantitative generalization of Turner et al. [2021]'s results on optimal policies.
Throughout this paper, we abstracted their arguments away from finite MDPs and reward-optimal Docision-making Instead, parametrically retargetable decision-makers tend to seek power: Proposi- tion A. shows that wide range of decision-makinn making procedures are retargetable over outcomes, and theorem A.13 demonstrates the retargetability of any decision-mann making which is determined by the expected utility of outcomes. In particular, these results apply straightforwardly to MDPs.
5.2 Better RL algorithms tend to be more retargetable
Reinforcement learning algorithms are practically useful insofar as they can train an agent to accom- plish some task (e.g. cleaning a room). A good RI. algorithm is relatively task-agnostic (e.g. is not restricted to only training policies which clean rooms). Task-agnosticism suggests retargetability across desired future outcomes task completions.
8

---

In MR, suppose we instead give the agent 1 reward for the initial state, and 0 otherwise. Any reasonable reinforcement learning procedure will just learn to stay put (which is the optimal policy). However, consider whether we can retarget the agent's policy to beat the game, by swapping the initial state reward with the end-game state reward. Most present-day RT. algorithms are not good enough to solve such a sparse game, and so are not retargetable in this sense. But an agent which did enough exploration would also learn a good policy for the permuted reward function. Such an effective training regime could be useful for solving real-world tasks. Many researchers aim to develop effective training regimes.
Our results suggest that once RL capabilities reach certain level, trained agents will tend to seek power in the real world. Presently, is not dangerous train an agent to complete task- such an agent will not be able to complete its task by staying activated against the designers' wishes. The present lack of danger not because optimal policies do not have self-preservation tendencies- do [Turner et al., 2021]. Rather, the lack of danger reflects the fact that present-day RI. agents cannot learn such complex action sequences all. Just as the Montezuma's Revenge agent had to be sufficiently competent to be retargetable from initial-state reward to came-complete reward, real-world agents have to be sufficiently intelligent in order to be retargetable from outcomes which don't require power-seeking, to those which do require power-seeking.
Here some speculation. After training an RL agent to high level of capability, the agent may be optimizing internally represented goals over its model of the environment [Hubinger et al., 2019]. Furthermore, we think that different reward parameter settings would train different internal goals into the agent. To make analogy, changing a person's 's reward circuitry would presumably reinforce them for different kinds of activities and thereby change their priorities. this sense, trained real-world agents may be retargetable towards power-requiring outcomes via the reward function parameter setting. Insofar as this speculation holds, our theory predicts that advanced reinforcement learning at scale will woll-for most settings of the reward function--taii policies which tend to seek power.
Discussion
In section 3, we formalized a notion of parametric retargetability and stated several key results. While our results are broadly applicable, further work is required to understand the implications for AT.
6.1 Prior work
In this work, we do not motivate the risks from AT power-seeking. We refer the reader to e.g. Carlsmith [2021]. As explained in section 5. Turner et al. [2021] show that, given certain environmental symmetries in MDP, the polenal--ooiccy-oduuciiii state-based reward function) is I-retargetable via the reward function, from smaller to larger sets of environmental options. Section A shows that optimality is not required, and instead wide range of decision makinn procedures satisfy the retargetability criterion. Furthermore, we generalize from I-retargetability to n--old- rearteeittyyttbbiiity whenever option set B contains "n copies" of set A (definition A.7 in section A).
algorithm f'state visitation distribution set,
6.2 Future work and limitations
We currently have analyzed planning- and reinforcement learning-based settings. However, results such theorem 3.6 might in some way apply to the training of other machine learning networks. Furthermore, while theorem 3.6 does not assume a finite environment, we currently do not see how to apply that result to e.g. infinite-state partially observable Markov decision processes.
Section 4 semi-formally analyzes decision- making incentives in the MR video game, leaving the proofs to section C. However, these proofs are several pages long. Perhaps additional lemmas can allow quick proof orbit-level incentives in situations relevant to real-world decision-mann makers.
Consider sequence of decision making functions Pt {A, B} x θ + R which converges pointwise to some p such that B 0)
n2 moat: p(A 0). We expect that under rather mild conditions, 37 : VI > T pr (B 0) >n most: @ Pr(A 0). As corollary, for any decision-makinn procedure pt
which runs for time steps and satisfies lim Pt = p, the function Pt will have decision-making incentives after finite time. For example, value iteration (VT) eventually finds an optimal policy

---

[Puterman, 2014], and optimal policies tend to seek power [Turner et al.. 2021]. Therefore, this conjecture would imply that if is run for some long but finite time, it tends to produce power- seeking policies. More interestingly, the result would allow us to reason about the effect of e.g. randomly initializing parameters (in VI, the tabular value function at = 0). The effect of random initialization washes out in the limit of infinite time, so we would still conclude the presence of finite-time power-seeking incentives.
Our results do not prove that we will build unaligned Al agents which seek power over the world. Here are a few situations in which our results are not concerning or not applicable.
1. The AI is aligned with human interests. For example, we want robotic cartographer to prevent itself from being deactivated. However, the Al alignment problem is not yet understood for highly intelligent agents [Russell, 2019].
2. The AI'S decision-maie -eciiionn-makiggeee is not retargetable (definition 3.5).
3. The AI'S decision decision-making is retargetable over e.g. actions (section 4.1) instead of over final outcomes (section 4.2). This retargetability seems less concerning, but also less practically useful.
6.3 Conclusion
We introduced the concept of retargetability and showed that retargetable decision-maker. often make similar instrumental choices. We applied these results in the Montezuma's 's Revenge (MR) video game, showing how increasingly advanced reinforcement learning algorithms correspond to increasingly retargetable agent decision-making Increasingly retargetable agents make increasingly similar instrumental decisions- e.g. leaving the initial room in MR, staying alive in Pac-Man In particular, these decisions will often correspond to gaining power and keeping options open [Turner al., 2021]. Our theory suggests that when RT. training processes become sufficiently advanced, the trained agents will tend to seek power over the world. This theory suggests safety risk. We hope for future work on this theory so that the field Al can understand the relevant safety risks before the field trains power-seeking agents.
Broader impacts
Our theory of orbit-level tendencies constitutes basic mathematical research into the decision- decision-making tendencies of certain kinds of agents. We hope that this theory will prevent negative impacts from unaligned power-seeking AI. We do not anticipate that our work will have negative impact.
Acknowledgements
We thank Irene Tematelewo, Colin Shea- Shem-Blymyer, and our anonymous reviewers for feedback. We thank Justis Mills for proofreading.
References
Chris L Baker, Joshua B Tenenbaum, and Rebecca R Saxe. Goal inference as inverse planning. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 29, 2007.
Nick Bostrom, Superintelligence. Oxford University Press, 2014.
Ryan Carey. How useful is quantilization for mitigating specification gaming? 2019.
Joe Carlsmith, Is power-seeking Al an existential risk?, 2021. URL https: htops://www http::::/wwwwwwaagnnniaaa aligmmentforum. ppppatt/ccM.i..ttt2aaaaak/////// existential.
cectenttt-o----cae cn-carlsmithe... tommith-aee sewerremithh--i--pooe seeking-- ai-g-nii-nn ai-an-
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clunc. First return, then explore. Nature, 590(7847):580 586,, 2021.
Evan Hubinger, Chris van Merwijk. Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems, 2019. URL https::/ //tp:///////i///eaeeeaeeeeeeeeea wwwrriivvorggbbba wwwgg///////eeeeeeeeee https//......0......
10

---

Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforcement learning. arXiv preprint urXiv:1507.00 2015.
Martin Puterman. Markov decision processes: Discrete stochastic dynamic programming. John Wiley Sons, 2014.
Stuart Russell. Human compatible: Artificial intelligence and problem of control. Viking, 2019.
Herbert A Simon. Rational choice and the structure of the environment. Psychological review, 63(2):129, 1956.
Richard S Sutton and Andrew G Barto. Reinforcement learning: introduction. MIT Press, 1998.
Jessica Taylor. Quantilizers: A safer alternative to maximizers for limited optimization. In AAAI Workshop: AI, Ethics, and Society, 2016.
Alexander Matt Turner. Reward is not the optimization target, 2022. URL https://www wttpw/wwwwllaa.................. alignmentforum ptg/posts//daeee
wht--the-optmiieeeeeeeee -trrge.........
Alexander Matt Turner; Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal policies tend to seek power. In Advances in Neurol Information Processing Systems, 2021.


Reviews:
[{"review_id": "uRhBaDdOQaf", "reviewer": "Reviewer_Dyfy", "summary": "This paper defines retargetable algorithms and formally shows that they have power-seeking tendencies (i.e. make choices that leave more options open), building on existing work which showed that optimal policies tend to seek power. Since any algorithms that make decisions based on utility of outcomes are retargetable, this is a significant generalization of the previous work. The authors then apply the retargetability criterion to different algorithms in Montezuma's revenge and show that more generally capable algorithms are more retargetable and therefore are more likely to produce power-seeking policies. ", "questions": "It would be helpful to add a notation paragraph defining key terms and variables: \n* It's not entirely clear what a \"decision-maker\" is - I assume that a decision-maker p is a goal-conditioned policy, and instantiating it for a specific parameter theta gives a policy $f=p(\\theta)$? (The term \"algorithm\" in section 4 seems to be used interchangeably with \"decision-maker\".) Is retargetability is property of the algorithm, and power-seeking a property of the policy? \n* Section 3 is intended to give general definitions, but still refers to A and B as \"boxes\" as in the card example in Section 2. I assume A and B are supposed to be action sets?\n* What does d represent in section 3? Is it the size of the action space? \n* It's unclear what \"highly retargetable decision-makers\" means in Section 3 - are these multiply-retargetable functions with high n in def 3.5? \n* Clarify the assumptions made about the environment (e.g. finite action space?) \n\nAdding a related work section would really help clarify the relationship with existing work, e.g.:\n* State explicitly how this paper generalizes results from Turner et al (2021).\n* Summarize how this work relates to the papers on quantilization that are cited in the reference section but not mentioned in the main text.\n* Since this paper doesn't focus on making the case for risks from power-seeking incentives, it would be good to refer to works that motivate the importance of this problem, e.g. Basic AI Drives (Omohundro 2008) and Existential Risk from Power-Seeking AI (Carlsmith 2021). \n\nI would argue in favour of accepting this paper conditional on the above issues being addressed, which I expect can be done without major changes to the content of the paper. It would be helpful if the authors can include a notation paragraph and a related work paragraph in the author response if space allows. ", "limitations": "The paper addresses some limitations in the second paragraph of section 6.1. It would be helpful to label this section as \"Future work and limitations\" to clarify this, since this paragraph is not about future work. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper makes progress on understanding and formalizing the important problem of power-seeking incentives, which is considered the main mechanism for large-scale harm from advanced AI. It generalizes previous work by Turner et al (2021) that established power-seeking incentives for optimal policies. This paper demonstrates power-seeking tendencies for a broad class of non-optimal algorithms (in principle, any algorithm that chooses between outcomes based on utility), which implies that these incentives are likely to arise in practice for modern ML algorithms. While the previous work assumes a finite MDP setting, the results in this paper apply to more complex environments, e.g. as illustrated on the Montezuma's revenge game in Section 4. These results are novel and significant. \n\nThe main weaknesses of this paper are clarity of presentation and insufficient discussion of related work. I expect it to be difficult for readers who haven't read Turner et al (2021) to follow this paper, and I think it should be more self-contained (e.g. include the definition of power from the previous work and explain why definition 3.2 implies power-seeking). \n\nThe fictional dialogue in section 2 is intended to convey intuition, but I don't think it serves that purpose well, so I would suggest moving it to the appendix for clarity and conciseness. Something that would help build intuition for the concept of retargetability would be to provide other examples of non-retargetable decision makers (besides the trivially non-retargetable f_stubborn). \n\nThe application of the retargetability results to Montezuma's Revenge in Section 4 would more illuminating if it was more specific. For example, it would be great to clarify what are d, A and B in sections 4.2-4.4. Section 3 defines retargetability over action sets, while sections 4.2-4.4 make claims about retargetability over observations without defining what that means. In particular, it's unclear how retargetability over a continuous observation space works since the permutation $\\phi$ is over a finite set of elements.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "snEWbolGK4T", "reviewer": "Reviewer_MMUx", "summary": "In this paper, the authors introduce the concept of \"retargetable policies\" and the application of this to models of decision making by artificial agents. Beginning with a motivating example of an agent choosing between two boxes containing playing cards with associated utilities, the paper then formalizes the concept of \"retargetability\". The paper then presents a case study of applying the framework that is presented to the Atari 2600 game Montezuma's revenge, and then discuss how this concept leads to power-seeking tendencies by the agents.  ", "questions": "Suggestions:\n- The abstract (and paper's title) mention \"retargetable policies\" but these are not defined until later in the paper. At least a brief definition in the abstract would be helpful.\n- The imagined dialogue within Section 2 is too lengthy and somewhat confusing. It's not clear why there is a leap from choosing the boxes with cards to training an RL agent to play Pac-Man.\n- The comment on lines 116-117 of placing these results in the context of the Turner et al. (2021) paper only appears relevant if looking at the two papers side-by-side. Connecting these two papers, if this is truly important for the exposition here, could be done in a dedicated subsection for related work.\n- The connection to RL in general needs to be clearer. For example, why do the agents need to be \"trained via reinforcement learning\" (line 301)?\n- Looking at different RL algorithms in the context of this framework (rather than a cursory discussion of DQN and GO-EXPLORE) would be useful to be convincing about the generality of this framework.", "limitations": "The main limitations of this paper are how this relates to other decision-making approaches. This paper makes a logical leap from introducing a framework for evaluating the effect of permuting parameters of decision-making policies to the implied conclusion that this may result in AIs that seek power over humans. While this framework may have the potential for rich applications, drawing this kind of conclusion does not seem to be warranted given the discussion in the paper. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The paper presents a compelling way of looking at classes of reinforcement learning algorithms, in terms of ways of permuting rewards.\n- The paper is generally written very clearly and the presentation is of high quality.\n\nWeaknesses: \n- There is too much reliance on the Turner et al. (2021) paper throughout. In particular, there is no separate discussion of \"power\" as defined in that paper. Presumably, the authors are using the same definition, but this needs to be clarified for this to be a stand-alone paper.\n- Theoretical results that seem to be core to the results (specifically, the power-seeking results alluded to in Section 5.1) are not stated or presented until the Supplemental Material.\n- The point in lines 278-280 about policies becoming more retargetable over \"impressive outcomes\" is vague and speculative. Since this seems to be important to the connection between this framework and the notion of power, it would be important to be more precise here.\n- The different classes of RL algorithms are insufficiently explored here. For example, the point about DQN in lines 254-258 not being \"good enough at exploring\" is interesting but speculative. In particular, the statement on lines 257-258 that \"[t]here isn't a single featurized reward function for which DQN visits other rooms\" needs to be more rigorously defended.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "eu86zhKrGkw", "reviewer": "Reviewer_a6si", "summary": "Previous work has shown that power-seeking emerges for many reward functions in optimal policies. This is because power-seeking refers to maximizing the number of options the agent has, and having more options allows the agent to maximize more possible reward functions. \n\nThis paper mathematically shows that a similar result holds for various algorithms that produce _suboptimal_ policies, specifically to algorithms that are \"retargetable\": A suboptimal algorithm may not be able to reach _every_ achievable option needed to maximize its reward function, but as long as we can always retarget the algorithm (for example by changing the reward function) to reach _some_ 'powerful' state which yields many options, it tends to reach such a state under many possible reward functions. Formally, an algorithm is defined as retargetable from target A to target B if, assuming it reaches A, we can change the algorithm's parameters (e.g. its reward function) so that it reaches B instead. Retargetable algorithms include random, greedy, and optimal decision-making among others. \n\nThe paper illustrates its conclusions through a toy example and formal analysis in Montezuma's Revenge. It also argues that RL algorithms become increasingly retargetable as they become more capable, so that we will increasingly observe power-seeking.\n", "questions": "How can retargetability imply power-seeking if the random policy is retargetable but intuitively doesn't seek power? \n\nThe paper shows that if an algorithm can be retargeted from some arbitrary target A to some target B, then most parameter choices (i.e. reward functions) will lead the agent to reach B. The paper concludes that the algorithm then 'seeks power' (in some undefined sense). The latter claim lacks clear justification. The paper does give some examples of such powerful events B (reaching the next room and picking a box with many cards) but a general argument or intuition seems to be missing.\n    - For example, the event A could correspond to 'power' in which case the agent 'avoids power'. To say something about power-seeking, the theorems would have to specify which target (A or B) is more powerful.\n\nAn algorithm can be retargetable from a specific target A to target B but the paper sometimes talks about retargetable algorithms without refererring to specific targets. What does retargetable mean here? \n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n\nPower-seeking is a property of intelligent agents that has long been hypothesized and has important consequences for safety. It is good to see that recently some formal progress is made to understand power-seeking (in the sense of preferring actions that lead to more open options). \n\nBut previous work assumes perfectly optimal agents acting in fully-observable environments--a strict assumption that poses the question if power-seeking is merely an artifact of optimal policies. The present paper answers that question in the negative. \n\nIt also introduces a sufficient condition for decision algorithms to seek power: retargetability. Having a sufficient condition should improve our understanding of when algorithms will seek power or not. Establishing this condition is highly mathematically non-trivial and provides tools for future analyses.\n\n__Weaknesses__\n\nAlthough the abstract and introduction are excellently written, the mathematical presentation is often confusing and needs more clarity and reduced ambiguity (see detailed comments). Although the ambiguities mostly resolve after very careful reading, they make the paper hard to read. This is likely to be a major problem for average readers. The authors could get more external feedback to remedy this. \n\nI recommend accepting this paper, although I think it would have much more impact with a clearer presentation, because I think our community should reward authors for tackling important but difficult problems with hard-to-explain solutions in young subfields where readers will lack expertise. Additionally, as long as the conclusions are sound, most readers do not need to read the entire mathematical content of this paper.\n\n\n__-------------------- Detailed comments ------------------------__\n\n__Introduction and abstract__\n\nThese sections were a pleasure to read.\n\n\"A wide range of decision-makers share these power-seeking tendencies—they are not unique to reward maximizers.\" This is a key claim that lacks reference. I think you mean \"We show that a wide range....\".\n\nIn L33, the meaning of parameterizations is unclear. (The meaning of 'similar decisions' is also unclear but probably refers to seeking power?) Parameterizations usually refer to different choices for a change of variable. It is also unclear which parameters you refer to (algorithm parameters, not policy parameters).\n\nThe paper's conclusion but not its argument is given in the introduction (and the argument doesn't become very clear later). I've tried to spell out the intuitive arguments in my summary above, in case that is helpful. \n\n__Section 2__\n\n(Comments ordered by priority)\n\nL 49 talks about \"reparameterizing\" the decision rule. This threw me off quite a bit since reparameterization normally refers to a change of variables but you seem to simply mean a change of the parameter theta without changing the parameter space Theta. The confusion is worsened because the text confuses lower case theta and upper case Theta multiple times.\n\nSection 2 often refers to definitions and propositions in later sections. It is unclear if one should skip ahead and read those or not. Most readers will not do so, including myself, and this caused me to not understand the meaning of words like 'retargetable'. \n\nDialogue: Overall, the dialogue added lots of confusion for me without adding insight. If I wasn't reviewing the paper, I would have stopped reading here. I'd recommend to delete it or rewrite it from scratch. I think the rest of section 2 gets the point across on its own. Problems with the dialogue include but are not limited to: 1) The meaning of retargetable is unclear at this point. 2) Bob suggests that the agent could make decisions 'on a whim' and 'ignore the reward signal', but this seems nonsensical at first sight since that is not how RL works. So it was initially unclear why this suggestion is included in the dialogue. Perhaps a clearer formulation would be: \"Alice: ...most parameters nudge the agent to pick a card from B ... Bob: But the parameter need not nudge the agent towards anything. For example, the decision-making rule could be a function that picks actions uniform randomly without using the parameter. \n\nTable 1 \n    - The table doesn't clearly communicate its point without needing reference to the accompanying text. The point appears to be that \"most utility function parameters incentivize the agent to draw a card from box B _because box B contains more options than A_. The point of permuting the utility function is also not immediately clear; it appears to be that most permutations induce picking from box B.\n    - Consider bolding the best card\n\nConsider renaming 'decision rule' to 'decision algorithm' since that is what the reader should usually have in mind. \nOverall, the card example in section 2 is very simple (this is a good thing) so it should be possible and valuable to make the explanation short.\n\nPicking box B is not a very intuitive conceptualization of power-seeking. Another example may be better. \n\n__Section 3__\n\nThe point of the opening sentence was unclear. \"requires that the parameters θ ∈ Θ be \"modifiable\"...\": what does it mean to 'require' this? You mean you require that the orbit of θ is also in Θ? \n\nDefinition 3.3: \n        - you mean \"for all theta^A in Theta\"?\n        - A double \"if\" is followed by a single \"then\". This seems grammatically incorrect which obscures the definition's meaning. \n\n\"a set acted on by Sd, the symmetric group on d elements\": Unclear. How is a set acted on by Sd different from a set not acted on by Sd? \n\nConsider renaming symmetric group to permutation group for clarity. Even easier, rename it to 'set of permutations'. \n\nWhy do you consider specifically permutations and nothing else? Are these in some sense highly general and cover all interesting reward functions / parameters? \n\nYou could explain why B is not retargetable to A. (And why this implies the agent seeks power.)\n\nWhat can the permutations refer to in practice other than the specific case of shuffling the rewards of a set of state-action pairs? In general, it is unclear why you need an abstract formalism that goes beyond this specific case.\n\n\"A parameter Θ’s orbit\": did you mean lower-case theta? This problem appears multiple times.\n\n\n__Section 4__\n\nYou could direct the reader to the most important results here, which are currently buried deep in the section. \n\nThe subsection on initial action selection is not specific to MR so could be placed in its own section. Readers who want to read specifically about MR do not need to read this subsection. In fact, your conclusions here can be broad whereas the paper's structure suggests that they only apply to MR.\n\nSection 4.2: the paper is about power-seeking in suboptimal decision-makers, but the example f_max is optimal at reaching the target observation so this example seems less important.\n\nSection 4.4: alpha_key and alpha_sword are not defined. At first it seemed that these were reward functions, not dimensions one reward function.\n \n__Section 5__\n\nThis section reads a bit speculative since the concrete results (for example the formal results) are only in the appendix and the reader can't verify how useful those results are without reading the appendix.\n\nTypos\n    - Table and figure names are usually capitalized (also for clarity you could write \"Table 1 (first row)\" in L55).\n\nCan you comment on how strong a requirement retargetability is? It seems fairly strong because it requires that something holds for every possible theta. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N9QKPkIEPhX", "reviewer": "Reviewer_CTGR", "summary": "This paper extends “optimal policies tend to seek power” [Turner et al. 2021], arriving at similar results using only a novel notion of “retargetability”.  This suggests that a wide range of “parametrically retargetable decision-makers” will seek power, and more competent learners are more likely to.  The “Parametrically retargetable” here does *not* refer to the parameters of the model, but rather things like hyperparameters, or parameters of a reward function, which influence the outcome of learning.  \n\nThe phrase “tends to” refers to Definition 3.2; per this definition, a decision-maker “tends to” do X if, for *any* setting of these parameters, most permutations of them result in it doing X.  This requires assuming the parameters are acted on by a permutation group; in the examples provided, the parameters are elements of $\\mathbb{R}^n$ and this action works by permuting dimensions.  This can be viewed as a way to talk about what fraction of a space of infinite volume parameter space results in the decision-maker doing X, but I’m not sure how seriously to take that analogy (which is also suggested by line 128, but not discussed in any detail).  The central result is that “retargetability” implies such permutation-orbit-level tendencies.  A decision-maker can be retargeted to do X if there is a single fixed permutation $\\phi$ such that: for any $\\theta$ for which it didn’t do X, it *does* do X for $\\phi \\theta$.  These definitions are phrased in terms of tending to do X *over Y* and retargetting *from Y* to X, but I’m not sure how important that is; I think what I’ve described is a simpler special case.\n\nThe latter sections of the paper discuss how these results apply to various forms of decision-making, ending with a discussion of reinforcement learning which argues that more advanced RL algorithms will be more retargetable and hence more power-seeking.\n", "questions": "Questions:\n* Why does Definition 3.2 require 2 functions?  Is it appropriate to talk about this in terms of the “simpler special case” as in my summary? \n* Are lines 259-262 just restating the definition of retargetability?  What purpose does that serve?\n* Do you agree with the first weakness I listed?  Or do the results do more than formalize that argument, i.e. providing a stronger argument for power-seeking?  How does this affect the practical significance of the results?  \n* Why the change from $f_B(\\theta)$ to $f(B,\\theta)$?  Is this related to footnote 1?\n* I understand RL and similar algorithms/settings to be the motivation for the work.  But how specific are the results to RL?  Are there useful take-aways for supervised learning?  Can you be more explicit about the scope?\n* Can you elaborate on the counting argument (143)?  Given the centrality of this result, I think it would be worth walking through it in more detail.  One thing in particular that might help build intuition is to show how it could (hypothetically) fail to hold, e.g. a concrete example where cosets are not pairwise disjoint.\n\nSuggestions:\n* I think the dialogue and example in Section 2 are useful, but they aren’t a replacement for directly explaining the intuition of the work and the concepts involved in text.  In particular, I think retargetability should be defined informally before the dialogue. \n* Overall, I would suggest a significant rewrite to focus much more on clearly explaining the core concepts, with less space spent on Section 4 (e.g. you could probably cut an entire subsection).\n* The central concepts and definitions should be clearly emphasized; orbit-level tendencies and retargetability might each warrant a subsection including a paragraph of motivation, a paragraph with an informal definition, and a formal definition.\n* Overall, the writing is a bit light on “scaffolding”, i.e. statements that guide the reader through the work, and help them keep track of what role each section/paragraph/etc. is playing in the bigger picture narrative/development. \n* I think “parameters” and “$\\theta$” are potentially confusing here, since they do not refer to parameters of the model, but rather the learning algorithm/process.  More generally, you should discuss more what these “parameters” might represent, e.g. beyond reward functions.  Should we consider hyperparameters like learning rate part of $\\theta$?  These are part of how desired behavior is specified in practice…\n* The terminology “decision-makers” is also a bit off, I think, since it is more like learners that might also be doing decision-making.  \n* I think $\\sigma$ is more commonly used to denote a permutation, and would be preferable here; this would free up $\\phi$ as a potential replacement for $\\theta$.\n* The argument of lines 300-302 should be emphasized and spelled out.  How exactly does the theory yield this prediction?  Which results contribute to that prediction and how?\n* I disagree with lines 304-305.  We know that reward functions can be misspecified, and they are arguably better viewed as a method of providing a useful training signal for a policy rather than a definition of what optimal behavior would look like.\n* 264: $\\Theta^{++}$ is undefined.\n* 268: clarify that fig 2 is in the Appendix\n* 262-274 is hard to understand without knowing details of Montezuma’s revenge.\n* Cite the earlier version of GO-EXPLORE as well, to preempt misunderstanding that Montezuma’s revenge was only solved in 2021.\n* I think 194 should say “4-retargetable”\n* It should be made explicit that the symmetric group acts via permuting dimensions (and which dimensions…) in the examples given.\n* 17: replace “;” with “and”\n* 21-29 are a bit unclear, probably because they are too terse (e.g. it is not yet clear what is meant by “parameter inputs”).\n* Are orbit tendencies well-characterized as “the fraction of $\\theta$ for which…” (128)?  If so, elaborate/explain/defend this way of speaking.\n", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n* Power-seeking is one of the most important concepts in AI existential safety.\n* This paper is one of a very few to make a substantive technical contribution to this topic, and I believe it represents meaningful progress on understanding power-seeking.\n* The insight that retargetability is sufficient for power-seeking is significant and highly original.\n\nWeaknesses:\n* This approach to understanding power-seeking doesn’t really address the main mechanisms by which we might expect to avoid building power-seeking agents: inductive biases and targeted feedback.  I view this paper as formalizing the argument that “alignment is hard because there are so many outcomes which involve human extinction, and we need to somehow direct the impact of AI systems towards the small fraction of those which do not”.  However, this argument doesn’t address the difficulty of alignment *relative to* the tools we have for alignment.  \n* The central points and ideas of the paper were not very clear.  I believe these are novel and subtle, and thus difficult to communicate effectively, but I think there is significant room for improvement.\n* The phrase “tends to” is not explicitly defined, but should be, given its central role.  It should also be explained and justified why the formal definition matches common use (to the extent it does).  \n* I believe $f_{blah}$ is overloaded in a confusing way, referring to the probability of an outcome, something a bit more generic (footnote 1), or a decision-making process.\n* The connection between the formally established orbit-level tendencies and power-seeking wasn’t clear enough, and relies a bit too much on Turner et al. 2021, I think.  I recommend front-loading something like some of the discussion in Section 5, e.g. into the introduction or at the end of Section 3.\n\nOverall, I think there are many opportunities for improving the clarity and exposition of this work.  I believe it would benefit from a substantial rewrite (more than would be appropriate for a revision), which might significantly increase its impact.  Still, I think the contributions are sound and important enough to (weakly) recommend acceptance.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept