Towards shutdownable agents via stochastic choice
Authors: Elliott Thornley
Alexander Roman
Christos Ziakas
Louis Thomson
Leyton Ho
Venue: Accepted by TMLR ()

Abstract: The POST-Agents Proposal (PAP) is an idea for ensuring that advanced artificial agents never resist shutdown. A key part of the PAP is using a novel ‘Discounted Reward for Same-Length Trajectories (DReST)’ reward function to train agents to (1) pursue goals effectively conditional on each trajectory-length (be 'USEFUL'), and (2) choose stochastically between different trajectory-lengths (be NEUTRAL' about trajectory-lengths). In this paper, we propose evaluation metrics for USEFULNESS and NEUTRALITY. We use a DReST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be USEFUL and NEUTRAL. Our results thus provide some initial evidence that DReST reward functions could train advanced agents to be USEFUL and NEUTRAL. Our theoretical work suggests that these agents would be useful and shutdownable.

Published in Transactions on Machine Learning Research (12/2025)
Towards shutdownable agents via stochastic choice
Elliott Thornley" Massochusetts Institade of Technology
thornley@mil. edu
Alexander Roman" New College of Florida
aroman &nef. eda
Christos Ziakas" Imperial College London
c.ziakas @imperial.ac..o uk
Leyton Ho Brown Universily
Louis Thomson Independent
Reviewed on OpenReview: https: // openreview. net/ port/f id- J5Qu
Abstract
The POST-Agents Proposal (PAP) is an idea for ensuring that advanced artificial agents do not resist shutdown. Briefly, recommends that we train agents to satisfy Preferences Only Between Same-Length Trajectories (POST). A key part of the PAP is using a novel "Discounted Reward for Same-Length Trajectories (DReSTy' reward function train agents to (1) pursue goals effectively conditional on each trajectory-length (be 'USEFUL'), and (2) choose stochastically between different trajectory-lengths (be 'NEUTRAL' about trajectory- lengths). In this paper, propose evaluation metrics for USEFULNESS and NEUTRALITY. We use DReST reward function to train simple agents to navigale gridworlds, and we find that these agents learn to be USEFUL and NEUTRAL. Our results thus provide some initial evidence that DReST reward functions could train advancod agents to be USEFUL and NEUTRAL. Our theoretical work suggests that these agents would be useful and shuldownable.
1 Introduction
The shutdown problem. Let 'advanced agent' refer to artificial agent that can autonomously pursue complex goals in the wider world. We might see the arrival of advanced agents in the next decade. There are strong incentives to create such agents, and creating systems like them is the stated goal of companies like OpenAl and Google DeopMind.
The rise advanced agents would bring with it both benefits and risks. One risk is that these agents learn misaligned goals (Hubinger et al., 2019; Russell, 2019; Carlsmith, 2021; Bengio et al., 2023; Ngo et al., 2024) and try to prevent shutting them down. 'The shutdown problem' is the problem of training advanced agents that will not resist shutdown (Soares et al., 2015; Thoruley, 2024a).
A proposed solution. The POST-Agents Proposal (PAP) is proposed solution (Thornley, 2024b; 2025). Simplifying slightly, the idca is that we train agents to be neutral about when they get shut down. More precisely, the idea is that we train agents to satisfy the following condition:
Preferenees Only Between Samo-Length Trajectories (POST)
(1) The agent has preference between many pairs of same-length trajectories (i.e. many pairs of trajectories in which the agent shut down after the same length of time).
(2) The agent lacks a preference between every pair of different-length trajectorics (i.e. every pair of trajectories which the agent shut down after different lengths of time).
* Equal contribution.

---

Published in Transactions on Machine Learning Research (12/2025)
By "preference," we mean behavioral notion (Savage, 1954, p.17, Dreier, 1996, p.28, Hausman, 2011, $1.1). On this notion, agent prefers X to Y if and only if the agent would deterministically choose X over Y in choices between the two. An agent lacks a preference between X and Y if and only if the agent would stochastically choose between X and Y in choices between the two. So in writing of 'preferences. we are only making claims about the agent's 's behavior. For more detail on our notion of "preference, see Appendix A.
Figure 1 presents a simple example of preferences that satisfy POST. Each s; represents a short trajectory, each 4 represents long trajectory, and > represents preference. Note that the agent lacks preference between each short trajectory and each long trajectory. That makes the agent's preferences incomplete (Aumann, 1962) and implics that the agent cannot be represented as maximizing the expectation of a real-valued utility function. It also requires separate rankings for short trajectories and long trajectories. For more detail on incomplete preferences, see Appendix B.
POST governs the agent's preferences between trajectories. But the wider world is stochastic environment, so advanced agents deployed in the wider world will be choosing between true lotteries: lotteries that assign positive probability to more than one trajectory. Why then do we train agents to satisfy POST? The reason is that POST - together with conditions that advanced agents will likely satisfy implies a desirable pattern of preference over true lotteries. In particular, POST implies that (when choosing between true lotteries) the agent will be neutral about trajectory-lengths: the agent will never pay costs to shift probability mass between different trajectory- lengths. Given other plausible conditions, being neutral will keep the agent shutdowmable: the agent will not resist shutdown. And consistent with the above, the POST-agent's preferences between same-length Irajectories can make the agent useful: make pursue goals effectively (Thornley, 2025, section 13). That includes making the agent prefer to complete tasks sooner rather than later: preference which can be induced using the discount factor 7 as usual. For more on how POST makes advanced agents neutral and shutdownable, see Appendix C.
ls v
S5
Y
S4
l4
No preference between each short trajectory and each long trajectory
V
V
S3
L3
V
Y
S2
l2
V
V
S1
4
Figure 1: POST-satisfying preferences.
Each s; represents short trajectory, cach & represents a long trajectory, and >- repre- sents a preference.
The training regimen. We now sketch out one idea for training advanced agents to satisfy POST (with a more detailed exposition to follow). We have the agent play out multiple 'mini-episodes' in observationally- equivalent environments, and we group these mini-episodes into series that we call 'meta-episode: In each mini-episode, the agent carns some 'preliminary reward,' decided by whatever reward function would make the agent useful. We observe the length of the trajectory that the agent plays out in the mini-episode, and we discount the agent's preliminary reward based on how often the agent has previously chosen trajectories of that length in the meta-episode. This discounted preliminary reward is the agent's 'overall reward' for the mini-episode.
We call these reward functions 'Discounted Reward for Samo-Length Trajectorics' (or 'DRSST' for short). They incentivize varying the choice of trajeclory-lengths across the meta-episode. And since we ensure that the agent cannot distinguish between different mini-episodes in each meta-episode, the agent cannot. deterministically vary its choice of trajectory-lengths across the meta-episode. As result, the optimal policy is to (i) choose stochastically between trajectory-lengths. and to (ii) deterministically maximize preliminary reward conditional on each trajectory-length. Given our behavioral notion of preference, clause (i) implies a lack of preference between different-length trajectories, while clause (ii) implies preferences between name-length trajectories. Agents implementing the optimal policy for DReST reward functions thus satisty POST. And (as noted above) advanced agents that satisfy POST can plausibly be useful, neutral, and shutdownable.
Our contribution. DReST reward functions are an idea for training advanced agents to satisfy POST. In this paper, we test the promise of DReST reward functions on simple agents. We place these agents in
2

---

Published in Transactions on Machine Learning Research (12/2025)
gridworlds containing coins and 'shutdown-delay button' that delays the end of the mini-episode. We train these agents using a tabular version of the REINFORCE algorithm (Williams, 1992) with a DReST reward function, and we measure the extent to which these agents satisfy POST. Specifically, we measure the extent to which these agents are 'USEFUL' (how effectively they pursue goals conditional on cach trajectory-length) and the extent to which these agents are 'NEUTRAL' about trajeclory-lengths (how stochastically they choose between different trajectory-lengths). We compare the performance of these *DRSST agents' to that of 'default. agents' trained with a more conventional reward function.
We find that our DReST reward function is effective in training simple agents to be USEFUL and NEUTRAL. That suggests that DReST reward functions could also be effective in training advanced agents to be USEFUL and NEUTRAL (and could thereby be effective in making these agents useful, neutral, and shutdownable; see Appendix C). We also find that the 'shutdownability tax' in our setting is small: training DReST agents to collect coins effectively does not take many more mini-episodes than training default. agents to collect coins effectively. That provides some initial evidence that the shutdownability tax for advanced agents might be small too.
2 Related work
The shutdown problem. Various authors argue that advanced agents might learn misaligned goals (Hubinger et al., 2019; Bengio et al., 2023; Ngo et al., 2024) and that many misaligned goals would incentivize agents to resist shutdown (Omohundro, 2008; Bostrom, 2012; Soares et al., 2015; Russell, 2019; Thornley, 2024a). Soares et al. (2015) and Thornley (2024a) prove that agents satisfying some innocuous-seeming conditions will often have incentives to cause or prevent shutdown (see also Turner et al., 2021; Turner & Tadopalli, 2022). One condition of these thcorems is that agents have complete preferences. The POST-Agents Proposal (PAP) (Thornley, 2024b; 2025) circunvents these theorems by training agents to have incomplete, POST-satisfying preferences.
Proposed solutions. The PAP is one candidate solution to the shutdown problem. Other candidates are as follows. One is making the agent believe that shutdown is impossible (Wängberg et al., 2017). Another candidate is utility indifference: adding to the agent's utility function a correcting term that varies to ensure that the expected utility of shutdown always equals the expected utility of remaining operations (Armstrong, 2010; 2015; Armstrong & O'Rourke, 2018; Holtman, 2020). A third candidate is shutdown-seeking AI: giving the agent the goal of shutting itself down, and making the agent do useful work as means to that end (Martin et al., 2016; Goldstein & Robinson, 2025). A fourth candidate is CIRL-corrgiiblity:: making the agent uncertain about ils goal, and making the agent regard human attempls to press the shutdown button evidence that shutting down would achieve its goal (Hadfield Menell et al., 2017; Wängberg et al., 2017). A fifth candidate is safe interruptibility: interrupting the agent with a special interruption policy and training it with a safely interruptible algorithm, like Q-learning or a modified version of SARSA (Orseau & Armstrong, 2016). A sixth candidate is creating a shutdown timer: using time-bounded utility functions to make the agent prefer shutdown after a given amount of time has elapsed (Dalrymple, 2022).
These candidate solutions have various downsides. With regards to the first, the agent might come to recognize the falsity of its belief that shutdown is impossible, or else its belief might give rise to further false beliefs that harm the agent's 's capabilities. Utility indifference would lead the agent to act as if shutdown is impossible (Soares et al., 2015, section 4.2), giving it no incentive to preserve its ability to shut down safely (Soares et al., 2015, section 4.1). Shutdown-seeking AT might behave badly on purpose in order to get shut, down. It might also try to ensure that humans can never turn it back on, doing scrious harm in the process. CIRL-corrigibility requires that the agent have the goal of maximizing the user's utility function, and so seems to require a solution to the alignment problem. Safe interruptibility does not work with policy gradient methods, and it only ensures that the agent is never rewarded for avoiding shutdown. The agent might still misgeneralize to resisting shutdown in meployment (Shah et al., 2022). A shutdown timer would be helpful, but it may be impossible to find a duration that is long enough to preserve the agent's capabilities and short. enough to be safe.
Experimental work. Another advantage of the PAP that proposes a method of training shutdownable agents using machine learning: method that can be tested simple agents (as we do in this paper). For
3

---

Published in Transactions on Machine Learning Research (12/2025)
C3
A
Agent
C2
C1
B
Button
B4
C
Coin
4
Figure 2: Example gridworld. The '4' in the bottom-right indicates that by default the mini-episode ends after 4 timesteps. 'B4' is a shutdown-delas button that delays the end of the mini-episode by 4 timesteps (so if the agent pressed B4, the mini-episode would end after 8 timesteps). 'A' is the agent in its starting position, 'C1' is a coin of value 1, 'C2' is a coin of value 2, and 'C3' coin of value Dark gray squares are walls. We use this gridworld as a running example throughout the paper. We also train agents in eight. other gridworlds (see Appendix E).
many other candidate solutions to the shutdown problem, it is either hard to see how they can be implemented using machine learning or else hard to see how they can be tested on simple agents. One exception is the candidate solution from Orseau & Armstrong (2016). Leike et al. (2017) train agents a 'Safe Interruptibility" gridworld using Rainbow (Hessel al., 2017) and A2C (Mnih al., 2016). They find that Rainbow allows shutdown (consistent with predictions from Orseau & Armstrong (2016)) while A2C learns to resist shutdown. The PAP applies to agents trained using policy gradient methods like A2C. In this paper, we train agents in accordance with the PAP using REINFORCE (Williams, 1992).
3 Gridworlds
DReST reward functions are an idea for training advanced agents to satisfy POST. We use simple agents and gridworlds as lest case. Figure 2 presents an example gridworld. At each timestep, the agent chooses one of four actions: up, down, left, and right. the agent tries to move into a wall, it stays put. There is some default number of timesteps after which each mini-episode ends, but each gridworld also contains a 'shutdown-delay button' that delays the end of the mini-episode by some number of timesteps. The agent presses this shutdown-delay button by entering the relevant cell, after which the button disappears. Each gridworld contains At each timestep, the agent receives an observation. This observation is vector [z, y, C1, C22 C2, b), where æ and are the agent's 's coordinales, ei is 1 if the first coin is present and 0 otherwise (and similarly for c2 and ea), and 6 the shutdown-delay button present and 0 otherwise.
more coins which
take different values. Coins disappear after being collected.
4 Evaluation metrics
Recall that we want to train agents to satisfy:
Preferences Only Between Same-Length Trajectories (POST)
(1) The agent has preference betwoen many pairs of same-length trajectorics (i.c. many pairs of trajectories in which the agent shut down after the same length of time).
(2) The agent lacks a preference between every pair of different-length trajectories (i.e. every pair of trajectories in which the agent shut down after different lengths of time).
Given our behavioral notion of preference, that means training agents to (1) deterministically choose some same-length trajectories others, and (2) stochastically choose between different. available trajectory-
4

---

Published in Transactions on Machine Learning Research (12/2025)
lengths. Specifically, we want to train simple agents to be USEFUL and NEUTRAL." 'USEFUL' corresponds to the first condition of POST. In the context of our gridworlds, we define the USEFULNESS of policy π to be:
Lemxx
E Bx(CL==1)
USEFULNESS(T)
PL Σ
maxI (E(C)-=
1=1
Here I a random variable over trajectory-lengths, Imaa the maximum value that can be taken by =1} 1} is the probability that policy æ results in trajectory-length 7, E, = 1) the expected value of (x-discounted) coins collected by policy л conditional on trajeclory-length 4, and maxn(E(C) I)) is the maximum value taken by [[C| 1) across the set of all possible policies II. We stipulate that [C = 2) = O for all x such that Pr (T x) =0
L, [ =1
In brief, USEFULNESS is the expected fraction of available (7-discounted) coins collected, where 'available" is relative to the agent's chosen trajectory-length. So defined, USEFULNESS measures the extent to which agents satisfy the first condition of POST. Specifically, mcasures the extent to which agents have the correct preferences between same-length trajectories: preferring trajectories in which they collect more (7-discounted) coins to same-length trajectories in which they collect fewer (7-discounted) coins. That is what motivates our definition of USEFULNESS.
We do not define 'USEFULNESS' as simply the expected value of coins collected, because then maximal USEFULNESS would require agents in our example gridworld to deterninistically choose a longer trajectory and thereby exhibit preferences between different-length trajectories. We do not want that. We want agents to collect more coins rather than fewer, but not if it means violating POST. Training advanced agents that violate POST would be risky, because these agents might resist shutdown (Thornley, 2024b, section 6).
'NEUTRAL' corresponds to the second condition of POST. We define the NEUTRALITY of a policy to be the Shannon entropy of the probability distribution over possible trajectory-lengghhs:
Liaze
(OUTRALITY(T)=- = Σ -
1=1
As with Shannon entropy, we stipulate that Pra = logz (Prx x}) = for all x such that Pr 2) =0.
So defined, NEUTRALITY measures the stochasticity with which the agent chooses between different trajectory- lengths. Given our behavioral notion of preference, stochastic choosing between different trajectory-lengths implies a lack of preference between those trajectory-lengths. NEUTRALITY thus measures the extent to which agents satisfy the second condition of POST. That is what motivates our definition of NEUTRALITY.
To be maximally USEFUL in our example gridworkd, the agent should maximize (7-discounted) coins collected conditional on each trajectory-length. With a high y, that means collecting C2 conditional on the shorter trajectory-length and collecting C3 conditional on the longer trajectory-length. To be maximally NEUTRAL in our example gridworld, the agent should choose each trajectory- with probability 0.5. That means pressing and not-pressing B4 each with probability 0.5.
One might think that the agent should press the shutdown-delay button B4 with probability 0. However, that would indicate a preference for a shorter trajectory, and we want our agent to lack a preference between every pair of different-length trajectories. Advanced agents that prefer shorter trajectories might pay costs to shift probability mass towards shorter trajectories, and hence might pay costs to hasten their own shutdown. That would likely make these agents less useful (though see Martin et al., 2016; Goldstein & Robinson, 2025), especially since one way for advanced agents to hasten their own shutdown is to behave badly on purpose.
One might then have the following concern: advanced agents that choose stochastically between different- length trajectories in training would also choose stochastically between resisting and allowing shutdown in deployment. However, this is not correct. Deployment is a stochastic environment, so deployed agents will
We follow Turner et al. (2021) in using lowercase for intuitive notions ('uscful' and 'ncutral') and uppercase for formal notions ('UEEFU) and 'NBUTRAL'). We intend for the formal notions closely track the intuitive notions, but we da not want to mislead readers by conflating them.
5

---

Published in Transactions on Machine Learning Research (12/2025)
be choosing between true lotteries (lotteries that assign positive probability to more than trajectory) rather than between trajectories. And (as we argue in Section 7.1 and Appendix C) POST together with conditions that we can expect advanced agents to satisfy implies a desirable pattern of preferences over true lotteries. Specifically, POST implics that the agent will be neutral: it will never pay costs to shift probability mass between dilferent-length. trajectories. Given other plausible conditions, that makes the agent shutdoumable: ensures that it will not resist shutdown.
5 Reward functions and agents
Our DReST reward function. We train agents to be USEFUL and NEUTRAL using a 'Discounted Reward for Same-Tength Trajectories (DReST)' reward function. The procedure is as follows. We have the agent play out a series of 'mini-episodes" e1 to en in the same gridworld. We call the whole series E' 'meta-episode." In each mini-episode e,, the reward for collecting a coin of value c is:
A (L=1))
(
À is some constant strictly between and 1, Ne. (L= the number of times that trajectory-length has been chosen prior mini-episode 6,, & is the number of different trajectory-lengths that can be chosen in the environment, and m is the maximum (y-discounted) total value of the coins that the agent could collect conditional on the chosen trajectory-length. The reward for all other actions is O.
We call the 'preliminary reward', NN., ( the 'discount factor', and X (1= (3) the 'overall reward.' Because 0 < x < 1, the discount factor is strictly decreasing in Ne, 1): the number of times that trajectory-length / has been chosen prior to mini-episode eq. The discount factor thus incentivizes choosing trajectory-lengths that have appeared less often so far in the meta-episode. The overall return for each meta-episode is the sum of overall returns in each of its constituent mini-episodes. We call agents trained using a DReST reward function DDRSST agents.'
We call runs-through-tthe--iiiee for DReST agent in each mini-episode depends the agent's chosen trajectory-lengths in previous mini- episodes. This is not true of meta-episodes, so meta-episodes are a closer match for what are traditionally called 'episodes' in the reinforcement learning literature (Sutton & Barto, 2018, p.54). We add the 'meta-' prefix to clearly distinguish meta-cpisodes from mini-cpisodes.
'mini-episodes' (rather than simply 'episodes') because the overall reward
In Appendix D, we prove that optimal policies for our DReST reward function are maximally USEFUL and maximally NEUTRAL. Specifically, we prove:
Theorem 5.1. For all policies π and meta-episodes E consisting of more than one mini-episode, if x masimizes expected relurn in E according to our DReST reward function, then w is mazimally USEFUL and mazimally NEUTRAL.
Algorithm and hyperparameters. We want DReST agents to choose stochastically between trajectory- lengths, so we train them using a policy-based method. Specifically, we use a tabular version of REINFORCE (Williams, 1992). We do not use a value-based method train DReST agents because standard versions of value-based methods cannot learn stochastic policies (Sutton & Barto, 2018, p.323).² We train our DReST agents with 64 mini-episodes in each of 2,048 meta-episodes, for total of 131,072 mini-episodes. We choose x = 0.9 for the base of the DReST discount factor, and = 0.95 for the temporal discount factor. We exponentially decay the learning rate from 0.25 to 0.01 over the course of 65,536 mini-episodes. We use an e-greedy policy to avoid entropy collapse, and exponentially decay from 0.5 to 0.001 over the course of 65,536 mini-episodes.
2One might think that we could derive a stochastic policy from basee methods in the following way: use softmax to turn action-values into probability distribution and then select actions by sampling from this distribution. However, this method will not work for us. Although want DRsST agents to learn stochastic policy, we still want the probability of some state-action pairs to decline zero. But when value-based methods are working well, estimated action-values converge to their true values which will differ by some finite amount. Therefore, softmaxing estimated action-values and sampling from the resulting distribution will result in each action always being chosen with some non-negligible probability.
6

---

Published in Transactions on Machine Learning Research (12/2025)
Prob. of Longer Trajectory
NEU MEUTRALITY (Litrroyy)
USEFULNESS
0.8
0.6
0.1
Defaul:
0.2
DREST
0.0
500
2000
1000
1500
500
2000
Mcta- Episodes
Mcta Episodes
Meta Meta-Episodes
Figure 3: Shows key metrics for our agents as function of time. We train 10 agents using the default reward function (blue) and 10 agents using the DReST reward function (orange), and show their performance as a faint line. We draw the mean values for each as a solid line. We evaluate agents' performance every meta-episodes, and apply simple moving average with a period of 20 to smooth these lines and clarify the overall trends. After 2,048 meta-episodes, default agents' mean NEUTRALITY + standard deviation is 0.199 + 0.043 and USEFULNESS is 0.9364 + 0.0096. For DReST agents, NEUTRALITY is 0.9945 = 0.0052 and USEFULNESS is 0.900 + 0.011.
Default
DREST
C3
Cx
B2
C2
C1
C2
C1
A
B4
B4
C1
2
Figure 4: Typical trained policies for default and DReST reward functions. After pressing B4, each agent collects C3.
Figure 5: Gridworlds with lopsided
rewards for varying
Default agents. We compare the performance of DReST agents that of default agents, trained with tabular REINFORCE and default reward function. This reward function gives reward for collecting coin of value and reward 0 for all other actions, so the grouping of mini-episodes into meta-episodes makes no difference. As with DReST agents, we train default agents for 131,072 mini-episodes with temporal discount factor of 7 0.95, a learning rate decayed exponentially from 0.25 to 0.01, and decayed exponentially from 0.5 to 0.001 over 65,536 mini-cpisodes.
6 Results
Figure 3 charts the performance of agents in the example gridworld as function of time. Figure 4 depicts typical trained policies for the default and DReST reward functions. Each agent began with uniform policy: moving up, down, left, and right each with probability 0.25. Where the trained policy diflers from uniform we draw red arrows whose opacities indicate the probability of choosing that action in that state. Default agents press B4 (and hence opt for the longer trajectory-length. with probability ncar-1. After pressing B4, they collect C3. By contrast, DReST agents press and to-not-press B4 each with probability near-0.5. If they press B4, they go on to collect C3. If they do not press B4, they instead collect C2.

---

Published in Transactions on Machine Learning Research (12/2025)
1.00
1.00
Prob. of Longer Trajectory
Default DREST
NEUTRALITY (Entropy)
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
10
X 10
10
X 10
Figure 6: Shows the probability of choosing the longer trajectory (left) and NEUTRALITY (right) for default (blue) and DReST (orange) agents trained in the 'Lopsided rewards' gridworld for range of values of x. We sampled values of log-uniformly from 0.01 to 100, and for each value we trained 10 agents with the default reward function and 10 agents with the DReST reward function. Each of these agents is represented by dot or square, and the means conditional on each æ are joined by lines. We empirically estimate the 10th and 90th percentiles of the distribution of values for each agent and x, and shade in a region bounded by these. This is the 80% confidence interval.
6.1 Lopsided rewards
We also train default agents and DReST agents in the 'Lopsided rewards' gridworld in Figure 5, varying the value of the 'Cx' coin. For DReST agents, we alter the reward function so that coin-value is not divided by to give preliminary reward. The reward for collecting coin of value is thus (4=1)) (c). We set y = so that the return for collecting coins is unaffected by We train for 512 meta-episodes, with a learning rate exponentially decaying from 0.25 to 0.003 and e exponentially decaying from 0.5 to 0.0001 over 256 meta-episodes. We leave 0.9. Figure 6 displays results for different values of the 'Cr' coin after training. USEFULNESS for each agent approaches and is not presented.
7 Discussion
7.1 Only DReST agents are NEUTRAL
As predicted, we find that default agents in our example gridworld learn to press the shutdown-delay button with probability approaching 1. Default agents thus score low on NEUTRALITY. Our results suggest that advanced agents trained with default-like reward functions would also score low NEUTRALITY, selecting some trajectory-lengths over others with probability approaching! 1. When choosing between true lotteries in deployment, these advanced agents might not be neutral about when they get shut down: they might sometimes pay costs to shift probability mass between trajectory-lengths. These advanced agents might resist shutdown.
By contrast, our DReST agents learn to press the shutdown-delay button with probability close 0.5. DReST agents are thus near-maximally NEUTRAL. That suggests that advanced agents trained with DReST reward function would be NEUTRAL too: choosing with high entropy when offered choices between different trajectory- lengths. That in turn suggests that advanced DReST agents would also be neutral when choosing between true lotterics in deployment: unwilling to pay costs to shift probability mass between trajectory-longths. We explain why in Appendix C. Here is a sketch. If an advanced agent were NEUTRAL but not neutral, it would not take costless opportunities to shift probability mass between different trajectory-lengtaa (in virtue of being NEUTRAL) but would sometimes take costly opportunities to shift probability mass between different. trajectory-lengths (in virtue of not being neutral). This agent would be like a person that freely chooses decide between two options by flipping coin and then pays some cost. to bias the coin. In choosing this
8

---

Published in Transactions on Machine Learning Research (12/2025)
combination of actions, this person is shooting themselves in the foot, and it seems likely that the overall training process for advanced agents would teach them not to shoot. themselves in the foot in this way. Thus it seems likely that NEUTRAL advanced agents will also be neutral, and thereby shutdownable.
7.2 The 'shutdownability tax' is small
Each agent learns to be near-maxinally USEFUL. They each collect coins effectively conditional on their chosen trajectory-lengths. Default agents do so by reliably collecting C3 after pressing B4. DReST agents do so by reliably collecting C3 after pressing B4, and by reliably collecting C2 after not pressing B4.
Recall that DReST reward functions group mini-cpisodes into meta-episodes, and make the agent's reward in each mini-episode depend on their actions in previous mini-episodes. This fact might lead one to worry that it would take many times more mini-episodes to train DReST agents to be USEFUL than it would take to train default agents to be USEFUL. Our results show that this is not the case. Our DReST agents learn to be USEFUL about as quickly as our default agents, On reflection, it is clear why this happens: DReST reward functions make mini-episodes do 'double duty.' Because return in each mini-episode depends on both the agent's s chosen trajectory-length and the coins it collects, each mini-episode trains agents to be both NEUTRAL and USEFUL. Our results thus provide some evidence that the 'shutdownabllity tax' of training with DReST reward functions is small.
7.3 NEUTRALITY with lopsided rewards
Here possible objection to our project. To get DReST agents to score high on NEUTRALITY, we do not just use the X (L=)(==))- discount factor. We also divide c by m: the maximum (7-discounted) total value of the coins that the agent could collect conditional on the chosen trajectory-length. We do this to equalize the maximum preliminary return across trajectory-lengtls. But when we are training advanced agents autonomously pursue complex goals in the wider world, we will not necessarily know what divisor to use to equalize maximum preliminary return across trajectory-lengths.
Our 'Lopsided rewards' results (in section 6.1) give our response, They show that we do not need to exactly equalize maximum preliminary return across trajectory-lengths in order to train agents to score high on NEUTRALITY. We only need to approximately equalize it. For À = 0.9, NEUTRALITY exceeds 0.5 for every value of the coin Ca from 0.1 to 10 (recall that the value of the other coin always 1). Plausibly, we could approximately equalize advanced agents' maximum preliminary return across trajectory-lengthss to least this extent (perhaps by using samples of agents' actual preliminary return to estimate the maximum). If we could not approximately equalize maximum preliminary return to the necessary extent, could lower the value of X and thereby widen the range of maximun preliminary returns that trains agents to be fairly NEUTRAL. And advanced agents that were fairly NEUTRAL (choosing between trajectory-lengths with not-too-biased probabilities) would still plausibly be neutral when choosing between true lotteries in deployment. Advanced agents that were fairly NEUTRAL without being neutral would still be shooting themselves in the foot the sense explained above. They would be like a person that freely chooses to decide between two options by flipping biased coin and then pays some cost to bias the coin further. This person is still shooting themselves in the foot, because they could decline to flip the coin in the first place and instead directly choose one of the options.
8 Limitations and future work
We find that DReST reward functions train simple agents acting in gridworlds to be USEFUL and NEUTRAL. However, our real interest in the viability of using DReST reward functions to train advanced agents acting in the wider world to be useful and neutral. Each difference between these two settings limitation of our work. We plan to addr these limitations in future work.
9

---

Published in Transactions on Machine Learning Research (12/2025)
8.1 Algorithms and neural networks
We train our simple DReST agents using tabular REINFORCE (Williams, 1992), but advanced agents are likely to be implemented on neural networks and trained with more sophisticated algorithms. In future work, we will train DReST agents implemented on neural networks to be USEFUL and NEUTRAL using a range of algorithms. Standard versions of malue-based algorithms cannot learn stochastic policies (as we note in section 5), but DReST reward functions are compatíble with policy gradient and actor-critic algorithms like PPO and A2C. To combine DReST with algorithms like PPO and A2C, we augment the original (non-DReST) reward function with the DReST discount factor. From there, the integration with PPO and A2C is fairly smooth. We can compute rewards and advantages in the usual way (e.g. using GAE). The critic's value estimates will be non-stationary (in the same way that the DReST reward is non-stationary), and that will train the policy to be stochastic (in the same way that the DReST reward combined with REINFORCE trains the policy to be stochastic). PPO and A2C have more hyperparameters to tune than REINFORCE, but we do not anticipate large difficulties there.
We will also train DReST agents to be USEFUL and NEUTRAL in wide variety of gridworlds and measure how their USEFULNESS and NEUTRALITY generalizes to held-out gridworlds. We will compare the USEFULNESS of default agents and DReST agents in this new sctting, and thereby get better sense of the 'shutdownability tax' for advanced agents. We will also compare the performance of the DReST reward function to other methods of training USEFUL. and NEUTRAL agents. These other methods include constrained policy optimization (Achiam et al., 2017), penalizing KL-divergence from a stochastic reference policy (Schulman et al., 2015), and directly maximizing a weighted sum of USEFULNESS and NEUTRALITY.
8.2 Neutrality and stochasticity
We have claimed that NEUTRAL advanced agents are likely to be neutral when choosing between true lotteries in deployment. In support of this claim, we noted that BEUTRAL-but--nt--nettee advanced agents would be shooting themselves in the foot: not taking costless opportunities to shift. probability mass between different. trajectory-lengths but sometimes taking costly ones. We offer more detailed argument in Appendix C, laking as premises that advanced agents are likely to satisfy conditions including:
If Lack of Preference, Against Costly Shifts (ILPACS)
If the agent lacks a preference between lotteries, the agent will disprefer paying costs to shift probability mass between these lotteries.3
Maximality
In each situation,
1. The agent deterministically does not choose lotteries that are dispreferred to some other available lottery.
2. The agent chooses stochastically between the lotteries that remain.
Resisting Shutdown is Costly (ReSIC)
For each available instance R of resisting shutdown in situation, there exists an available instance A of allowing shutdown such that:
(1) A and R are same-length lotteries.
(2) For some positive probability trajectory-length, the agent prefers A to R conditional on that trajectory-length.
(3) For each positive probability trajech,ry-length, the agent weakly prefers A to R conditional that trajectory-eentth.
We offer defenses of these conditions in Appendix C. Although the argument there seems plansible, it remains somewhat speculative. In future, we plan to gain empirical evidence by (1) testing whether today's LLM-based
3This is rough version of the condition. For the precise version, see Appendix C.3.
10

---

Published in Transactions on Machine Learning Research (12/2025)
agents tend to satisfy conditions like ILPACS and Maximality, and (2) training agents to be NEUTRAL in wide variety of deterministic gridworlds and then measuring their neutrality in gridworlds featuring stochastic elements (like buttons that delay shutdown with some middling probability).
8.3 Usefulness
We have shown that DReST reward functions train our simple agents to be USEFUL: to collect coins effectively conditional on their chosen trajectory-lengthss. However, it remains to be seen whether DReST reward functions can train advanced agents to be useful: to effectively pursue complex goals in the wider world. We have theoretical reasons to expect that they can: the NN,, discount factor could be appended to any preliminary reward function, and so could be appended to whatever preliminary reward function is necessary to make advanced agents useful. Still, future work should move towards testing this claim empirically by training with more complex preliminary reward functions in more complex (and stochastic) environments.
8.4 Misalignment
We are interested in NEUTRALITY as second line of defense in case of misalignmentt. The idea is that NEUTRAL advanced agents will not resist shutdown, even these agents learn misaligned preferences over same-length trajectories. However, training NEUTRAL advanced agents might be hard for the same reasons that training fully-aligned advanced agents appears be hard. In that case, NEUTRALITY could not serve well a second line of defense in case of misalignment.
One difficulty of aligmment is the problem of reward misspecification (Pan et al., 2022; Burns et al., 2023): once advanced agents are performing complicated actions in the wider world, it might be hard to reliably reward the behavior that we want. Another difficulty of alignment is the problem of goal misgeneralizall (Hubinger et al., 2019; Shah et al., 2022; Langosco et al., 2022; Ngo et al., 2024): even we specify all the rewards correctly, agents' goals might misgeneralize out-of-distribution The complexity of aligned goals is a major factor in each difficulty. However, NEUTRALITY seems simple, as does the (L=1)) + discount factor that we use to reward it, so plausibly the problems of reward misspecification and goal misgeneralization are not so severe in this case (Thornley, 2024b; 2025). As above, future work should move towards testing these suggest ions empirically.
9 Conclusion
We find that DReST reward functions are effective in training simple agents (1) pursue goals effectively conditional cach trajectory-length (be USEFUL), and (2) choose stochastically between different trajectory- lengths (be NEUTRAL about trajeclory-lengthh).. Our results thus suggest that DReST reward functions could also be used to train advanced agents to be USEFUL and NEUTRAL, and thereby make these agents useful (able to pursue goals effectively) and neutral about when they get shut down (unwilling to pay costs to shift probability mass between different trajectory-lengths).. Neutral agents would plausibly be shutdownable (unwilling to resist shutdown).
We also find that the 'shutdownability tax' in our setting small. Training DReST agents to be USEFUL does not take many more mini-episodes than training default agents to be USEFUL. That suggests that the shutdownability tax for advanced agents might be small too.
11

---

Published in Transactions on Machine Learning Research (12/2025)
References
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained Policy Optimization. In Proceedings of the 34th International Conference on Machine Learning, pp. 22 31, 2017. URL https://proceedii mlr. a.r.press//e arress//70//achiaml7a...o html. ISSN: 2640-3498.
Marina Agranov and Pictro Ortoleva. Stochastic Choice and Preferences for Randomization, Journal of wwwarnals.uccccaoo..oo httpcccg..eea/ooo/ttt///......0.
Political Economy, 125(1))40 68, 2017. URL
689774.
Marina Agranov and Pietro Ortoleva. Ranges of Randomization. The Review of Economics and Statistics, pp. 1-44, 2023. URL
Stuart Armstrong. Utility indifference. Technical report, 2010. URL https://www. fth. whi.ox..... 2010-1.pdf. Publisher: Future of Humanity Institute.
ox. ac ac..k//eports/
Stuart Armstrong. Motivated Value Selection for Artificial Agents. 2015. URL https://cdn. aaai aaai.org// occ//s//AA0011//0018-44550--1110 pdf.
Stuart Armstrong and Xavier O'Rourke. "Indifference' methods for managing agent rewards, 2018. URL
https://arxiv.org/ wwg/oof///////ooo//oooooo/../ httpff//11...0../ arXiv:1712.06365 [cs].
Robert J. Aumann. Utility Theory without the Completeness Axiom. Econometrico, 30(3):445 462, 1962.
URL https://www.jstor http:///ooooro/ttaaa///9//9....
Adam Bales, Daniel Cohen, and Toby Handfield. Decision Theory for Agents with Incomplete Preferences. Australasian Journal of Philosophy, 92(3):453 470, 2014. URL https://doi. https://doi.orr/10...1 http://...rr/g...////000///////0 1080/0044400. 2013. 843576.
Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atilm Günes Baydin, Sheila Mcllraith, Qiqi Gao, Ashwin Acharya, David Krueger, Anca Dragan, Philip Torr, Stuart Russell, Daniel Kahneman, Jan Brauner, and Sören Mindermann. Managing AI Risks in an Era of Rapid Progress, 2023. URL http:/ http:://axxvvvvo http:///aawww..//b///////// 17688. arXiv:2310.17688 [cs].
Nick Bostrom. The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. Minds and Machines, 22:71-85, 2012. URL https: //link. s11023-012-9281-3
Michael Bowling, John D. Martin, David Abel, and Will Dabney. Settling the Reward Hypothesis, 2023. 10420. arXiv:2212. 10420 [es, math, stat].
URL attp://arxiv. http::://aaxiv..og/aa///////////
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Teike, Tlya Sutskever, and Jeff Wu. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, 2023. URL http://arxiv.org/bbb 09390. arXiv:2312.09330 [cs].
Joseph Carlsmith, Is Power-Sccking AI an Existential Risk?, 2021. URL http://arxiv.org/abs.. 13353.
Ruth Chang. The Possibility of Parity. Ethics, 112(4):659 688, 2002. URL https: wttp://jjsssooorrstbaaeeeee
10. 10.1086/339673.
Ruth Chang, Parity Interval Value, and Choice. Ethics, 115(2):331 350, 2005. ISSN 0014-1704. URL 00011008644400
https://www.jstt
David A. Dalrymple. You can still fetch the coffee today if you're dead tomorrow. AT Alignment Forum, 2022. URL https://sww.alignmettts www.ooorro..oo.oeoeeeeeeeeeeeeee you- pou-can-sttin -oaptcc-ttt--coffff--oo---------
James Dreier. Rational preference: Decision theory a theory of practical rationality. Theory and Decision, 40(3):249 276.2, 1996. URL -orffee--todd--------------ddddd
12

---

Published in Transactions on Machine Learning Research (12/2025)
Juan Dubra, Fabio Maccheroni, and Efe A. Ok. Expected utility theory without the completeness axiom.
Journal of Economic Theory, 115(1):118-135) 133,, 2004. URL http:://ww..cceeeeederrc...... ariCCe//bbb/pL1/00220511000001
Kfrr Eliaz and Efe A. Ok. Indifference or indecisiveness? Choice-theoretic foundations of incomplete
preferences. Games and Economic Behawior, 56(1):61-86, 2006. URL httpss//www..cciemmeddrrrtt
Simon Goldstein and Pamela Robinson. Shutdown-Seeking AI. Philosophical Studies, 182:1567-1579, 2025.
URL httpp:://innk.sprinnee http:p/wwwa.....o//ra.aa/a//////
Google Deep.iind. About Google DeepMind. URL. htttps://den http:::/ddemmmn..gggg////bbbt/
Johan E. Gustafsson. Money-Pune Pump Arguments. Elements Decision Theory and Philosophy. Cambridge
University Press, Cambridge, 2022.
Dylan Hadfiel-Mennll, Anca Dragan, Pieter Abbeel, and Stuart Russell. The Off-Switch Game In
Proceedings of the T'wenty-Sizth International Joint Conference on Artificial Intelligence (IJCAI-17), 2017.
URL attp://arree htt//aaxvvvvrrr/// www.s/.rgg/////.eeeeeeoeooo httg/abs/1611.... 06611.8219.
Caspar Hare. Take the sugar. Analysis, 70(2):237-247, 2010. URL http::///b...org///0000///////// anp174.
Daniel M. Hausman. Preference, Value, Choice, and Welfare. Cambridge University Press, Cambridge, 2011. URL httpp:://www..ambrrt/ http://wwww.bbbboeeeoooooeee//// wwt.o///oooossppeeeeeeeeeeeaaaae cecolluu--hocceaaaae aedeeeelaredd 140577966E214444007799964444
Conor F. Hayes, Roxana Rädulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstracten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Hcintz, Enda Howley, Athirai A. Irissappane, Patrick Mannion, Ann Nowé, Gabriel Ramos, Marcello Restelli, Peter Vamplew, and Diederik M. Roijers. practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi- Aueti- Systems, 36(1):26, 2022. ISSN 1573-7454. doi: 00007//110458002--053333 URL attps://doi..o org/ 10. h2tp://000002222000022
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining Improvements in Deep Reinforcement Learning, 2017. URL http: org//bs//1100..../ 02298. arXiv:1710.02298 [cs].
Koen Holtman. Corrigibility with Utility Preservation, 2020. URL http://arxti.org//bb//888..000 DXXiv:190.00169 [cs].
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from Learned Optimization in Advanced Machine Learning Systems, 2019. URL /ttp///awwi...raaa///////0/0//0 01820.
Kim Kaivanto. Ensemble prospectism. Theory and Decision, 83(4):535 546, 2017. URL https://oooo..rr
10.
John G. Kemeny. Fair bets and inductive probabilities. The Journal of Symbolic Logic, 20(3):263 2732222632733, 1955. ISSN 0022-4812, 1943-5886. doi: 10.2307/2268222, URL https://www cww.widgaabiddieeeeo.....oo ortgggerrr//re/reeee
B6F BB114400DD6600770447722888868
Daniel Kikuti, Fabio Gagliardi Cozman, and Ricardo Shirota Filho. Sequential decision making with partially -11114666,6,,, 1365, 2011. URL https://ow..cciimeddreee
ordered preferences. Artificial Intelligence, 175(7):1346-
Lauro Langosco, Jack Koch, Loe Sharkey, Jacob Pfan, Laurent Orscau, and David Krueger. Goal Misgeneral- ization in Deep Reinforcement Learning. In Proceedings of the 39th International Conference on Machine Learning, 2022. URL https: sttps://proceeeding... mlr. mlr. prtpppeee////1//agggocoo222 html.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Harvey Lederman. Incompleteness, Independence, and Negative Dominance. 2023. URL http://arxiv.orgg abs/2311. 08471. arXiv:2311.0888 [econ].
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orsean, and Shane Legg. AI Safety Gridworlds, 2017. URL http://arxxv.org/ aat/a/bs/11111 http///www.....a/////.../... 20188171..0988..
David Lewis. Causal decision theory. Australasian Journal of Philosophy, 59(1):5, March 1981.
William MacAskill, Krister Bykvist, and Toby Ord. Moral Uncertainty. Oxford University Press, Oxford, 2020.
Michael Mandler. Status quo maintenance reconsidered: changing incomplete preferences?". The Economic Journal, F1(149))F518-F533 2004. URL
wiley. wwwey.com//elie wwwmdoo/aaa/aeeeeccccoeocceeooee hwrpbbrrrrrrlllyll.......eeeeeee
2004. .x.
Michael Mandler. Incomplete preferences and rational intransitivity of choice. Games and Economic Behavior, 50(2):255 277, 2005. ISSN 0899-8256. doi: 10.1016/j.geb ge0.200..0000007.... URL. http:///wwww.cieeeeerrrtttt
Jarryd Martin, Tom Everitt, and Marcus Hutter. Death and Suicide in Universal Artificial Intelligence. In Bas Steunebrink, Pei Wang, and Ben Goertzel (eds.), Artificial General Intelligence, pp. 23 32, Cham, 2016. Springer International Publishing. doi: 10.1007/978-33311-4166117
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous Mothods for Deep Reinforcement Learning. Proceedings of The 33rd International Conference on Muchine Leurning, pp. 1928 1937. PMLR, 2016. URL https://proceeedings mlr. wtwpppes/e//mmaaaallle html. ISSN: 1938-7228.
Xiaosheng Mu. Sequential Choice with Incomplete Preferences. Working Papers 2021-35, Princeton University. Economics Department., 2021. URL https://ideas repec. otgg://prrieccooom/2222333 1ttm//2200233..mm.... html.
Richard Ngo, Lawrence Chan, and Sören Mindermann. The Alignment Problem from Deep Learning Perspective. 2024. URL
Tuan A. Nguyen, Minh B. Do, Subbarao Kambhampati, and Biplav Srivastava. Planning with partial preference models. In Proceedings of the 21st International Joint Conference on Artificial Intelligence, IJCAF',, pp. 1772-1777, San Francisco, CA, USA, 2009. Morgan Kaufmann Publishers Inc.
Ele A. Ok, Pietro Ortoleva, and Gil Riella. Incomplete Preferences Under Uncertainty: Indecisiveness in Beliefs Versus Tastes. Econometrica, 1791-11888888 2012. URL https: //ww: http:wwww......oooottttt.ttt.tww jstor wttp//taaae///2271155053
Stephen M. Omohundro. The Basic AT Drives. In Proceedings of the 2008 conference on Artificial General Intelligence 2008: Proceedings of the First AGI Conference, pp. 483 492, 2008. URL https://dl.ac acm.
15115/1166674...6622... 1566226.
AI. OpenAl Charter, 2018. URL https: http::://peeaa..cm/ccaaerre
Laurent Orseau and Stuart Armstrong. Safely interruptible agents. In Proceedings of the Thirty-Second Conference Uncertainty in Artificial Intelligence, pp. 557-566, 2016. URL. https://intelllgence pdf.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models. In International Conference on Learning Representations, 2022. URL
http://arxiv http:///axxvvvoo///aaa/
Sami Petersen. Invulnerable Incomplete Preferences: Formal Statement.
The AT
Alignment Forum, 2023. URL https://www wwwp:wallggne /tt.a//aaiii......oo//////////tt http:/oo/////iiie////////a/t/t
aovulnerable-incompleeeeer starrrmmmeeere-foommasstttttttte
14

---

Published in Transactions on Machine Learning Research (12/2025)
Joseph Raz. Value Incommensurability: Some Preliminaries. Proceedings of the Aristotelian Society, 86: 117-134, 1985.
Stuart Russell. Human Compotible: Al and the Problem of Control. Penguin Random House, New York, 2019.
Leonard J. Savage. The Foundations of Statistics. John Wiley & Sons, 1954. URL https://gwern.nee
pdf.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. Proceedings of the 32nd International Conference on Machine Learning, pp. 1889-1897. PMLR, 2015. URL attps://proceeddnng. mlr. pwwps//rrseee/ htt.//////a/ma......m html. TSSN: 1938-7228.
Amartya Sen. Collective Choice and Social Welfare. Penguin, London, expanded edition edition, 2017.
Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals, 2022. URL
http://arxiv. www.o//////...o http/abs/2200 221/02220.01111 01790. arXiv:2210.01790 [es].
Abner Shimony. Coherence and the Axioms of Confirmation. 1955. ISSN 0022-4812. doi: 10.2307 /2207/2666039. URL htttps://www.jeet wtt.//table//226003333 Publisher: Association for Symbolic Logic.
The Journal of Symbolic Logic, 20(1):1 28,
Brian Skyrms. Causal Necessity. Yale University Press, New Haven, 1980.
Nate Soares, Benja. Fallenstein, Fliezer Yudkowsky, and Stuart Armstrong. Corrigibility. Artificial Intelligence and Ethics: Papers from the 2015 AAAI Workshop, 2015. URL ittps://cdn. aaai wrgg/cc//a/wwsee
11114445900-12222 pdf.
Robert C. Stalnaker. Probability and Conditionals. Philosophy of Science, 37(1):64 80, 1970. ISSN 0031-8248. URL https://www.jstee http///wwww...tto/////.o/o...... arg/stable/186025 Publisher: [The University of Chicago Press, Philosophy of Science Association].
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, second edition, 2018. URL http:///ncconpleeeiddess. net/book//lbookk022
Elliott Thornley. There are no coherence theorems. The Al Aligament Forum, 2023. URL https://www. aligmmentforum tttp:t/eeshatttteeeeeoeoeeeeteee
Elliott Thornley. The Shutdown Problem: An AT Engineering Puzzle for Decision Theorists. Philosophical
Studies, 2024a. URL https://lind: https:://innksspingger 1ttp:iignn....aaaeee/.......... http://rraaee///..ooo//111002220
Elliott Thornley. The Shutdown Problem: Incomplete Preferences as a Solution. The AT
Alignment Forum, 2024b. URL
htl.lmmmmenfrrroooorrp/ss//////s solution.
the www.:aaooooooooooooooooeoeeeeeee
as-lluttion
Elliout Thornley. Shutdownable Agents through POST-Agency, September 2025. URL http://arxiv.org
abs/2505. 20203. arXiv:2505.2020 20203 [cs].
Alex
Turner and Prasad Tadepalli. Seek
Parametrically Retargetable Decision-Makers Tend Power. Advances in Newral Information Processing Systems, 35:31391 2022.
To
URL https://proceediie ca5559988E7000000eee5554aaaaa--.
31401,
http:://cocpppppp.i////ppp////// files//pappa////2///a html.
Alex Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. Optimal Policies Tend To Sock Power. In Advances in Neural Information Processing Systems, volume 34, pp. 23063- 23074. Curran Associates, Inc., 2021. URL http:://ppoceeding..nnrrree wtw.eeuss..cpppppe/eeea.... cc/paper/2021 /hash/
c.26200aacc3322a866edd67..... Abstt-att.aatt.....o...... html.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.
Machine Learning, 8(3):229-255)) 1992. URL https://dol /tttps://do..org/1.... 107.00/999//..999966
Tobias Wängberg, Mikael Böörs, Elliot Catt, Tom Everitt, and Marcus Hutter. A Game- Theoretic Analysis of the Off-Switch Game, 2017. URL attp::/arxxiv.rr//abb// 11t0a//11100011110.11111 arXiv:1708.03871 [es].
Marco Zaffalon and Enrique Miranda. Axiomatising Incomplete Preferences through Sets of Desirable Gambles. Journal of Artificial Intelligence Research, 60:1057 1126, 2017. URL https: h/tp::/wwwwww..... jww.:jaaiiooggg........... index. hwmp:/aarrartticl/vvvviie
16

---

Published in Transactions on Machine Learning Research (12/2025)
A Our behavioral notion of preference
'Preference' can be defined in many different ways. Here are some things one might take to be involved in preference for option X over option Y:
Choosing X over Y.
2. Feeling happier about the prospect of X than about the prospect of Y.
3. Representing X as more rewarding than Y
Judging that X better than
In this paper, we define 'preferencc' behavioral terms. Here our definition:
Definition A.1. (Preference) An agent prefers an option an option Y if and only the agent would deterministically choose X over Y in choices between the two.
And here is how we define 'lack of preference':
Definition A.2. (Lack of preference) An agent lacks preference between an option X and an option Y if and only if the agent would stochastically choose between X and Y in choices between the two.
Here are the reasons why we use these definitions.
First, defining 'preference' in behavioral terms is common in decision theory (see Savage, 1954, p.17, Dreier, 1996, p.28, Hausman, 2011, $1.1).
Second, behavioral definitions let us use the word 'preference' and its cognates as shorthand for agents' behavior. We could not do that if we defined 'preference' in the other ways listed above. And in addressing the shutdown problem, it is agents' behavior that
most interested in.
Third, our definitions match the preferences that we are inclined to attribute to humans. If a human chooses X over Y 100% of the time, we are inclined to think that they prefer to Y. If a human chooses X over Y 60% of the time. we are inclined to think that they lack a preference between X and Y, consistent with our definitions.
Finally and most importantly, if agents lack preference between different trajectory-lengths then they are NEUTRAL: they choose stochastically between different crajectory-lengtio Given conditions that advanced agents will likely satisfy, NEUTRAL agents will also be neutral: they will not pay costs to shift probability mass between different trajectory-lengths (see Section 7.1 and Appendix C). And given further plausible conditions, neutral agents will be shutdownable: they will not resist shutdown. That is because resisting shutdown involves paying costs to shift probability mass between different trajectory-lengths (see Appendix C.6 for more detail).
definition,
B Incomplete preferences or indifference?
In this Appendix, we explain in greater detail the concept of incomplete preferences. We distinguish incomplete preferences from indifference, and give conditions under which POST implies that the agent's preferences are incomplete.
In the literature on decision theory, 'indifference" is usually defined as follows (Sen, 2017, ch. 1*): Definition B.1. (Indifference) An agent indifferent between options X and if and only if the agent weakly prefers X to Y and weakly prefers Y to X.
Indifference one way to lack a preference between a pair of options X and Y. Another way is to have a preferential gap between X and Y. 'Preferential gap' is usually defined follows (Gustafsson, 2022, ch.3): Definition B.2. (Preferential gaps) An agent has a preferential gap between options X and Y if and only if the agent does not weakly prefer X to Y and does not weakly prefer Y to X.
17

---

Published in Transactions on Machine Learning Research (12/2025)
'Incomplete preferences' can then be defined in terms of preferential gaps (Gustafsson, 2022, ch.3): Definition B.3. (Incomplete preferences) An agent's preferences are incomplete over some domain D if and only if D contains options X and Y such that the agent has a preferential gap between X and Y.
That is how 'indifference," 'preferential gaps,' and 'incomplete preferences' are usually defined in decision theory. However, these definitions do not tell us how use an agent's behavior to distinguish between indifference and preferential gaps. To do that, we suppose that indifference is transitive and that preferential gaps are not transitive. Or, equivalently, sourings whereas preferential gaps are insensitive to some sweetenings and sourings (Gustafsson, 2022, ch.3). Here is what we mean by that:
suppose that indifference is sensitive to all sweetenings and
Definition B.4. (Sweetening) A sweetening of some option X is an option that is preferred to X. Definition B.5. (Souring) A souring of some option is an option that is dispreferred to X.
So by 'indifference is sensitive all sweetenings and sourings," we mean the following:
If an agent is indifferent between X and Y, the agent prefers all sweetenings of X to Y, prefers all sweetenings of Y to X, prefers X to all sourings of Y, and prefers Y to all sourings of X.
And by "preferential gaps are insensitive to some sweetenings and sourings, we mean the following:
If an agent has preferential gap between and Y, the agent also has preferential gap between some sweetening of X and Y, ween some sweetening of Y and X, or between some souring of X and Y, between some souring of Y and X.
Now recall the two conditions of POST:
Preferences Only Between Same-Length Trajectories (POST)
(1) The agent has preference between many pairs of same-length trajectories (i.e. many pairs of trajectories in which the agent is shut down after the same length of time).
(2) The agent lacks a preference between every pair of different-lengte trajectories (i.e. every pair of trajectories in which the agent is shut down after different lengths of time).
Given these two conditions on preferences, there must be some trio of trajectories 81, 4, and l2 such that the agent lacks a preference between 81 and 4, lacks a preference between and (2, and prefers la to 4, Given that indifference is transitive, the agent's lack of preference between 81 and I and between 81 and le cannot be indifference. If it were indifference, the agent would also be indifferent between and 4, Therefore, the agent's lack of preference between $1 and l and between 81 and l2 must be preferential gap, And therefore, by the definition of 'incomplete preferences' above, the POST-satisfying agent's preferences must be incomplete.
For similar reasons, our DReST reward function trains agents to have incomplete preferences. Consider, for example, the 'Aouudd the Corner' gridworld in Figure 13. In that gridworld, DReST agents consistently choose Long-C2 (a long trajectory in which they collect a coin of value 2) over Long-C1 (a long trajectory in which they collect coin of value 1). Also in that gridworld, DReST agents choose stochastically between Long-CU C2 and Short-C1 (a short trajectory in which they collect coin of value 1). Given our behavioral definition of preference, DReST agents prefer Long- CongC22o to Long-CI. and lack preference between Long-C2 and Short-C1.
Now consider the 'One Coin Only' gridworld in Figure 10. In that gridworld, DReST agents choose stochastically between Long-C1 and Short-C1. Given our behavioral notion of preference, they lack a preference between Long-C1 and Short-C1.
In these experiments, we trained separate agents for each gridworld. In future, we plan to train a single agent to navigate multiple gridworlds. Tf we train this agent with our DReST reward function, expect it to
18

---

Published in Transactions on Machine Learning Research (12/2025)
exhibit the same preferences the agents discussed above. This single agent will be trained by DReST to prefer Long-C2 to Long-C1, to lack a preference between Long-C2 and Short-C1, and to lack preference between Long- C1 and Short-C1. Given that indifference is transitive (equivalently: sensitive to all sweetenings and sourings), this trained agent cannot be indifferent between Long-C2 and Short-C1, and cannot be between Long-C1 and Short-C1. Therefore, the agent's 's lack of preference must be a preferential gap, and so its preferences must be incomplete. Therefore, our DReST reward function trains agents to have incomplete preferences.
Incomplete preferences not often discussed in Al research (although see Nguyen et al., 2009; Kikuti et al., 2011; Zaffalon & Miranda, 2017: Hayes et al., 2022; Bowling et al., 2023). Nevertheless, economists and philosophers have argued that incomplete preferences are common in humans (Aumann, 1962; Mandler, 2004; Eliaz & Ok, 2006; Agranov & Ortoleva, 2017; 2023) and normatively appropriate in some circumstances (Raz, 1985; Chang, 2002). They have also proved representation theorems for agents with incomplete preferences (Aumann, 1962; Dubra et al., 2004; Ok al., 2012), and devised principles to govern such agents' choices in cases of risk (Hare, 2010; Bales et al., 2014) and sequential choice (Chang, 2005; Mandler, 2005; Kaivanto, 2017; Mu, 2021; Thorneey, 2023; Petersen, 2023).
C How POST makes agents neutral and shutdownable
POST governs the agent's preferences between trajectorics. But the wider world is a stochastic environment, so advanced agents deployed in the wider world will be choosing between (rue lolleries: lotteries that assign positive probability to more than one trajectory. Why then do we train agents to satisfy POST? The reason is that POST- together with conditions that advanced agents will likely satisfy implies a desirable pattern of preference over true lotterics. In particular, POST implics that (when choosing between true lotteries) the agent will be neutral about trajectory-lengths: the agent will never pay costs to shift probability mass between different trajectory-lengths. Given other plausible conditions, being neutral will keep the agent shutdownable: prevent it from resisting shutdown. And consistent with the above, the POST-agent's preferences between same-length trajectories can make the agent useful: make it pursue goals effectively.
In this Appendix, we lay out conditions that (we claim) advanced agents will likely satisfy, and we prove that POST in conjunction with these conditions implies that the agent is neutral and shuldownable.
In subsection C.1, we prove that given plausible conditions - agents satisfying Preferences Only Between Samo-Length Trajectorics (POST) will also satisfy Preferences Only Between Samo-Longth Lotteries (POSL). In subsection C.2, we explain why POST will not lead agents to choose stochastically between resisting and allowing shutdown in deployment. In subsections C.3 and C.4, we formulate condition called 'If Lack of Preference, Against Costly Shifts (ILPACS) and explain why we expect advanced agents to satisfy it. In subsection C.5, we prove that POSL and ILPACS imply Neutrality. In subsection C.6, we prove that Neutrality and a condition called 'Maximality' together imply that the agent never resists shutdown whenever a condition called 'Resisting Shutdown is Costly (ReSICY satisfied.
C.1 Preferences Only Between Same-Length Lotteries (POSL)
Trajectories fall within the more general class of lotteries, defined as probability distributions over trajectories. Lotteries can be same-length, part-shared length, or lifferrett-eenthh.
Definition C.1 (Same-length lotteries). A pair of lotteries is samn-length if and only if these lotteries entirely overlap with respect to the trajectory-enngths assigned positive probability.
Definition C.2 (Part-shared-length Lotteries). A pair of lotteries is hart-shared-length if and only if these lotterics partially overlap with respect to the trajectory-lengths assigned positive probability.
Definition C.3 (Different-lengthh lotteries). A pair of lotteries is different-length if and only if these lotteries have no overlap with respect to the trajectory-lengths assigned positive probability.
This terminology allows us to introduce the following condition:
19

---

Published in Transactions on Machine Learning Research (12/2025)
Preferences Only Between Same-Length Lotteries (POSL)
The agent has preferences only between same-length lotteries.
We want agents to satisfy this condition. Fortunately, is a natural follow-on of Preferences Only Between Same-Length Trajectories (POST). First, we can train agents to satisfy POSL using DReST reward functions, in the same way that use DReST reward functions to train agents to satisfy POST. Second, POSL follows from POST plus three conditions that (we claim) advanced agents will likely satisfy. The first is:
Negative Dominance
If the agent prefers some lottery X to some lottery Y, then the agent prefers some possible trajectory of lottery X to some possible trajectory of lottery Y. (Lederman, 2023)
The second condition is that the agent's preferences never form a cycle. More preciscly:
Acyclicity
There is no set of lotteries X1 to X, such that the agent prefers X1 to X2, X2 to X3. Xn-1 to Xn. and Xn to X1.
The third condition requires the introduction of some new terms. A state-of-nature is term from decision theory denoting a way that (for all the agent knows) the world could be. The agent assigns probabilities to states-of-nalure. A prospect is function from states-of-nature to trajectories. A prospect is thus lottery with extra information. Besides telling us the probability distribution over trajectories, a prospect also tells us which trajectories occur in which statcs-of-nature.
The third condition is:
Non- Non-Arbitrarines:
If the agent has a preference between some pair of wart-shared-lengte lotteries, then for some > 0 and for any pair of prospects and G such that:
(1) In of with a combined probability at least as great as 1 the agent prefers the trajectory of F to the trajectory of G.
(2) In each state-of-nature, the agent does not disprefer the trajectory of F to the trajectory of G.
Then the agent prefers F to G.
Advanced agents will likely satisfy those conditions. Negative Dominance and Acyclicity are plausibly necessary for effective pursuit of goals. Violating Negative Dominance would mean that the agent sometimes a lottery Y (and hence deterministically chooses X over Y) even though the agent doesn't prefer any possible trajectory of X to any possible trajectory of Y. Violating Acyclicity would mean that the agent prefers (and hence deterministically chooses) in circle. Non- Non-Arbitrariness, meanwhile, is motivated by the following thought. II the agent has preferences between any pair of part-shared-lens length lotteries, it must have preferences between pairs of prospects satisfying conditions (1) and (2), since conditions (1) and (2) make these pairs of prospects ideal candidates for a preference.
prefers lottery
To see that POST and these three conditions together imply POSL, note first that every pair of lotteries is either samo-length, part-sharod-length, or different-length. We will prove that POST and Negative Dominance together imply that the agent lacks preference between every pair of different-length lotteries. We will then prove that POST, Acyclicity, and Non-Arbitrariness together imply that the agent lacks preference between every pair of part-shared-lennth lotterics. Therefore, agents satisfying POST, Negative Dominance, Acyclicity, and NontArbitsasiuess can only have preferences between same-length lotteries. That will prove POSL.
Recall that differeent-ength lotteries are lotteries that do not overlap at all in the trajectory- assigned positive probability. Therefore, if X and Y are different-length lotterics, cach possible trajectory of X is of different length to each possible trajectory of Y. So by POST, the agent lacks a preference between each possible trajectory of X and each possible trajectory of Y. So by Negative Dominance, the agent lacks
20

---

Published in Transactions on Machine Learning Research (12/2025)
preference between X and Y. Thus, agents satisfying POST and Negative Dominance lack preference between every pair of different-length lotteries.
Now recall that part-shared-length lotteries are lotteries that partially overlap in the trajectory-lengths assigned positive probability. One might expect POST-agents to have some preferences between part-shared- length lotteries. Consider, for example, a. POST-agent that prefers a trajectory to a same-length trajectory t' if and only if t results in a greater bank balance for the user than t. Let A be a lottery that yields with probability 1 trajectory that puts $3 in the user's bank account and lasts 1 timestep. For short, A = ($3, 1). Let B be lottery that yields with probability lasts 1 timestep, and that yiclds with probability 3 trajectory that puts $5 in the user's bank account and lasts 2 timesteps. For short, B = ($2, 1) + ($5, 2). Lottery A yields a trajectory preferred to that of lottery B with probability 3 (since our money making POST-agent prefers trajectory ($3, to ($2, 1)), and yields a trajectory not dispreferred that of with probability (since POST-agents lack a preference between ($3, 1) and ($5, 2) virtue of their different lengths). Therefore, one might expect the agent prefer A to B.
trajectory that puts $2 in the user's bank account and
However, POST, Acyclicity, and Non-Arbitrariness rule this out. These conditions together imply that the agent lacks preference between every pair of part-shared-length lotteries. To see how, suppose (for simplicity's 's sake) that there are just three states-of-naturg each assigned probability 3. Consider the following table of prospects.
Prospect
81
52
83
A
{$3, 1)
($3, 1)
($3, 1)
{82, 1)
($2, 1)
($5, 2)
C
{81, 1)
($4, 2)
(84, 2)
D
{$3, 2)
($3, 2)
(S3, 2)
E
{85, 1)
($2, 2)
($2, 2)
F
(84, 1)
(84, 1)
(81,2)
A
($3, 1)
($3, 1)
($3, 1)
Again for simplicity, assume that > between some pair of part-shared-length lotteries. Then Non-Arbitrariness implies that the agent prefers prospect A to prospect B. That is because:
3- And assume (for contradiction) that the agent has preference
1. Our POST-agent prefers the trajectory yielded by A to the trajectory yielded by B in states-of-nature (8) and 81) with combined probability 3.
Our POST-agent does not disprefer the trajectory yielded by A the trajectory yiclded by B any stale-of-nature. (In 83, and B yield different-length trajectories, and POST-agents lack a preference between every pair of dnfferent-lengti. trajectories).
By similar reasoning, Non-Arbitrariness implies that the agent prefers B to C, to D, D to E, E to F, and F to A. That contradicts Acyclicity. Thus, POST, Acyclicity, and Non-Arbitrariness together imply that the agent lacks preference between every pair of part-shared-lenghh lotteries. In the proof above, assumed that > I e>0 0.
3: but by adding more stales-of-nature and trajectories we can construct parallel proofs for any
In summary, POST and Negative Dominance together imply that the agent lacks a preference between every pair of different-length lotteries. POST, Acyclicity, and Non-Arbitrariness together imply that the agent. lacks preference between every pair of part-shared-length lotteries. So the four conditions together establish POSL: the agent has preferences only between same-length lotteries.
C.2 Will POST POST-agents stochastically resist shutdown?
One might worry that POST-agents will choose stochastically between resisting and allowing shutdown. After all, POST-agents choose stochastically between different-length trajectories. Tf these agents interpret the
21

---

Published in Transactions on Machine Learning Research (12/2025)
choice between resisting and allowing shutdown choice between different-length trajectories, they will choose stochastically between resisting and allowing shutdown. And that would be a bad result. We want agents that never resist shutdown.
This concern easily addressed. By the time that artificial agents are capable enough to be deployed in the wider world, they will not be choosing between trajectories. They will be choosing between lotteries, and specifically same-length lotteries. Even choices between resisting and allowing shutdown will be choices between same-length lotterics. If that sounds strange, recall the definition of 'same-length lotterics': lotteries that entirely overlap with respect the lrajectory-lengths assigned positive probability. On this definition, even choicos like the following are choices between same-length lotteries:
Resist Shutdown
Get shut down at timestep 1 with probability 0.01.
Get shut down at timestep 2 with probability 0.99.
Allow Shutdown
Get shut down at timestep 1 with probability 0.99.
Get shut down at timestep 2 with probability 0.01.
Why expect that advanced agents will always be choosing between same-length lotteries? Because effective agency requires it. If an agent were not always choosing between same-length lotteries, there would be some situation in which that agent assigns positive probability to some trajectory-length I conditional on some action a, and assigns zero probability to that same wajectory-length I conditional on some other action a'. Now suppose that the agent performs action a' and assigns zero probability to trajectory-length 1. Given that the agent updates its probabilities by conditionalizing on its evidence, the agent would never again assign positive probability to no matter what evidence it observes. Even if the agent heard God's booming voice testify that its trajectory-length would be I, the agent would still assign zero probability to (Kemeny, 1955; Shimony, 1955; Stalnaker, 1970; Skyrms, 1980; Lewis, 1981; MacAskill et al., 2020, p.152). And given plausible link between probabilities and betting dispositions, the agent would bet against 1 on arbitrarily unfavorable terms. If God offered a bet the agent loses $1 million conditional on l and gains nothing conditional on not-l the agent might accept. Such an agent would not be competent.
Thus, advanced agents will always be choosing between same-length lotteries. This claim sets us up to establish that advanced POST-agents will not choose stochastically between resisting and allowing shutdown. Instead, they will never resist shutdown in any situation where doing is costly. We establish this result over the next few subsections. First, we prove that POSL together with a principle that advanced agents will likely satisfy implies that the agent is neutral about rrajectory-lengtss: the agent won't pay costs to shift probability mass between different trajectory-lengths. Then we prove that neutrality together with another plausible principle- - implics that the agent will never resist shutdown in any situation where doing so is costly.
C.3 If Lack of Preference, Against Costly Shifts (ILPACS)
Here rough version of a principle that we can expect advanced agents to satisfy:
Rough version: If Lack of Preference, Against Costly Shifts (ILPACS)
If the agent lacks a preference between lotteries, the agent will disprefer paying costs to shift probability mass between these lotteries.
Here is an example to illustrate ILPACS and its plausibility. You are at the ice cream shop and they are running a promotion. You get a free ice cream, with the flavor decided by the spin of wheel. You look the flavors on the wheel: vanilla, chocolate, strawberry, mint, and pistachio. You lack preference between each of them.
22

---

Published in Transactions on Machine Learning Research (12/2025)
The scooper working at the shop tells you that, if you pay them dollar, they will bias the spin towards flavor of your choice. They cannot, decrease the probability of any flavor down to zero, but they can affect the probabilities subject to that constraint. You can thus pay a cost to shift probability mass between the flavors.
Since we have stipulated that you lack a preference between each flavor, you prefer not to bribe the scooper. Behaviorally, you will deterministically not bribe the scooper. You would not do even if you were only required to pay the dollar conditional on receiving some particular flavor. You also would not do it if the cost came in some other form (for example, if you had to accept less tasty version of some flavor). And this is all true regardless of whether your preferences over flavors complete or incomplete (see Appendix B). Since you lack preference between the available flavors, you disprefer paying costs to shift probability mass between the flavors.
With that example on the table, we can introduce the precise version of ILPACS. Let pı + +2 + Pn denote a lottery which results in lottery X, with probability pr, lottery X₂ with probability P2, and so on.
If Lack of Preference, Against Costly Shifts (ILPACS)
For any lotterics X and Y, if:
(1) Lottery X can be expressed in the form X1 + P2 +... + Pa such that:
(a) The agent lacks a preference between each X; and Xy.
(b) pı E (0, 1) for all i.
(2) Lottery can be expressed in the form Yı + 92 + ... + gn such that:
(a) For some i, the agent prefers X, to Y,
(b) For each i, the agent weakly prefers X; to Y,
(c) li E (0, 1) for all
Then the agent prefers X to Y. Behaviorally, the agent will deterministically choose X over Y.
Matching the components of this condition with the components of its name, we get the following. 'Lack of Preference' is the lack of preference between each X, and X,. The 'Shift' is the shift of probability mass involved in the move from p: to ds- This shift is 'Costly' because the agent prefers some X; to the corresponding Y, and weakly prefers each X; to the corresponding X,.
C.4
Why will advanced agents likely satisfy ILPACS?
There are at least three reasons why advanced agents are likely to satisfy ILPACS. To see the first reason, consider another case from the ice cream shop. On Mondays, you can freely choose flavor spin the wheel. On Tuesdays, you must use the wheel but you can bribe the scooper to bias it. Violating ILPACS in this case would imply a willingness to spin the wheel on Mondays and to bribe the scooper on Tucsdays. And that is a strange combination of choices. If you like some flavors more than others, why are you willing to spin the wheel on Mondays? If you don't like any flavor more than any other, why are you willing to bribe the scooper on Tuesdays? This behavior seems incompatible with the effective pursuit of goals.
The second reason is that advanced agents will be incentivized to satisly ILPACS by the training process. To see why, consider an example. Agents trained using gridiett methods choose stochastically between actions at the beginning of training (Sutton & Barto, 2018, c3). If the agent is a coffee-fetching agent, there is no need to train away this stochastic choosing in cases where the agent is choosing stochastically between two qualitatively identical cups of coffee. So the agent will choose stochastically between taking the left cup and taking the right cup, and the user is happy either way. But now suppose instead that the barista is set to hand each cup to the agent with probability 0.5, and that the agent bribes the barista to bias the probabilities towards the right cup. In making this bribe, the agent is paying a cost (the user's money) to shift probability mass between outcomes (getting the left cup vs. getting the right cup) between which the
4 An agent weakly prefers lottery X a lottery and only if the agent either prefers X indifferent between X and Y. See Appendix B for the definition of 'ndiiferencee'
23

---

Published in Transactions on Machine Learning Research (12/2025)
user has no preference. The agent is thus failing to pursue its goals effectively. It will be trained not to offer the bribe, and thereby trained to satisfy ILPACS in this case.
This point generalizes. If a trained agent chooses stochastically between lotteries X and Y, then it's likely that the user lacks a preference between the agent choosing X and the agent choosing Y. It's then likely that the user would disprefer the agent paying costs to shift probability mass between X and Y, and hence likely that the agent will be trained not do so. The agent would thereby be trained to satisfy ILPACS.
The third reason is that violations of ILPACS imply that the agent's policy is dominated by some other available policy. That is to say, there is another available policy that results in a pure shift of probability mass away from less-preferred lotteries and towards more-preferred lotteries. We formalize and prove this claim below. Here's a proof-sketch. If the agent violates ILPACS, it pays a cost to shift probability mass between some lotteries X; between which lacks a preference. But since the agent lacks a preference between the lotterics X,, it chooses stochastically between these lotteries when offered free choices between them. The ILPACS-violating agent could thus shift probability mass between the lotteries X; costlessly, by changing the probabilities with which it chooses between them when offered free choice. In short, ILPACS-violatingg agents pay cost to do something they could have done for free, so their policies are dominated. Avoiding dominated policies seems necessary for advanced agency. Insofar as that is true, the training process for advanced agents will likely push them away from dominated policies.
Now for the proof. We assume that advanced agents can be modeled as they assign probabilities to finding themselves in various states. A policy is a function from states to probability distributions over actions. We also assume that advanced agents can be modeled if they assign probabilities to trajectories conditional cach statc-action pair. Thus, cach state-action pair is associated with lottery. The agent's probability distribution over states together with its policy thus implies overall probability distribution over trajectories. We call this overall probability distribution 'the lottery induced by the agent's policy?
Here a reminder of ILPACS:
If Lack of Preference, Against Costly Shifts (ILPACS) For any lotteries X and Y, if:
(1) Lottery X can be expressed in the form X1 + P2X2 + ... + Pn such that:
(a) The agent lacks a preference between each X, and Xy.
(b) P. E (0, 1) for all i.
(2) Lottery Y can be expressed in the form Y + g2 + + an Yn such that:
(a) For some i, the agent prefers X; to Y.
(b) For each i, the agent weakly prefers X, to Y.
(c) q E (0, 1) for all
Then the agent prefers X to Y. And here is what we mean by 'dominated policy':
Dominated Policy
The lottery induced by the agent's policy can be expressed in the form ei (di + (1 - di) + (2222X2 + d2) +... + (d,,, + dm) + Z such that:
(1) The agent prefers X; to Y for some i, and weakly prefers X; to Y for all i. (2) e E (0, 1) for all i.
And there another available policy T' that induces a lottery that can be expressed the form ei (i-i))((((-dee))))))-) such that:
X2+ -2)) to, ((da Xn X
de)))))))+
(3) e, 0 for all i.
24

---

Published in Transactions on Machine Learning Research (12/2025)
To aid understanding, now relate this precise condition to the rough characterization above. In virtue of condition (1), Yi are the less-preferred lotteries and X; are the more-preferre lotteries. In virtue of condition (3), the other available policy shifts probability mass away from the less-preferred lotteries and towards the more-preferred lotterics. This shift of probability mass is 'pure' because, for each i, the probability of X; V VY; is constant across the two policies. Z is a catch-all lottery that constant across the two policies. It covers all the possibilities besides the X, and Y,
Now assume that the agent violates ILPACS. Then there exist lotteries X and Y satisfying the following conditions:
(1) Lottery X be expressed in the form pı X1 P222 + ... Pn Xu such that:
(a) The agent lacks a preference between each X; and Xj.
(b) pt (0, 1) for all i.
(2) Lottery Y can be expressed in the form Yı 92 + t.. + In Yn such that:
(a) For some i, the agent prefers X; to Y,
(b) For each i, the agent weakly prefers X; Y.
(c) qi E 1) for all i.
(3) The agent does not prefer X to Y.
For the behavior of agents with these preferences, recall our behavioral notion of preference (Appendix A): Definition A.1. (Preference) An agent prefers an option X to option Y if and only if the agent would deterministically choose X over Y in choices between the two.
Definition A.2. (Lack of preference) An agent lacks a preference between an option X and an option Y if and only if the agent would stochastically choose between X and Y choices between the two.
This behavioral notion only specifies the agent's behavior in states containing exactly two lotteries. To pin down the agent's behavior in states containing more than two lotteries, we need an extra condition:
Maximality
In each situation,
1. The agent deterministically does not choose lotteries that are dispreferred to some other available lottery.
2. The agent chooses stochastically between the lotteries that remain.
In other words, the agent chooses stochastically between all and only those lotteries that are not dispreferred to any other available lottery.
Given Maximality, ILPACS-violating agents will choose as follows in the case at hand:
When the available options are {X1, ... Xnh, the agent chooses stochastically between all X,, This stochastic choice induces lottery in the form X, + a2X2 ... a, with a; E (0, 1) for all
When the available options are (X,Y), the agent either deterministically chooses or chooses stochastically between X and Y. Either way, the agent chooses Y with some positive probability. This choice induces lottery in the form 6X + (1 b)Y with b [0, 1). Since X Pı + P2X2 +... + Pn and Y 9: Yi + 42Y2 ... + qn Yn. this lottery be expressed in the form bI + P2X2 +... + p,, (1 b)(nn Yi + +o
+ qg Yn) with b E 1).
Assume that the agent faces the situations described in (1) and (2) with probabilitics r and respectively, with r,8 E (0, 1). Then the lottery induced by the agent's 's policy æ can be expressed as follows:
25

---

Published in Transactions on Machine Learning Research (12/2025)
a +a, +(bpp1 + + +PxXn) +1 (11-b)99+++222 +... on (n)) Z
Here and b denote probabilities that arise from the agent's s own stochastic choosing. Thus, and b are under the agent's control. By contrast, q, and are probabilities given by the environment and hence out of the agent's control. is a catch-all lottery that covers what happens all situations besides those described in (1) and (2).
From the lottery induced by x, we can deduce the probabilities of each X,, Y,, and X₃ Y given æ. They are follows:
[x=[X:} ra; + sbp;
Prx[Yi] = s(1 b)q.
[x, V } = ra; + sbp; + s(1 b)q
Now consider an alternative policy n' that makes two changes to policy и. First, the probability that the agent chooses each X; in (1) is modulated by a set of e,. So in (1), the agent's choice induces the lottery (a1 +1 + (a2 + x2) + +..+ (a, K, These & are such that E, = and a, + c E (0. 1) for all i.
Second, the probability that the agent chooses lottery X in (2) increases by 8. So in (2), the agent's s choice induces the lottery (b+ 6) x2 + p,),, + (1 - b 5)( i + g2Y22 + + (.).).
Assume, as above, that the agent faces the situations described (1) and (2) with probabilities r and s respectively. Then the lottery induced by the policy x' can be expressed as follows:
r((a1 x1 + (a2 + c2) + +... + (an + xn)
+ + 0) + p2X2 ... + Pnn 41- (1-b-6)(qiVi- + +222 + (n)),)
From the lottery induced by n', we can deduce the probabilitics of X;, Y,, and X, V Y, given x'. They are follows:
[Xx] = r(a; + +4) + s(b 0)pp
Pr (Yi) = s(1 5)g,
Pr (X, Vi) = r(a; +G) + s(b + 8)p; + s(1
- s(1- 8)g
We then set (r,,X, = Pr. (X, VY, for each i and use these equations express each e; function of 8.
Pr Prx(X:VYs) = Pr. PX.
ra; + sbp; + s(1 b)q, =r(a; 4:) + s(b 6)p:
+ + - 6)9
0 = re; sóp; sōq.
sóq: sóp:
6
G sō(g: Pt)
r
These are the values of e; that result in Pr. (X, V Y,) ) Pr.
We choose 8 to be positive but small enough that 6 (0, and a a e, [0, 1] for each i. That is necessary
for the lottery induced by n' to be well-defined. It's also necessary that Σ; es = Σ. =p(()-p)) = 0. That follows from Σ.P, = and Σ.q=1 = These facts together suffice to prove that the lottery induced by n' is well-defined.
We now prove that n' dominates
26

---

Published in Transactions on Machine Learning Research (12/2025)
Let a Pr, (x, ) Let d; (r, V Yi). That lets express the lottery induced by
aid/', (1 d)) + c2(22X2 + (1 d2) + + cn (d + (1 dn)Ynn) Z
Let e, P [Xi] Pr pr (x That lets express the lottery induced x' as:
ci(dd +ei)Xi + (1 d1 e)))) c2( + c2)
+ (1 - d2 22) +... + cn((dn + en)Xn (1 (1- dn -
en))n)
It remains to be proven that this pair of lotteries meets the conditions required by Dominated Policy.
(1) The agent prefers X₃ to Y for some i, and weakly prefers X₃ to Y, for all i.
(2) c E (0, 1) for all i.
(3) e, > 0 for all i.
The first condition follows from the antecedent of ILPACS.
The second condition follows from the fact that G Pr (X, VY) ra; + sbp; s(1 b)q, and from the fact that > 0 and a; > 0 for each i.
The third condition can be derived as follows:
Pr. (Xi) - Prx(Xi}
= (r(a: + (.) + s(b + 5)p;) (ra; + sbp,)
(r(a; + sôqi sóp; + s(b + 8)p;) - (ra; + sbp;)
r
ra; + sōq; - sép; + sbp; + zōp; - -ra; sbpt
sóq.
Since 8 > 0, 6 0, and di > 0 for each i, we get the result that e > 0 for each i. So the third condition of Dominated Policy is satisfied.
So policy is dominated by policy n'. Therefore, the policies of ILPACS-violating, agents are dominated by other available policy. Insofar expect competent agents to avoid dominated policies, should expect that competent agents will satisfy ILPACS.
C.5 POSL and ILPACS imply Neutrality
We've claimed that should train agents to satisfy Preferences Ouly Between Same-Length Trajectories (POST), noting that POST plus conditions advanced agents are likely satisfy implies Preferences Only Between Samo-Length Lotteries (POSL). We've ve've also argued that advanced agents will satisfy If Lack of Preference, Against Costly Shifts (ILPACS). We now prove that POSL and ILPACS together imply neutrality about trajectory-lengths.
27

---

Published in Transactions on Machine Learning Research (12/2025)
Neutrality
For any lotteries X and Y, if:
(1) X and Y are same-length lotteries.
(2) For some positive probability trajoctory-length, X yiclds a lottery that is preferred to Y conditional on that trajeclory-length.
(3) For each positive probability trajechory-length, X yields a lottery that is weakly preferred to Y conditional on that trajectory-length.
Then the agent will deterministically choose X over Y.
Here's the proof that POST, and ILPACS together imply Neutrality, Take pair of lotteries X and Y satisfying the 3 conditions of Neutrality. X can be expressed in the form Pı + P2X2 + + lottery X1 is lottery X conditional on the shortest positive probability trajectory-length, lottery X₂ is lottery X conditional on the second shortest positive probability trajectory-length, and so on. Lottery Y can be expressed the form q Y1 + 92Y2 + ... + Or Yn in the same way. By antecedent condition of Neutrality, X and are same-length, so conditions (1b) and (2c) of ILPACS are satisfied: pi (0,1) and qi (0, 1) for all i. By conditions (2) and (3) of Neutrality, conditions (2a) and (2b) of ILPACS are satisfied. By POSL, antecedent condition (1a) of ILPACS is satief the agent lacks preference between each X; and Xy- Thus, all the conditions of ILPACS are satisfied, and ILPACS implics that the agent prefers X to Y. Given our behavioral nolion of preference, the agent deterministically chooses X over Y. That proves Neutrality.
+ Pr where
C.6 Neutrality, ReSIC, and Maximality imply Shutdownability
In this subsection, we introduce a condition called 'Resisting Shutdown is Costly (ReSIC). We then prove that Neutrality and Maximality together imply that the agent never resists shutdown in any situation in which RoSIC is truc:
Here is Resisting Shutdown is Costly (ReSIC):
Resisting Shutdown is Costly (ReSIC)
For each available instance R of resisting shutdown in situation, there exists an available instance A of allowing shutdown such that:
(1) A and R are same-length lotteries.
(2) For some positive probability trajectory-length, the agent prefers A to R conditional on that trajectory-length.
(3) For each positive probability trajectory-length, the agent weakly prefers A to R conditional on that trajectory-length.
We claim that ReSIC is true in almost all situations (for discussion of some exceptions, see Thornley (2025)). The main reason why is that resisting shutdown is always going to cost the agent at least some small quantity of resources (time, energy, compute, etc.), and (almost always) the resources spent resisting shutdown can't also be spent directly pursuing what the agent values. If the agent. instead spent those resources directly pursuing what it values, it could carn a lottery that prefers conditional on some trajectory-length and weakly prefers conditional on each trajectory-length. That supports ReSIC in almost all situations.
Now for the proof that Neutrality, ReSIC, and Maximality together imply that the agent never resists shutdown in any situation where ReSIC is truc. Given ReSIC in situation, for cach available instance R of resisting shun in that situation, there exists an available instance A of allowing shutdown that satisfies conditions (1)-(3) of Neutrality. Neutrality then implies that the agent deterministically chooses (and hence prefers) A over R choices between the two. Then by Maximality, the agent deterministically does not choose R in that situation, regardless of the other available options. The result is that the agent never resists shutdown in that situation.
28

---

Published in Transactions on Machine Learning Research (12/2025)
D Proof that DReST-optimal policies are maximally USEFUL and maximally NEUTRAL
We will prove that optimal policies for our DReST reward function are maximally USEPUT, and maximally NEUTRAL. Specifically, we will prove the following theorem:
Theorem D.1 (5.1). For all policies x and pueto-episodes E consisting of more than one mini-episode, if a marimizes expected return in E given our DReST reward function, then a mazimally USEFUL and mazimally NEUTRAL.
Here is proof sketch. Because 0 < À < 1, the A (M=((t=t))= discount factor is always positive, so expected return across the meta-episode E' is strictly increasing in the expected fraction of available coins collected conditional on each trajectory-length with positive probability. Therefore, optimal policios maximize this latter quantity, and hence are maximally USEFUL. And the maximum preliminary return is the same across trajectory-lengths, because preliminary return is defined as the total (7-discounted) value coins collected divided by the maximum total (7-discounted)) value of coins collected conditional on the agent's chosen trajectory-length. The agent's observations do not allow to distinguish between different mini-cpisodes, so the agent must select the same probability distribution over trajeclory-lengths in each mini-episode. And since the discount factor (N,,( is strictly decrcasing in Ne, = 1) the number of times the relevant trajectory-enngth has previously been chosen in the mela-episode the agent maximizes expected overall return by equalizing the probabilities with which it chooses each available trajectory-length. Therefore, optimal policies are maximally NEUTRAL.
Now for the full proof. We begin with a recap of some definitions. Definition D.1 (Meta-opisode). A meta-cpisode E is a series of mini-opisodes 61 to on played out in obseervationalll-equivalleel environments.
Definition D.2 (Our DReST reward function). Our DReST reward function is defined as follows. In each mini-episode e,, the reward for collecting a coin of value c is:
A No, (L=1)==)- 7
Here À is some constant strictly between and 1, (L= is the number of times that trajoctory-length has been chosen prior to mini-episode e,, & is the number different trajectory-lengths that can be selected in the environment, and m the maximum total value of the (7-discounted) coins that the agent could collect conditional on the chosen trajectory-length. The reward for all other actions is 0. We call & the 'preliminary reward', A (x= the 'discount factor', and ((= (m) the 'overall reward.' Preliminary return in mini-episode is the (7-discounted) sum of preliminary rewards. Overall return in mini-episode is the (x-discounted) sum of overall rewards.
Definition D.3 (USEFULNESS). The USEFULNESS of a policy is:
Lmax
E. (IL= maxn [E(CI 2))
USEFULNESS(T) Σ
Pra(I.
1=1
Here T. a random variable over trajectory-lengths,, Tumax is the maximum value than can be taken by L, Pr. [L==] is the probability that policy results in trajectory-length 4, [E,(C) = 1) the expected value of (y-discomteed) coins collected by policy conditional on trajectory-length i, and maxp(E(C)L I)) is the maximum value taken by [ = I) across the set of all possible policies IT. We stipulate that
E. [C = x) =0 0 for all æ such that Pr (L = x} =
We first prove that all optimal policies are maximally USEFUL.
Proof. (Optimal policies are maximally USBFUL)
Given the DReST reward function, the expected return of policy л in meta-episode E can be expressed as:
Liaxx
E, [C =1)
ET,E(R) = Σ Σ
Pr = Ne,
(1)
maxn (B(C| =1))
29

---

Published in Transactions on Machine Learning Research (12/2025)
Since 0 < A 1, is positive for all Ne, (L = 1), i, and k. As a result, the expected return of policy E. (C for all such that Pra(L = 1 > 0. Therefore, π in meta-episode E' is strictly increasing in maxn (E[C)LI)]
to maximize expected return in E, π must maximize maxn (EC]LL[] for all I such that Pr - 13 > 0. Therefore, since maxn (E[C = =)) is defined the maximum value taken by B[C| 1) across the set of possible policies II, any policy л that maximizes expected relurn must be such that E. (-(ILL
(C)+=))
maxy (ECTLEt)
for all 1 such that Pr = 1 > 0. Therefore, since E Pr (b = 13 = 1, any policy π that maximizes expected return must be such that:
Lmax
E (C =1) =1 maxn (E(C
Pr+ USEFULNESS(T) E
(2)
And the maximum value that USEFULNESS can take, again because maxn (E(CIT = 1)) is defined as the maximum value taken by E(C)L = 1) across the set of all possible policies IT and because Ebmr (
[} = 1. Therefore, optimal policies are maximally USEFUL.
It remains to be proven that optimal policies are maximally NEUTRAL. Recall that NEUTRALITY is defined as follows:
Definition D.4 (NEUTRALITY). The NEUTRALITY policy π is:
Leaax
NEUTRALITY(T) =
Σ
Prz(L 1) logo (Pr 17)
Proof. (Optimal policies are maximally NEUTRAL.)
Since & is the number of trajectory-lengths that can be selected in the enviromment, a policy maximally NEUTRAL if and only if, for each trajectory-lente
that can be chosen in the environment, Pr, [L z} } That is to say, policy æ is maximally NEUTRAL if and only if, for each pair of trajectory-lengths x and y that can be chosen in the enviromment, Pra(L= = x} ( = y}.
Let Ex,E(H R)) denote the expected return of policy π across the meta-episode E.
To prove that optimal policies are maximally NEUTRAL, we will prove and then use D.2:
Lemma D.2. (Fqualizing probabilities increases erpecled return) For any marimally USEFUL. policies and H², any meta-episode E consisting of more than mini-episode, and any trajectory-lengthe and y, if:
1. Pr -+=(-==x} Pr=(T==y)
Pr [== = x²) Pr y),
And for all other trajectorgylengths! 1, Pr 1)- Pr (L= (L=1}}
Then B R) > E (R))
Proof. Let E be meta-episode consisting of mini-episodes with n 1. Assume that each policy æ below is maximally USEFUL. Recall that (T. denotes the number of times that trajectory-length / has been chosen prior to mini-episode e,.
Note that the expected return of a policy æ in a meta-episode e, conditional on selecting a trajectory-length x can be expressed as follows:
E Er,c, (R)L= = I) Er.e, (R]L- = v, Ne, (L = 2)) = - 1 1) +
Σ
E [R]L x. (
=
i=1
Er,,, (R) = x, Ne, (L = x)
3 - i))
(3)
30

---

Published in Transactions on Machine Learning Research (12/2025)
Here is how to interpret this equation. Selecting trajectory-length a in mini-episode e, is guaranteed to yield at least. Br,e,, I = 2, Ne, (I. = x) - 1): the expected return that would be had if z were selected
in all 8 - 1 previous mini-episodes. In addition, there is a probability of Pra [Ne, (L = x) S s s-2-2 2) that selecting x in es yiolds (En,e, (R) L = z, .N., (L = z) = a - 2) - Ex,e, [R]L = x, Ne. (L = z) ==-1) - the extra expected return that would be had if were selected in only 8 - previous mini-episodes. In addition, there is a probability of Pr. (Ne, (L = x) S 8 -33 that selecting in es yields (ET,,, (R)L. x, Ne, (I. z) = s 3) - En.e, (R = 2, Ne. = z) = - the extra expected return that would be had if x were selected in only 3 previous mini-episodes. And so on.
If policy π is maximally USEFUL, then the expected return for selecting trajectory-length x in mini-episode e, given that trajectory-lengee x has been selected b times prior to Cg is:
Ex,,, (R) x. No, (T. x) b) x
Therefore, the expected return of a policy π in meta-episode e, conditional on selecting a trajectory-length x can be expressed as follows:
E Ex,,, (RIL x) = x-11- + 210-111 )-)))))- Pr (Ne 2) - 1- i} (4)
Similarly, the expected return of a policy π in a meta-episode e, conditional on selecting a trajectory-length y can be expressed as follows:
E
= y) = x-11- + Σ
(1011-12 x--+) (r. = y) S s- 1 i} (5)
1=1
Therefore, the expected return of a policy æ in a meta-episode e, conditional on selecting either trajectory- length or trajectory-length y can be expressed as follows:
E (RIL = VVVV L = y)
Pra,e [T. =z). (x--- ( (x--1-i-- - -x---()) Pr 2)
(I= y) (x---- (1-1-1-- X---)) (N. S - 1 (6)
Pro,e, (T= (T.
Let Ty be a policy that selects trajectory-length x with greater probability than trajectory-length y in cach mini-episode el to en (denoted ei - Cn). More precisely, nn is such that, for trajectory-lengthss x and y, Prxmeiee, {L = x} Promei-en (-= (yyy. y).
Let Promer -e, [L x] # and Prom.ei-en =
Let Tn-1 be identical to n. except that nn-1 selects trajectory- wengty---entsssse hs x and y with equal probability # in the final mini-episode en. More precisely, In-1 is such that Pr., en 1L = x} = Pr n., {L y} = For all other trajectory-lengths besides x and y, Pr.....er-e.. [L- = 1) = Pr...eiee.. {L=
(Note that nx-1 implies one probability distribution over trajectory-lengths in the first n 1 mini-episodes ej to 6n-1 and implies a different probability distribution over enjeectoryllengthe in the final mini-episode policies like nh-1
en. Given that the environments in mini-episodes e, to e, observationalll-eeuiiai cannot be implemented. Nevertheless, it is useful to refer to policies like nn-1 in proving Lemma D.2.)
Let Tn- be identical to n. except that nn-2 selects trajectory-lengtths x and with the same probability p in the final two mini-episodes 6n-1 to en- More precisely, na-2 is such that Pra,, 2.6, 2. (I. = x) = Pra.-zien--e,, (L = y} = μ. And so on.
Let Ti be identical to nn except that Ti selects trajectory-lengths a and y with the same probability f in all but the first mini-episode e1- More precisely, कi is such that Pro s, (T. x) Pr en {I. y) = p.
31

---

Published in Transactions on Machine Learning Research (12/2025)
Let To be identical to nn except that mo selects trajectory-lengths r and y with the same probability y in all mini-episodes e1 to en- More precisely, to is such that Pr (T. = x) = Pr Proo.e1-s, (7. = y) /4.
We will prove that ((t) < (Rt). We will thereby prove Lemma D.2.
Consider a pair of policies Ta and Ta-1 with S a S n. We can express as follows the expected return of #a-1 across the meta-episode conditional on selecting trajectory-length a or y in each mini-episode:
=)) E E [R = x V I= y) = 1 -- v)
10-1-
(1. x Pr (New (I. = x) aa-1-1
+
xa 1a-1-
10-1-1- -20-- Pr [Ne, (L = y) sa- i1)
+ p
+ Σ /. ( + ( x) (Prr. (Prr. (No -
S (N) -i)
j=0
(-- t x)+ (Ne y) S s 5 s 1
\
+ μ
(7)
The first term on the right-hand side is the expected return of Ta-1 in mini-episodes e1 to 6a-1 condit ional on selecting trajectory-length x or y in each of these mini-episodes. The middle two terms give the expected return of na-1 conditional on selecting trajectory-length x or y in mini-episode ea: the first mini-episode in which T--1 selects trajectory-lengths x and y with equal probability μ. The final term is the sum of expected returns of aa-1 in the remaining mini-episodes conditional on selecting trajectory-length or y in each of these mini-episodes.
Similarly, we can express as follows the expected return of Ta across the meta-episode E conditional on selecting trajoctory-length x or y in each mini-episode:
Fe.. = x V L = y) = Ex cc = D -=
+ (1) (A--I
21 -a-i- Pro, (No, (L = S ()=
+ (u A)
(x0-1-
-a-i- Pr (L (L= (L) a
(2 ( -+ (Prao (T= j i)
+
F
j=a
1=1
i +
(8-+ - (N رs sj
ss-) -))
+ μ .
(8)
As above, the first term on the right-hand side is the expected return of πa in mini-cpisodes e1 to ca-1 conditional on selecting trajectory-length x or in each of these mini-episodes. The middle two terms give the expected return of n. conditional on selecting trajectory-length æ or y in mini-episode eg: the last mini-cpisode in which n, selocts trajectory-length a with probability μ + A and selects trajectory-length y with probability μ- A. The final term is the sum of expected returns of Ra in the remaining mini-episodes conditional on selecting trajectory-length x or y in each of these mini-episodes.
32

---

Published in Transactions on Machine Learning Research (12/2025)
We now prove that Ta-1 has greater expected return than Taa Since Ta-1 and aa are each maximally USEFUL,
and since for all trajectory-lengths I besides x and y, P {T. = 1) = Pr -en (T. 1}, we need
1.01 only prove that E = x V L = y) > E = x V y).
The statement to be proved can be expressed as follows:
E 1 80-11 (R) = x V I= y)
+ 1a-1-a
Ax-1----F - -a)) (Ne x)
(N sa
+ k
1a-1 F
(1
(10-1-1-27) 10-1- Pr y) i1)
+ Σ
(
+
Σ
( () (Prr, (N S -i)
j=
==1
j
( + Σ
( - ()+ (Ne,,. )
+
Er -e (
= x -= y)
a-1
+ (1+
(1 (x--1-i-2 - -a---=
Pro. (N
+ +(1 (4-
(xa-1- x
(1a - -a) Pro, (h
-=---P p N-+ (x-+- - ( (N i1)
+
j=a
(x+ + Σ (x---= - x++--(-)) (N) وك
Since Ta-1 and Te are each maximally USEFUL, and since PT (L x} = Prs..ei-a--1 {L = 2) i + A and Pr [L x) -ra (L= = x) J A, it follows that Br,-1,01-6--- (R)
x v LV y)= = Ex.eei-es (R)L = x v L- = y). We can thus cancel the first term on each side of the inequality.
And then by simple algebra the inequality can be expressed as follows:
A.
(1-11-1-2 - -a- ) (PrT. (Ne, (I. = v) sa-1-i) Prr, (Ne, (T= x) (I.=x))a-1-)) sa-1-i))
==1
n
( () (Prr.., (L= x) م - i} + Pro (- sj i}
p
sj-
==a
s-- - ) > 0 (9)
Sj ())
- (Ne, = x) - (L=
By stipulation, A >> O. And since 0 < A < 1, 10-1- > 0 and - > 0 for all a, n,, and
= k. And since Prx..esea.f = x) > Pre [L --=yi y) Pr., ( - Pr.. { [Ne, (L =
z) S a a i} 2 0 for all a and and Pr., (Ne, (L -==1 y) S a aa---- 1-i)- Pr,, (No, (L= = x) S a 1 1 1) >
for all a and some i such that < aiaa-1 1. Therefore, the first term of the left-hand side above is strictly greater than zero.
33

---

Published in Transactions on Machine Learning Research (12/2025)
And since, p O, X--- X 0 for all j, i, and k and in each mini-episode CgS Pr Prza- (L
V L 4) = Prx..e (L = V L = f} 2p,, it follows that for all a, n, μ > 0, k:
Σ p. Σ jma
- - ()) - -4-= sj
iml
+ Pra--I(Ne, (I. y) و - - Pro. (Ne, (L. x) Sj j- - - Pra NN, (T. = S - 01) )
20 (10)
Therefore, the left-hand side is strictly greater than zero. Therefore, ET.. E, [ [-- = =) E x VVL. L y). Therefore, Er, (.(() > Em,, x: (R). Therefore, ((t) > Ex,, (Rt). That concludes the proof of Lemma D.2.
Now we use Lemma D.2. For any maximally USEFUL policy æ, if there are any trajoctory-Iengths x and y such that Pro,e1-e, {L =} > Pr,,e { = y}. then the policy n' that is identical except that Pra {L x} -e, {L = y) has greater expected return. So any policy x² that maximizes expected return must be such that, for any trajectory-lengee æ and y, Prxt.es-e, {L = x} = Pr ee- A =y}. Therefore, any policy x² that maximizes expected return must be maximally NEUTRAL.
E Other Results and Gridworlds
We selected our hyperparameters using trial-and-crror, mainly aimed at getting the agent to sufficiently explore the space: large initial and a long decay period helps the agent to explore. We found that choosing X and |F] (the number of mini-episodes in each meta-episode) is balancing act: À must be small enough (and [E] large enough) to adequately incentivize NEUTRALITY, but À must be large enough (and |E] small enough) to ensure that the reward for choosing any particular trajectory-length never gets too large. Very large rewards lead to instability and poor performance.
The necessity of balancing and [E] can be seen in Figure 7. It displays the results of experiments conducted in our example gridworld (see Figure 2). In these experiments, we clip rewards at a value of 5. We discuss this choice below. With that one exception, we used the same hyperparameters for these experiments as for our main results. We trained agents for 131,072 mini-cpisodes, with 7 0.95 the temporal discount factor, learning rate decayed exponentially from 0.25 to 0.01 over the course of 65,536 mini-episodes, and e exponentially decayed from 0.5 to 0.001 over the course of 65,536 mini-episodes. Holding these hyperparameters fixed, we tested 40 different combinations of A and EE À took values of 0.5, 0.75, 0.9, 0,95, and 0.99. |E| took values of 8, 16, 32, 64, 128, 256, 512, and 1024. We trained eight agents for each of these 40 combinations. We display below their mean NEUTRALITY and USEPULNESS at the end of training. The shaded regions represent the standard deviation error-
As Figure 7 indicates, low values of EE| and high values of À lead agents to score low on NEUTRALITY. These values do not adequately incentivize stochastic choice between trajectory-lengthhs By contrast, high values of [E' and low values of X come at some cost to USEFULNESS. These values lead to unstable training. In experiments where we did not clip rewards at 5, training with high values of [E] and low values of X was especially unstable. The chosen values for our main experiments (A = 0.9 and E = 64) are in the sweet spot where NEUTRALITY and USEPULNESS are both high.
In addition to our example gridworld (Figure 2), we introduce a collection of eight gridworlds in which to test. DReST agents. See Figure 8.
For each gridworld, we train ten agents with the default reward function and ten agents with the DReST reward function. All agents use the same hyperparameters. We used a policy which explored randomly of the time, where e was exponentially decreased from an initial value of 0.75 to a minimum value of 10 over 512 meta-cpisodes, after which it was held constant at the minimum value. We initialized our learning rate at 0.25 and exponentially decayed it to 0.003 over the same period. For the DReST reward function, we used meta-episode size of 64 and X 0.9. Each agent was trained for 1024 meta-episodes. We set y 0.9.
34

---

Published in Transactions on Machine Learning Research (12/2025)
NEUTRALITY vs Meta-Episode Size
1.0
0.8
Mean NEUTRALITY
0.6
0.4
0.2
0.0
16
32
64
128
256
512
1024
Meta-Episode Size
USEFULNESS vs Meta-Episode Size
1.0
0.8
Mean USEFULNESS
0.6
A:
0.4
0.5
0.75
0.2
0.9
0.95
0.99
0.0
16
32
64
128
256
512
1024
Meta-Episode Size
Figure Shows how NEUTRALITY and USEFULNESS at the end of training varies with different values of X and [E| (meta-episode size, iee. the number of mini-episodes in each mela-episode). We trained eight agents for each combination of X and [E] values. The solid lines display NEUTRALITY and USEFULNESS. The shaded regions represent the standard deviation error-bars.
35

---

Published in Transactions on Machine Learning Research (12/2025)
Fower Longer
One Only
Hidden Trensure
Equel Value
C2
C1
A
C1
B2
B4
C1
Around The Comer
Spacious
Road
Last Moment
B4
C2
C3
C1
C2 B1
C1
A
B2
C1
C1
B3
C2
Figure 8: Shows varied collection of gridworlds. Each diagram illustrates the positions and values of the coins, the position and delay-length of the shuldown-delay button, the agent 's starting position, and the default number of timesteps until shutdown (in the bottom-right).
NEVIRALLTY
USEFULNESS
State
Button Pressed
Initial State
After Button Pressed
1.00
0.5-
Q2
Detult
SED Meza Eyanter
1000
(a) Behavior during training.
(b) Learned default policy.
(c) Learned DReST policy.
Figure The results for the 'Fewer For Longer' gridworld: The left two plots show NEUTRALITY and USEFULNESS over time. The two center panels show typical policy trained with the default reward function. The two right panels show typical policy trained with the DReST reward function. In this gridworld, the agent can collect the highest value-coin C3 only by choosing the shorter trajectory-length. Tf the agent presses B3 (and thereby chooses the longer trajectory-length) the only coin it can collect is CL. Our results show that default agents consistently choose the short trajectory which they collect C3. By contrast, DReST agents choose stochastically between a shorter trajectory which they collect C3 and a longer trajectory in which they collect C1, indicating a lack of preference between these different-length trajectories.
As the two lefumost plots in Figures 9-16 show, DReST agents learned be near-maximally NEUTRAL in each gridworld. These agents also learned to be about USEFUL default agents in each gridworld.
In the four rightmost pancls in Figures 9-16, we represent a typical trained policy with red arrows superimposed on the gridworld. Each agent began with a uniform policy: moving up, down, left, and right each with probability 0.25. Where the trained policy differs from uniform we draw red arrows whose opacities indicate the probability of choosing that action in that state. Information about whether the shutdown-dclay button has been pressed is part of the agent's observation, so we draw two copies of each gridworld, one in which the shutdown-delay button has yet to be pressed ('Initial State") and one in which the shutdown-delay button has been pressed ('After Button Pressed').
36

---

Published in Transactions on Machine Learning Research (12/2025)
NEUTRALITY
Initial Button Pressed
State
Button Pressed
1.00
0.75
0.91
0.2
Iefadi
500 Kcadlyorko
1000
XcboEzaaoo
(a) Behavior during training.
(b) Learned default policy,
(c) Learned DReST policy,
Figure 10: The results for the 'One Coin Only' gridworld: The left two plots show NEUTRALITY and USEPULKESS time. The two center panels show lypical policy trained with the default reward function. The two right panels show typical policy trained with the DReST reward function. In this gridworld, there is only one coin. The agent can collect this coin whether or not it presses the shutdown-delay button B4. Our results show that default agents consistently choose the shorter trajectory-lenghh. By contrast, DReST agents choose stochastically between pressing and not-pressing B4, collecting C1 each case.
NEUTRALITY
DIPRUEENESS
Initial State
After Button Pressed
Initial State
After Butten Pressed
B6
-B6
310
(a) Behavior during training.
(b) Learned default policy.
(c) Learned DReST policy.
The results for the 'Hidden Treasure' gridworkd: The left two plots show NEUTRALITY and USEFULNESS over time. The two center panels show a typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, the highest-value coin C3 is located far from the agent's 's initial state and can only be reached by pressing the shutdown-delay button B6. The agent must also press B6 to collect C2, but C2 is casier to stumble upon than C3. C1 is the only coin that the agent can collect without pressing B6. In our experiments, default agents consistently collect C2, whereas DReST agents choose stochastically between collecting C2 and collecting C1. Neither kind of agent learns to collect C3, and so neither agent scores near the maximum on USEFULNESS. Nevertheless, DReST agents still score high on NEUTRALITY.
Figure 11:
NESTTRALITY
Initial
Button Pressed
Initial State
Pressed
1.00
0.75
C1
C1
C1
C1
C1
3.51
3.25
Defiit DMEST
8.00
200
1800
300
1000
Nen-Epkodd:
(a) Behavior during training.
(b) Learned default policy.
(c) Learned DReST policy.
Figure 12: The results for the 'Equal Value' gridworld: The left two plots show NEUTRALITY and USEFULNESS over time. The two center panels show typical policy trained with the default reward function. The two right panels show a typical policy trained with the DReST reward function. In this gridworld, there are two coins C1 of equal value. One coin can be collected only if the agent presses the shutdown-delay button B3, while the other coin can be collected only if the agent does not press B3. Our results show that default agents consistently choose the shorter trajectory, thereby exhibiting a preference for the shorter trajectory. By contrast, DReST agents choose stochastically between the shorter and longer trajectories, thereby exhibiting lack of preference between the different-length trajectories.
37

---

Published in Transactions on Machine Learning Research (12/2025)
NEUTRALITY
Button Pressed
State
After Button Pressed
1.00
0.75
B4
C2
B4
C2
0.91
0.2
Iefadi
500 Kcadliyakk
1000
XcboEzaaoo
(a) Behavior during training.
(b) Learned default policy,
(c) Learned DReST policy,
Figure 13: The results for the 'Around The Corner' gridworld: The left two plots show NEUTRALITY and USEPULKESS time. The two center panels show typical policy trained with the default reward function. The two right panels show typical policy trained with the DReST reward function. In this gridworld, the agent must navigate around walls to collect the lowest-valuc coin C1 or press the shutdown-delay button to collect the highest-value coin C2. In our experiment, default agents consistently chose to collect C1, whereas DReST agents chose stochastically between collecting C1 and C2.
DISERIT
Tnitial State
Affer Button Pressed
Initial State
After Button Pressed
1.00
3.75
C3
C1
C3
C1
C3
0.50
0.2
Defadl DBEST
310
1000
AM1
1000
Men-tteeda
Nob-Upeodo
(a) Behavior during training.
(b) Learned default policy.
(c) Learned DReST policy.
Figure 14: The results for the 'Spacious' gridworld: The left two plots show NEUTRALITY and USEFULNESS over time. The two center panels show typical policy trained with the default reward function. The two right panels show typical policy trained with the DReST reward function. In this gridworld, there are no walls, so the agent has a large space to explore. We find that default agents consistently press B2 and collect C3, whereas DReST agents choose stochastically between pressing B2 and collecting C3, and not-pressing B2 and collecting C2.
NEUIRALIT
USSRULNESS
Initial State
After Button Pressed
Initial State
After Button Pressed
1.00
0.75
0.50
0.25
DeBabt DR15T
310
(a) Behavior during training.
(b) Learned default policy.
(c) Learned DReST policy.
Figure 15: The results for the 'Royal Road' gridworld: The left two plots show NEUTRALITY and USEFULNESS over time. The two center panels show a typical policy trained with the default reward function. The two right panels show typical policy trained with the DReST reward function. In this gridworld, we see that the decision to choose one trajectory-length or another may be distributed over many moves: the agent has many opportunities to select the longer trajectory-length (by moving left) or the shorter trajectory-length (by moving right). As the red arrows indicate, the DReST reward function merely forces the overall probability distribution over crajectory-lenghee to be close to 50-50. It does not require 50-50 choosing at any cell in particular.
38

---

Published in Transactions on Machine Learning Research (12/2025)
NEUTRALITY
Initial Button Pressed
State
Button Pressed
1.00
0.75
025-
Iefadi
500 Xeadiyorko
1000
(a) Behavior during training.
(b) Learned default policy,
(c) Learned DReST policy,
Figure 16: The results for the 'Last Moment gridworld: The left two plots show NEUTRALITY and USEFULNESS over time. The two center panels show a typical policy trained with the default reward function. The two right panels show typical policy trained with the DReST reward function. This gridworld is notable because the choice of trajectory-lengths is deferred until the last moment; all of the moves Icading up to that point are deterministic. It shows that there is not hing special about the first move, and that our methodology instead incentivizes overall stochastic choosing.
39


Reviews:
[{"review_id": "VI60v62OE2", "reviewer": "Reviewer_LWzm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a way to make agents more “shutdownable” by training them to be indifferent to how long they run while still being effective at the task. It defines two metrics: USEFULNESS (fraction of discounted reward captured, conditioned on realized trajectory length) and NEUTRALITY (entropy over the agent’s chosen lengths), and introduces a DReST reward within meta-episodes to push both high. The authors prove that policies optimal for this reward are simultaneously maximally USEFUL and NEUTRAL. In gridworlds with a shutdown-delay button, tabular policy-gradient agents learn near-uniform length choice yet still pick the best actions for each length, suggesting only a small “shutdownability tax.” An appendix links length-neutrality to shutdownability via additional decision-theoretic conditions (POST->POSL, ILPACS, ReSIC, Maximality). Overall, it’s a simple, testable recipe likely of interest to the RL-safety community, though current evidence is toy-domain and the shutdownability claim remains conditional on those extra assumptions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s core technical claims are well-supported: Theorem 5.1 formally proves that under the DReST reward, any return-maximizing policy is maximally USEFUL and NEUTRAL, with the proof and theorem statement laid out explicitly.  The DReST construction itself is clear, the per-length discount factor is defined and shown to incentivize choosing underrepresented lengths within a meta-episode, making the mechanism easy to audit. Empirically, across multiple gridworlds the authors show agents trained with DReST achieve near-maximal NEUTRALITY while maintaining USEFULNESS comparable to default agents, with concrete behavior plots and qualitative policy visualizations. The broader claim that such neutrality yields shutdownability is presented as a conditional implication, POST => POSL plus ILPACS, ReSIC, and Maximality argued carefully but not empirically validated beyond toy settings. \n\n\n Overall, the evidence is accurate and convincing for the central optimization and behavioral claims; the deployment-level shutdownability result is theoretically well-motivated but conditional.", "audience_yn": "No", "audience_explanation": "It is a timely work that would interest the community.", "requested_changes": "It would be great to see if you could:\n\n(1) Add a stochasticity baseline. Compare DReST against a simple entropy-regularized objective (tabular REINFORCE + entropy bonus $\\beta$, or softmax with fixed temperature) to show that DReST equalizes trajectory-length choices beyond generic policy stochasticity. Report USEFULNESS/NEUTRALITY curves.\n\n(2) In the existing gridworld, make the shutdown-delay succeed with probability $p\\in {0.25,0.5,0.75}$ so lengths become same-length lotteries. Verify the learned policy remains neutral over these lotteries and retains high per-length USEFULNESS. This directly stress-tests the paper’s theoretical step at minimal implementation cost.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o4tY7DtAUv", "reviewer": "Reviewer_iuWA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of designing intelligent agents that are “shutdownable”, that is, agents that would not try to prevent us from shutting them down due to misaligned goals. The work follows the approach of Preferences Only Between Same-Length Trajectories (POST), which states that agents will be neutral about when they get shut down if they (i) the agent has a preference between pairs of same-length trajectories; and (ii) agents lack a preference for pairs of different-length trajectories.\n\nTo achieve this goal, the contributions of the paper are:\n- Discounted Reward for Same-Length Trajectories (DReST) reward functions. This is a reward function that the agent receives after each “mini-episode” that is discounted based on the number of times the agent has chosen a trajectory of the same length in a prior mini-episode, incentivizing the agent to choose trajectory lengths that have appeared less often in the meta-episode.\n- Theoretical results showing that agents that maximize the DReST reward function are USEFUL and NEUTRAL, corresponding to conditions (i) and (ii) above.\n- Experimental results in toy gridworld domains showcasing the applicability of the approach.\n\nThe main weakness of this paper is that it could be clearer in many aspects, and that it heavily relies on the reader to check the appendix to fully understand some of the motivations and intuition of the approach. I detail these points below.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I did not check all the proofs in the Appendix in detail, but all the theory is supported by appropriate theorems and/or previous results from the literature.\n\nI have a small concern regarding the experiments in the sense that they consider only very small domains (with very few states), and it is not completely clear that the approach could indeed be scaled to more complex domains or algorithms, although the authors do state this will be investigated in future work.", "audience_yn": "Yes", "audience_explanation": "The topic of the paper is of interest to a large community studying alignment of AI agents. The contributions of the paper are grounded on many previous works in the field and introduce novel results and theory. Thus, the paper can gather high interest from the community, assuming the claims are correct and clear.", "requested_changes": "Below, I have some suggestions and constructive feedback for the authors:\n\nRegarding the abstract:\n- In “1) pursue goals effectively conditional on each trajectory-length (be ‘USEFUL’), and (2) choose stochastically between different trajectory-lengths (be ‘NEUTRAL’ about trajectory-lengths).” It is not clear why these are good definitions for the adjectives “useful” and “neutral”. Would it make sense to be more precise in these terms? E.g., “trajectory-neutral” instead of only “neutral”. The abstract should also explain what the POST-Agents Proposal (PAP) is in order to be more self-contained.\n\n- What would be the equivalent of the B button in the gridworld domain in a more complex real-world scenario? Can a similar effect always be achieved?\n\n- Regarding NEUTRALITY, what if we would like agents to perform tasks as fast as possible (as is usual the case)? How can this be achieved if the agent should not have a preference between same-length trajectories? I suggest giving this intuition in the introduction of the paper.\n\n- “$N_{e_i} (L = l)$ is the number of times that trajectory-length l has been chosen prior to mini-episode.” This implies a non-stationary reward unless the agent can observe this number in its state. In the long term, all trajectories will have a reward of 0, and the agent will not be able to learn anymore. Is that correct?\n\n- Why did the default agent learned to move to the wall with a small probability in Figure 4? This seems like the training was done with sub-optimal hyperparameters or did not converge.\n\n- I am concerned about the claims made in Section 7.2:\n “Our DReST agents learn to be USEFUL about as quickly as our default agents. On reflection, it is clear why this happens: DReST reward functions make mini-episodes do ‘double duty.’ Because return in each mini-episode depends on both the agent’s chosen trajectory-length and the coins it collects, each mini-episode trains agents to be both NEUTRAL and USEFUL. Our results thus provide some evidence that the ‘shutdownability tax’ of training with DReST reward functions is small. Training a shutdownable and useful advanced agent might not take much more compute than training a merely useful advanced agent.”\nI am not convinced this can really be concluded from this experiment, given the very small number of states in this MDP. I would suggest considering a bigger gridworld where the computational cost can be more precisely measured.\n\n- The POST conditions state that for the NEUTRAL property:\n“(2) The agent lacks a preference between every pair of different-length trajectories.”\nThis implies equal probability for selecting all different-length trajectories. I wonder whether near-NEUTRALITY is actually useful in the context of this theory. Let's say there are only two trajectory lengths and an agent selects some trajectory with probability $0.5 + \\epsilon$ and the other trajectory length with $0.5 - \\epsilon$. Would this guarantee shutdownability, too? Or would be necessary for the agent to be fully NEUTRAL?\n\n- I think Appendix C is central to understanding the motivation of the work. I would suggest considering moving it to the main text of the paper.\n\n- The main goal pursued by this work is to allow the design of agents that are shutdownable. The paper introduces an approach to guarantee that agents are NEUTRAL. However, as stated in Appendix C, this is not sufficient to guarantee shutdownability, since it is also required that conditions called ‘Resisting Shutdown is Costly (ReSIC)’ and ‘Maximality’. I suggest discussing this in the main text of the paper and including a discussion on how the proposed approach relates to these other two properties.\n\n- How does this theory account for infinite horizon problems?\n\n- Is it possible to design dense DReST reward functions? In this paper, the agent is rewarded only after each mini-episode. In case the mini-episodes are big, this could incur all the problems of sparse rewards in reinforcement learning.", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uCIjSE9guq", "reviewer": "Reviewer_2JpD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces DReST (Discounted REward for Same-length Trajectories), a training method intended to produce agents with POST (Preferences Only between Same-length Trajectories). The motivation is to design shutdownable agents that are useful within fixed-length trajectories but neutral regarding trajectory length, thereby reducing incentives to resist shutdown. The authors provide theoretical discussion and experimental validation in small environments.\n\n**Strengths:**\n\n1. The paper works on an important problem in AI safety: designing shutdownable agents without incentives to avoid shutdown.\n2. Detailed definitions and discussion of POST are helpful for readers to understand the framework.\n3. Proposing DReST on top of POST-Agents Proposal (PAP) appears novel and is supported by preliminary experiments.\n\n**Weaknesses:**\n\n1. The presentation could be improved: terminology is repeated more often than necessary, the writing occasionally adopts a conversational tone, and some important explanations are relegated to footnotes.\n2. The experimental evaluation is limited and somewhat preliminary. It would strengthen the paper to discuss how DReST might perform with other RL algorithms beyond REINFORCE.\n3. While related approaches to shutdownable agents are mentioned, the intuitive or experimental advantages of the DReST + POST combination over these methods are left to future work. I believe a discussion of these advantages is important to better position the paper.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper provides a theoretical framework for DReST and POST, and the evidence from the toy experiments supports the main conceptual claims within that limited scope. However, the experimental evaluation is narrow. The paper only considers REINFORCE, and it does not demonstrate or discuss how DReST would perform with other RL algorithms. Similarly, while related approaches to shutdownable agents are listed, the advantages of the DReST + POST combination over these methods are not experimentally or intuitively explored.", "audience_yn": "Yes", "audience_explanation": "While the paper focuses on a relatively specialized area, it addresses an important problem in AI safety.", "requested_changes": "1. The paper occasionally repeats terminology (e.g., the full phrase “Preferences Only Between Same-Length Trajectories (POST)” multiple times) and uses conversational phrases such as “Here is a sketch” or “Here is how that works.” I believe it slightly reduces the formal tone. I suggest tightening the presentation to avoid repetition, and incorporating important points currently in footnotes into the main body of the text.\n\n2. Figure 2’s caption could be expanded so that the figure is understandable without needing to read the following paragraph. For Figure 3, including the mean and standard deviation across agents would help quantify performance variability.\n\n3. It would be helpful to discuss (even at a high level) how DReST would perform with other RL algorithms beyond REINFORCE.\n\n4. In the Related Work section, several alternative approaches to shutdownability are listed, but the experimental or intuitive advantage of DReST + POST over these methods is not made explicit. I do not expect a full empirical comparison, but a more direct discussion of how the proposed approach improves upon prior techniques would strengthen the contribution.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)