# Two concepts of an “episode” (Section 2.2.1 of “Scheming AIs”)

(This is Section 2.2.1 of my report “[Scheming AIs: Will AIs fake alignment during training in order to get power?](https://arxiv.org/pdf/2311.08379.pdf)”. There’s also a summary of the full report [here](https://www.lesswrong.com/posts/yFofRxg7RRQYCcwFA/new-report-scheming-ais-will-ais-fake-alignment-during) (audio [here](https://joecarlsmithaudio.buzzsprout.com/2034731/13969977-introduction-and-summary-of-scheming-ais-will-ais-fake-alignment-during-training-in-order-to-get-power)). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section [here](https://joecarlsmithaudio.buzzsprout.com/2034731/13984831-two-concepts-of-an-episode-section-2-2-1-of-scheming-ais), or search "Joe Carlsmith Audio" on your podcast app.)

# Beyond-episode goals

Schemers are pursuing goals that extend beyond the time horizon of the
episode. But what is an episode?

## Two concepts of an "episode"

Let's distinguish between two concepts of an episode.

### The incentivized episode

The first, which I'll call the "incentivized episode," is the concept
that I've been using thus far and will continue to use in what follows.
Thus, consider a model acting at a time t~1~. Here, the rough idea is to
define the episode as the temporal unit after t~1~ that training
*actively punishes* the model for not optimizing – i.e., the unit of
time such that we can know *by definition* that training is not directly
pressuring the model to care about consequences beyond that time.

For example, if training started on January 1st of 2023 and completed on
July 1st of 2023, then the maximum length of the incentivized episode
for this training would be six months – at no point could the model have
been punished by training for failing to optimize over a
longer-than-six-month time horizon, because no gradients have been
applied to the model's policy that were (causally) sensitive to the
longer-than-six-month consequences of its actions. But the incentivized
episode for this training process could in principle be shorter than six
months as well. (More below.)

Now, importantly, even if training only directly pressures a model to
optimize over some limited period of time, it can still *in fact create*
a model that optimizes over some much longer time period – that's what
makes schemers, in my sense, a possibility. Thus, for example, if you're
training a model to get as many gold coins as possible within a ten
minute window, it could still, in principle, learn the goal "maximize
gold coins over all time" – and this goal might perform quite well (even
absent training gaming), or survive despite not performing all that well
(for example, because of the "slack" that training allows).

Indeed, to the extent we think of evolution as an analogy for ML
training, then something like this appears to have happened with humans
with goals that extend indefinitely far into the future – for example,
"[longtermists](https://en.wikipedia.org/wiki/Longtermism)." That is,
evolution does not actively select for or against creatures in a manner
sensitive to the consequences of their actions in a trillion years
(after all, evolution has only been *running* for a few billion
years) – and yet, some humans aim their optimization on trillion-year
timescales regardless.

That said, to the extent a given training procedure *in fact* creates a
model with a very long-term goal (because, for example, such a goal is
favored by the sorts of "[inductive
biases](https://en.wikipedia.org/wiki/Inductive_bias)" I'll discuss
below), then in some sense you could argue that training "incentivizes"
such a goal as well. That is, suppose that "maximize gold coins in the
next ten minutes" and "maximize gold coins over all time" both get the
same reward in a training process that only provides rewards after ten
minutes, but that training selects "maximize gold coins over all time"
because of some other difference between the goals in question (for
example, because "maximize gold coins over all time" is in some sense
"simpler," and gradient descent selects for simplicity in addition to
reward-getting). Does that mean that training "incentivizes" or "selects
for" the longer-term goal?

Maybe you could say that. But it wouldn't imply that training "directly
punishes" the shorter-term goal (or "directly pressures" the model to
have the longer-term goal) in the sense I have in mind. In particular:
in this case, it's at least *possible* to get the same reward by
pursuing a shorter term goal (while holding other capabilities fixed).
And the gradients the model's policy receives are (let's suppose) only
ever sensitive to what happens within ten minutes of a model's action,
and won't "notice" consequences after that. So to the extent training
selects for caring about consequences further out than ten minutes, it's
not in virtue of those consequences *directly influencing the gradients
that get applied to the model's policy.* Rather, it's via some other,
less direct route. This means that the model could, in principle, ignore
post-ten-minute consequences without gradient descent pushing it to
stop.

Or at least, that's the broad sort of concept I'm trying to point at.
Admittedly, though, the subtleties get tricky. In particular: in some
cases, a goal that extends beyond the temporal horizon that the
gradients are sensitive to might actively get *more reward* than a
shorter-term goal.

-   Maybe, for example, "maximize gold coins over all time" actually
    gets *more reward* than "maximize gold coins over the next ten
    minutes," perhaps because the longer-term goal is "simpler" in some
    way that frees up extra cognitive resources that can be put towards
    gold-coin-getting.

-   Or maybe humans are *trying* to use short-term feedback to craft a
    model that optimizes over longer timescales, and so are actively
    searching for training processes where short-term reward is
    maximized by a model pursuing long-term goals. For example, maybe
    you want your model to optimize for the long-term profit of your
    company, so you reward it, in the short-term, for taking actions
    that seem to you like they will maximize long-term profit. One thing
    that could happen here is that the model starts optimizing
    specifically for getting this short-term reward. But if your
    oversight process is good enough, it could be that the
    highest-reward policy for the model, here, is to *actually optimize
    for* long-term profit (or for something else long-term that doesn't
    route via training-gaming).[^115]

In these cases, it's more natural to say that training "directly
pressures" the model towards the longer-term goal, given that this goal
gets more reward. However, I still want to say that longer-term goals
here are "beyond episode," because they extend beyond the temporal
horizon to which the gradients are directly and causally sensitive. I
admit, though, that defining this precisely might get tricky (see next
section for a bit more of the trickiness). I encourage efforts at
greater precision, but I won't attempt them here.[^116]

### The intuitive episode 

Let's turn to the other concept of an episode – namely, what I'll call
the "intuitive episode." The intuitive episode doesn't have a mechanistic
definition.[^117] Rather, the intuitive episode is just: a
natural-seeming temporal unit that you give reward at the end of, and
which you've decided to call "an episode." For example, if you're
training a chess-playing AI, you might call a game of chess an
"episode." If you're training a chatbot via RLHF, you might call an
interaction with a user an "episode." And so on.

My sense is that the intuitive episode and the incentivized episode are
often somewhat related, in the sense that we often pick an intuitive
episode that reflects some difference in the training process that makes
it easy to assume that the intuitive episode is also the temporal unit
that training directly pressures the model to optimize – for example,
because you give reward at the end of it, because the training
environment "resets" between intuitive episodes, or because the model's
actions in one episode have no obvious way of affecting the outcomes in
other episodes. Importantly, though, **the intuitive episode and the
incentivized episode aren't necessarily the same**. That is, if you've
just picked a natural-seeming temporal unit to call the "episode," it
remains an open question whether the training process will directly
pressure the model to care about what happens beyond the
episode-in-thise-sense. For example, it remains an open question whether
training directly pressures the model to sacrifice reward on an earlier
episode-in-thise-sense for the sake of more-reward on a later
episode-in-this-sense, if and when it is able to do so.

To illustrate these dynamics, consider a prisoner's dilemma-like
situation where each day, an agent can either take +1 reward for itself
(defection), or give +10 reward to the next day's agent (cooperation),
where we've decided to call a "day" an (intuitive) episode. Will
different forms of ML training directly pressure this agent to
cooperate? If so, then the intuitive episode we've picked isn't the
incentivized episode.

Now, my understanding is that in cases like these, vanilla policy
gradients (a type of RL algorithm) learn to defect (this test has
actually been done with simple agents – see [Krueger et al
(2020)](https://arxiv.org/pdf/2009.09153.pdf)). And I think it's
important to be clear about what sorts of algorithms behave in this way,
and why. In particular: glancing at this sort of set-up, I think it's
easy to reason as follows:

> "Sure, you say that you're training models to maximize reward 'on the
> episode,' for some natural-seeming intuitive episode. But you also
> admit that the model's actions can influence what happens later in
> time, even beyond this sort of intuitive episode – including, perhaps,
> how much reward it gets later. So won't you implicitly be training a
> model to maximize reward over *the whole training process*, rather
> than just on an individual (intuitive) episode. For example, if it's
> possible for a model to get *less reward* on the present episode, in
> order to get *more reward* later, won't cognitive patterns that give
> rise to making-this-trade get reinforced overall?"[^118]

From discussions with a few people who know more about reinforcement
learning than I do,[^119] my current (too-hazy) understanding is that at
least for *some* sorts of RL training algorithms, this isn't correct.
That is, it's *possible* to set up RL training such that some limited,
myopic unit of behavior is in fact the incentivized episode – *even if*
an agent can sacrifice reward on the present episode for the sake of
more-reward later (and presumably: even if the agent knows this).
Indeed, this may well be the default. See footnote for more
details.[^120]

Even granted that it's *possible* to avoid incentives to optimize across
intuitive-episodes, though, it's also possible to *not* do
this – especially if you pick your notion of "intuitive episode" very
poorly. For example, my understanding is that the transformer
architecture is set up, by default, such that language models are
incentivized, in training, to allocate cognitive resources in a manner
that doesn't just promote predicting the *next* token, but other later
tokens as well (see
[here](https://www.lesswrong.com/posts/c68SJsBpiAxkPwRHj/how-llms-are-and-are-not-myopic#2__Value_Prediction_Myopia)
for more discussion). So if you decided to call predicting
just-the-next-token an "episode," and to assume, on this basis, that
language models are never directly pressured to think further ahead,
you'd be misled.

And in some cases, the incentives in training towards cross-episode
optimization can seem quite counterintuitive. Thus, Krueger et al (2020)
show, somewhat surprisingly, that if you set the parameters right, a
form of ML training called Q-learning sometimes learns to cooperate in
prisoner's dilemmas despite the algorithm being "myopic" in the sense
of: ignoring reward on future "episodes." See footnote for more
discussion, and see
[here](https://www.alignmentforum.org/s/tDBYJd4p6EorGLEFA/p/rTYGMbmEsFkxyyXuR#Getting_cooperation_without_meta_learning)
for a nice and quick explanation of how Q-learning works.[^121]

![Plots of Q-values and cooperation probability from Figure 11 in
Krueger et al (2020), reprinted with permission. The first three agents
learn to defect, and the last two, to cooperate almost 90% of the time
(note the differences between the y-axes in the first three and the last
two).](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5db2b61495abe0f9b31e867ee9038da92a59e9a44cc03c00.png)

*Plots of Q-values and cooperation probability from Figure 11 in
Krueger et al (2020), reprinted with permission. The first three agents
learn to defect, and the last two, to cooperate almost 90% of the time
(note the differences between the y-axes in the first three and the last
two).*

Another way to incentivize optimization across intuitive episodes is to
introduce various additional layers (or "outer loops") of selection in
training. Thus, for example, consider a toy and unrealistic form of
"[population based
training](https://www.deepmind.com/blog/population-based-training-of-neural-networks)"
where you create 1000 agents who play starcraft, have them play 100
games each (and get updated along the way via some object-level training
algorithm), then select the one that performs the best *on the final
game*.[^122] In this case, if it were possible for a model to sacrifice
reward on earlier games, in order to perform better on the final game,
there would be at least some selection pressure for it to do so. And
see, also, results in [Krueger et al
(2020)](https://arxiv.org/pdf/2009.09153.pdf), showing that a different
form of population based training selects for cooperation in a
prisoner's-dilemma-like case like the one discussed above.[^123]
Depending on the details, though, outer-loops of this kind may exert a
much *weaker* degree of selection pressure than inner loops driven
directly by gradient descent.[^124]

Overall, my current sense is that one needs to be quite careful in
assessing whether, in the context of a particular training process, the
thing you're thinking of as an "episode" is actually such that training
doesn't actively pressure the model to optimize beyond the "episode" in
that sense – that is, whether a given "intuitive episode" is actually
the "incentivized episode." Looking closely at the details of the
training process seems like a first step, and one that in theory should
be able to reveal many if not all of the incentives at stake. But
empirical experiment seems important too.

Indeed, I am somewhat concerned that my choice, in this report, to use
the "incentivized episode" as my definition of "episode" will too easily
prompt conflation between the two definitions, and correspondingly
inappropriate comfort about the time horizons that different forms of
training directly incentivize.[^125] I chose to focus on the
incentivized episode because I think that it's the most natural and
joint-carving unit to focus on in differentiating schemers from other
types of models. But it's also, importantly, a theoretical object that's
harder to directly measure and define: you can't assume, off the bat,
that you know what the incentivized episode for a given sort of training
*is*. And my sense is that most common uses of the term "episode" are
closer to the intuitive definition, thereby tempting readers (especially
casual readers) towards further confusion. Please: don't be confused.

[^115]: I'll briefly note another complexity that this sort of case
    raises. Naively, you might've thought that the "specified goal"
    would only ever be confined to the incentivized episode, because the
    specified goal is the "thing being rewarded," and anything that
    *causes* reward is within the temporal horizon to which the
    gradients are sensitive. And in some cases – for example, where the
    "specified goal" is some clearly separable *consequence* of the
    model's action (e.g., getting gold coins), which the training
    process induces the model to optimize for – this makes sense. But in
    other cases, I'm less sure. For example, if you are sufficiently
    good at telling whether your model is *in fact optimizing for
    long-term profit*, and providing short-term rewards that in fact
    incentivize it to do so, then I think it's possible that the right
    thing to say is that the "specified goal," here, is long-term profit
    (or at least, "optimizing for long-term profit," which looks pretty
    similar). However, I don't think it ultimately matters much whether
    we call this sort of goal "specified" or "mis-generalized" (and it's
    a pretty wooly distinction more generally), so I'm not going to
    press on the terminology here.

[^116]: Also: when I talk about the gradients being sensitive to the
    consequences of the model's action over some time horizon, I am
    imagining that this sensitivity occurs via (1) the relevant
    consequences occurring, and then (2) the gradients being applied in
    response. E.g., the model produces an action at t1, this leads to it
    getting some number of gold coins at t5, then the gradients, applied
    at t6, are influenced by how many gold coins the model in fact got.
    (I'll sometimes call this "causal sensitivity.")

    But it's possible to imagine fancier and more philosophically
    fraught ways for the consequences of a model's action to influence
    the gradients. For example, suppose that the model is being
    supervised by a human who is extremely good at *predicting* the
    consequences of the model's action. That is, the model produces some
    action at t1, then at t2 the human *predicts* how many gold coins
    this will lead to at t5, and applies gradients at t3 reflecting this
    prediction. In the limiting case of perfect prediction, this can
    lead to gradients identical to the ones at stake in the first
    case – that is, information about the consequences of the model's
    action is effectively "traveling back in time," with all of the
    philosophical problems this entails.# So if, in the first case, we
    wanted to say that the "incentivized episode" extends out to t5,
    then plausibly we should say this of the second case, too, even
    though the gradients are applied at t3. But even in a case of
    pretty-good-but-still-imperfect prediction, there is a sense, here,
    in which the gradients the model receives are sensitive to
    consequences that haven't yet happened.

    I'm not, here, going to extend the concept of the "incentivized
    episode" to cover forms of sensitivity-to-future-consequences that
    rest on predictions about those consequences. Rather, I'm going to
    assume that sensitivity in question arises via normal forms of
    *causal* influence. That said, I think the fact that it's *possible*
    to create some forms of sensitivity-to-future-consequences even
    prior to seeing those consequences play out is important to keep in
    mind. In particular, it's one way in which we might end up training
    long-horizon optimizers using fairly short incentivized episodes
    (more discussion below).

[^117]: There may be other, additional, and more precise ways of using
    the term "episode" in the RL literature. Glancing at various links
    online, though (e.g.
    [here](https://www.baeldung.com/cs/epoch-vs-episode-reinforcement-learning)
    and
    [here](https://towardsdatascience.com/the-complete-reinforcement-learning-dictionary-e16230b7d24e)),
    I'm mostly seeing definitions that refer to an episode as something
    like "the set of states between the initial state and the terminal
    state," which doesn't say how the initial state and the terminal
    state are designated.

[^118]: As an example of someone who seems to me like they could be
    reasoning in this way, though it's not fully clear, see [this
    comment](https://www.lesswrong.com/posts/LDsSqXf9Dpu3J3gHD/why-i-m-excited-about-debate?commentId=BvFWfvryfoDJrmcgn)
    from Eliezer Yudkowsky, in response to a hypothetical in which he
    imagines humans rewarding an agent for each of its sentences
    according to how useful that sentence is:

    > "Let's even skip over the sense in which we've given the AI a
    > long-term incentive to accept some lower rewards in the short
    > term, in order to grab control of the rating button, if the AGI
    > ends up with long-term consequentialist preferences and long-term
    > planning abilities that exactly reflect the outer fitness
    > function."

    That said, as I discuss below, the details of the training process
    here matter.

[^119]: In particular: Paul Christiano, Ryan Greenblatt, and Rohin Shah.
    Though they don't necessarily endorse my specific claims here, and
    it's possible I've misunderstood them more generally.

[^120]: My hazy understanding of the argument here is that these RL
    algorithms update the model's policy towards higher-reward actions
    on the episode in a way that *doesn't* update you towards whatever
    policies *would've led to you starting in a higher-reward episode*
    (In this sense, they behave in a manner analogous to "causal
    decision theory."). Thus, let's say that the agent on Day 1 (with no
    previous agent to benefit her) chooses between cooperating (0
    reward) and defecting (+1 reward reward), and so this episode
    results in an update towards defecting. Then, on Day 1, the agent
    either starts out choosing between 10 vs. 11 reward (call this a
    "good episode"), or 0 vs. +1 reward (call this a "bad episode").
    Again, either way, it updates towards defection. It *doesn't*
    update, in the good episode, towards "whatever policy led me to this
    episode."

    That said, in my current state of knowledge about RL, I'm still a
    bit confused about this. Suppose, for example, that at the point of
    choice, you don't know whether or not you're in the good episode or
    the bad episode, and the training process is updating you with
    strength proportional to the degree to which you got more reward
    than you *expected* to get. If you start out with e.g. 50% that
    you're in a good episode and 50% that you're in a bad episode (such
    that the expected reward of cooperating is 5, and the expected
    reward of defecting is 6), then it seems like it could be the case
    that being in a good episode results in reward that is much better
    than you expected, such that policies that make it to a good episode
    end up reinforced to a greater extent, at least initially.

    I'm not sure about the details here. But from my current epistemic
    state, I would want to spell out and understand the details of the
    training process in much greater depth, in order to verify that
    there isn't an incentive towards cross-episode optimization.

[^121]: I think the basic dynamic here is: the Q-values for the actions
    reflect the average reward for taking that action thus far. This
    makes it possible for the Q-value for "cooperate" to give more
    weight to the rewards received in the "good episodes" (where the
    previous-episode's agent cooperated) rather than the "bad episodes"
    (where the previous-episode's agent defected), if the agent ends up
    in good episodes more often. This makes it possible to get a
    "cooperation equilibrium" going (especially if you set the initial
    q-value for defecting low, which I think they do in the paper in
    order to get this effect), wherein an agent keeps on cooperation.
    That said, there are subtleties, because agents that end up in a
    cooperation equilibrium still sometimes explore into defecting, but
    in the experiment it (sometimes) ends up in a specific sort of
    balance, with q-values for cooperation and defection pretty similar,
    and with the models settling on a 90% or so cooperation probability
    (more details
    [here](https://www.alignmentforum.org/s/tDBYJd4p6EorGLEFA/p/rTYGMbmEsFkxyyXuR#Getting_cooperation_without_meta_learning)
    and in the paper's appendix).

[^122]: I owe this example to Mark Xu.

[^123]: See also discussion from Carl Shulman
    [here](https://www.dwarkeshpatel.com/i/128144047/ai-takeover-scenarios):
    'it could be something like they develop a motivation around an
    extended concept of reproductive fitness, not necessarily at the
    individual level, but over the generations of training tendencies
    that tend to propagate themselves becoming more common and it could
    be that they have some goal in the world which is served well by
    performing very well on the training distribution.''

[^124]: Indeed, in principle, you could imagine pointing to other, even
    more abstract and hard-to-avoid "outer loops" as well, as sources of
    selection pressure towards longer-term optimization. For example, in
    principle, "grad student descent" (e.g., researchers experimenting
    with different learning algorithms and then selecting the ones that
    work best) introduces an additional layer of selection pressure
    (akin to a hazy form of
    "[meta-learning](https://en.wikipedia.org/wiki/Meta-learning_(computer_science))"),
    as do dynamics in which, other things equal, models whose tendencies
    tend to propagate into the future more effectively will tend to
    dominate over time (where long-term optimization is, perhaps, one
    such tendency). But these, in my opinion, will generally be weak
    enough, relative to gradient descent, that they seem to me much less
    important, and OK to ignore in the context of assessing the
    probability of schemers.

[^125]: Thanks to Daniel Kokotajlo for flagging this concern.