# [Intro to brain-like-AGI safety] 5. The “long-term predictor”, and TD learning

*(Last revised: January 2026. See changelog at the bottom.)*

5.1 Post summary / Table of contents
====================================

*Part of the* [*“Intro to brain-like-AGI safety” post series*](https://www.alignmentforum.org/s/HzcM2dkCq7fwXBej8)*.*

In [the previous post](https://www.alignmentforum.org/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor), I discussed the “short-term predictor”—a circuit which, thanks to a learning algorithm, emits an output that predicts a ground-truth supervisory signal arriving a short time (e.g. a fraction of a second) later.

In this post, I propose that we can take a short-term predictor, wrap it up into a closed loop involving a bit more circuitry, and we wind up with a new module that I call a “long-term predictor”. Just like it sounds, this circuit can make longer-term predictions, e.g. “I’m likely to eat in the next 10 minutes”. This circuit is closely related to Temporal Difference (TD) learning, as we’ll see.

I will argue that there is a large collection of side-by-side long-term predictors in the brain, each comprising a short-term predictor in the [Learning Subsystem](https://www.lesswrong.com/posts/wBHSYwqssBGCnwvHg/intro-to-brain-like-agi-safety-2-learning-from-scratch-in) (more specifically, in the [extended striatum](https://scholar.google.com/scholar?cluster=4402146436136297171&oi=gsb&hl=en&as_sdt=0,22)—putamen, nucleus accumbens, lateral septum, part of the amygdala, etc.) that loops down to the [Steering Subsystem](https://www.lesswrong.com/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and) (hypothalamus and brainstem) and then back up via a dopamine neuron. These long-term predictors make predictions about biologically-relevant inputs and outputs—for example, one long-term predictor might predict whether I’ll feel pain in my arm, another whether I’ll get goosebumps, another whether I’ll release cortisol, another whether I’ll eat, and so on. Moreover, one of these long-term predictors is essentially a value function for reinforcement learning.

All these predictors will play a major role in motivation—a story which I will finish in the [next post](https://www.alignmentforum.org/posts/qNZSBqLEh4qLRqgWW/intro-to-brain-like-agi-safety-6-big-picture-of-motivation).

*Table of contents:*

*   §5.2 starts with a toy model of a “long-term predictor” circuit, consisting of the “short-term predictor” of the [previous post](https://www.alignmentforum.org/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor), plus some extra components, wrapped into a closed loop. Getting a good intuitive understanding of this model will be important going forward, and I will walk through how that model would behave under different circumstances.
*   §5.3 relates that model to [Temporal Difference (TD) learning](https://en.wikipedia.org/wiki/Temporal_difference_learning), which is closely related to a “long-term predictor”. I’ll show two variants of the long-term predictor circuit, a “summation” version (which leads to a value function that approximates the sum of future rewards), and a “switch” version (which leads to a value function that approximates the *next* reward, whenever it should arrive, which may not be for a long time). The “summation” version is universal in AI literature, but I’ll suggest that the “switch” version is probably closer to what happens in the brain. Incidentally, these two models are equivalent in cases like AlphaGo, wherein reward arrives in a lump sum at the very end of each episode (= each game of Go).
*   §5.4 will relate long-term predictors to the neuroanatomy of the striatum and brainstem.
*   §5.5 will offer five lines of evidence that lead me to believe this story: (1) It’s a sensible way to implement a biologically-useful capability; (2) It’s introspectively plausible; (3) It’s evolutionarily plausible; (4) It’s neuroscientifically plausible; (5) It’s compatible with the psychology literature on “Pavlovian conditioning”, “revaluation”, “incentive learning”, and more.

5.2 Toy model of a “long-term predictor” circuit
================================================

A “long-term predictor” is ultimately nothing more than a short-term predictor whose output signal helps determine its own supervisory signal. Here’s a toy model of what that can look like:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c0b462ccfa7c9c2d7b5eb32fe1045c4ef872c4b26f42b149.png)

Toy model of a long-term prediction circuit. I’ll spend the next couple subsections walking through how this works. **Note:** For this and all similar diagrams in this post, every block at every moment is running in parallel, and likewise every arrow at every moment is carrying a numerical value. So this is *NOT* a control-flow diagram for serial code; rather, it’s the kind of diagram you might see describing an FPGA, for example.

*   The blue box is the short-term predictor of [the previous post](https://www.alignmentforum.org/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor). It optimizes its output signal such that it approximates what the supervisor signal will be in 0.3 seconds (as an example).
*   The purple box is a 2-way switch. The toggle on the switch is controlled by genetically-hardwired circuitry (gray oval), according to the following rules:
    1.  By and large, the switch is in the bottom setting (“defer-to-predictor mode”). This setting is akin to the genetically-hardwired circuitry “trusting” that the short-term predictor’s output is sensible, and in particular producing the suggested amount of digestive enzymes.
    2.  If the genetically-hardwired circuitry gets a signal that I’m eating something *right now*, and that I don’t have adequate digestive enzymes, it flips the switch to “override mode”. Regardless of what the short-term predictor says, it sends the signal to manufacture digestive enzymes.
    3.  If the genetically-hardwired circuitry has been asking for digestive enzyme production for an extended period, and there’s still no food being eaten, then it again flips the switch to “override mode”. Regardless of what the short-term predictor says, it sends the signal to *stop* manufacturing digestive enzymes.

Note: You can assume that all the signals in the diagram can vary continuously across a range of values (as opposed to being discrete on/off signals), with the exception of the signal that toggles the 2-way switch.[^nmsgsjxd32] In the brain, smoothly-adjustable signals might be created by, for example, rate-coding—i.e., encoding information as the frequency with which a neuron is firing.

Next, let’s walk through the properties of this toy model.[^d1mr9i8su2a]

5.2.1 Toy model walkthrough: preliminaries
------------------------------------------

Here’s a little summary table for this subsection, that I’ll explain in the text below.

|     | **Override mode** | **Defer-to-predictor mode** |
| --- | --- | --- |
| Training incentive | Anticipate override signal | Keep output steady |
| AI intuitions | Supervised learning | TD learning |
| Neuro intuitions | Innate reactions | Self-fulfilling prophecies |

### 5.2.1.1 What is the short-term predictor getting trained to do?

The short-term predictor is a learning algorithm. Given infinite time, we normally expect learning algorithms to settle into some steady-state configuration where they no longer update. We can think of this configuration as “what we are training it to do”. So, in this toy model, what are we training the short term predictor to do?

Well, first, if there’s an override with ground truth *R*, we are training the short-term predictor to be *already* outputting *R*, immediately before that override.

And second, while we’re in defer-to-predictor mode, we are training the short-term predictor to keep its output steady. After all, any change in its output would create a (transient) discrepancy between the short-term predictor’s output, versus its supervisor, thus causing an update.[^t5hi9l336w]

(Clarification: I’m using the phrase “keep its output steady”, but that glosses over the key detail that we are training it to “keep its output steady” in a way that privileges one temporal direction over the other: the output *now* is treated as what the output *should have been* in the immediate past. Whereas in the other direction, if the short-term predictor was making a bad prediction in the past, but now it has new information that reveals the error, we are *not* training it to “keep its output steady” by sticking with the old bad prediction. That’s just not what the learning algorithm updates would do.)

So putting those two observations together, in this toy model, **we are training the short-term predictor to always output what the next override will be**, even if that override won’t happen for a very long time. (Or if there’s randomness that makes the next override unpredictable, we are training it to predict the expectation value of the next override.)

### 5.2.1.2 AI intuitions: supervised learning, and Temporal Difference (TD) learning

If you’re an AI reader, think of the overrides as like sporadic “injections of ground truth”, turning the setup into a kind of supervised learning.

And separately, think of defer-to-predictor mode as a version of TD learning (more on which below).

AI people might be wondering: In AI textbooks, TD learning is notoriously bad at long-term planning, because the correct behavior “walks backwards” one (fraction-of-a-second) timestep per attempt. And yet, humans will plan a vacation a month in advance, without having previously gone on (1 month) / (0.3 seconds) ≈ 10,000,000 vacations. Is that evidence that TD learning can’t be involved? No! The trick is, in the brain, the context data going into the short-term predictor *already* contains temporally-extended latent variables. For example, the mental concept “I’m getting lunch” is active in your mind 10 minutes in advance of eating, as you get on the bus to the restaurant, and is *also* active in your mind at the moment when you bite into the sandwich. So nothing has to propagate back one-timestep-at-a-time over the course of thousands of lunches; instead it takes as few as *one* iteration for the short-term predictor to learn that the “I’m getting lunch” latent variable predicts digestive enzymes. I’ll come back to this point below.

### 5.2.1.3 Neuro intuitions: innate reactions, and self-fulfilling prophecies

Meanwhile, neuro readers should think of override mode as innate reactions to innate triggers, often related to homeostasis, like salivating in response to salty food in the mouth, or physiological arousal in response to pain. As discussed in [Post 3](https://www.lesswrong.com/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and), these reactions generally happen entirely within the Steering Subsystem (brainstem, hypothalamus, etc.).

And then think of defer-to-predictor mode as “self-fulfilling prophecies”. For example, as you hold the cracker in your hand, you *expect* that you’re about to salivate, and that expectation turns into reality. Similarly, as you go to start a painful procedure, you *expect* that you’re about the have physiological arousal, and that expectation turns into reality. The Steering Subsystem has no direct evidence either way, so it “defers” to the guesses coming from the Learning Subsystem.

5.2.2 Toy model walkthrough: the case of static context
-------------------------------------------------------

Next, let’s walk through what would happen in this toy model, in the specific case that ***the “context” is static for some extended period of time***. For example, imagine a situation where some ancient worm-like creature is digging in the sandy ocean bed for many consecutive minutes. Plausibly, its sensory environment would stay pretty much constant as long as it keeps digging, as would its thoughts and plans (insofar as this ancient worm-like creature has “thoughts and plans” in the first place). Or if you want another example of (approximately) static context—this one involving a human rather than a worm—hang on until the next subsection.

In the static-context case, let’s first consider what happens when the switch is sitting in “defer-to-predictor mode”: Since the output is looping right back to the supervisor, there is no error in the supervised learning module. The predictions are correct. The synapses aren’t changing. Even if this situation is very common, it has no bearing on how the short-term predictor eventually winds up behaving.

The times that *do* matter for the eventual behavior of the short-term predictor are those rare times that we go into “override mode”. They inject ground truth into the system, and produce an error signal in the short-term predictor’s learning algorithm, changing its adjustable parameters (e.g. synapse strengths).

After enough life experience (a.k.a. “training” in ML terminology), the short-term predictor should have the property that *the overrides balance out*. There may still be occasional overrides that *increase* digestive-enzyme production, and there may still be occasional overrides that *decrease* digestive-enzyme production, but those two types of overrides should happen with similar frequency. After all, if they didn’t balance out, the short-term predictor’s internal learning algorithm would gradually change its parameters so that they *did* balance out.

And that’s just what we want! We’ll wind up with appropriate digestive enzyme production at appropriate times, in a way that properly accounts for any information available in the context data—what the animal is doing right now, what it’s planning to do in the future, what its current sensory inputs are, etc.

### 5.2.2.1 David-Burns-style exposure therapy—a possible real-life example of the toy model with static context?

As it happens, I recently read [David Burns’s book *Feeling Great*](https://www.amazon.com/Feeling-Great-Revolutionary-Treatment-Depression/dp/168373288X) ([my review](https://www.lesswrong.com/posts/jqTeghCJ2anMHPPjG/book-review-feeling-great-by-david-burns)). David Burns has a very interesting approach to exposure therapy—an approach that happens to serve as an excellent example of how my toy model works in the static-context situation!

Here’s the short version. (Warning: If you’re thinking of doing exposure therapy on yourself at home, *at least* read the whole book first!) Excerpt from the book:

> For example, when I was in high school, I wanted to be on the stage crew of *Brigadoon*, a play my school was putting on, but it required overcoming my fear of heights since the stage crew had to climb ladders and work near the ceiling to adjust the lights and curtains. My drama teacher, Mr. Krishak, helped me overcome this fear with the very type of exposure techniques I’m talking about. He led me to the theater and put a tall ladder in the middle of the stage, where there was nothing nearby to grab or hold on to. He told me all I had to do was stand on the top of the ladder until my fear disappeared. He reassured me that he’d stand on the floor next to me and wait.
> 
> I began climbing the ladder, step by step, and became more and more frightened. When I got to the top, I was terrified. My eyes were almost 18 feet from the floor, since the ladder was 12 feet tall, and I was just over 6 feet tall. I told Mr. Krishak I was in a panic and asked what I should do. Was there something I should say, do, or think about to make my anxiety go away? He shook his head and told me to just stand there until I was cured.
> 
> I continued to stand there in terror for about ten more minutes. When I told Mr. Krishak I was still in a panic, he assured me that I was doing great and that I should just stand there a few more minutes until my anxiety went away. A few minutes later, my anxiety suddenly disappeared. I couldn’t believe it!
> 
> I told him, “Hey, Mr. Krishak, I’m cured now!”
> 
> He said, “Great, you can come on down from the ladder now, and you can be on the stage crew of *Brigadoon*!”
> 
> I had a blast working on the stage crew. I absolutely loved climbing ladders and adjusting the lights and curtains near the ceiling, and I couldn’t even remember why or how I’d been so afraid of heights.

This story seems to be beautifully consistent with my toy model here. David started the day in a state where his short-term-predictors output “extremely strong fear reactions” when he was up high. As long as David stayed up on the ladder, those fear-reaction short-term-predictors kept on getting the same context data, and therefore they kept on firing their outputs at full strength. And David just kept feeling terrified.

Then, after 15 boring-yet-terrifying minutes on the ladder, some innate circuit in David’s brainstem issued an *override*—as if to say, “C’mon, nothing is changing, nothing is happening, we can’t just keep burning all these calories all day. It’s time to calm down now.” The short-term-predictors continued sending the same outputs as before, but the brainstem exercised its veto power, and forcibly reset David’s cortisol, heart-rate, etc., back to baseline. This “override” state immediately created *error signals* in the relevant short-term-predictors in David’s amygdala! And the error signals, in turn, led to model updates! The short-term predictors were all edited, and from then on, David was no longer afraid of heights.

Here’s the diagram for this situation; make sure you can follow all the steps.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8bca975138049c03705282dc334640cc9934707453e0d374.png)

5.2.3 Toy model walkthrough, assuming changing context
------------------------------------------------------

The previous subsection assumed static context lines (constant sensory environment, constant behaviors, constant thoughts and plans, etc.). What happens if the context is *not* static?

If the context lines are changing, then it’s no longer true that learning happens *only* at “overrides”. If context changes in the absence of “overrides”, it will result in changing of the output, and *the new output will be treated as ground truth for what the old output should have been*. Again, this seems to be just what we want: if we learned something new and relevant in the last second, then our current expectation should be more accurate than our previous expectation, and thus we have a sound basis for updating our models.

5.3 Value function calculation (TD learning) as a special case of long-term prediction
======================================================================================

At this point, ML experts may recognize a resemblance to [Temporal Difference (TD) learning](https://en.wikipedia.org/wiki/Temporal_difference_learning). It’s not quite the same, though. The differences are:

**First**, TD learning is usually used in reinforcement learning (RL) as a method for going from a reward function to a value function. By contrast, I was talking about things like “digestive enzyme production”, which are neither rewards nor values.

In other words, there is a generally-useful motif that involves going from some immediate quantity X to “long term expectation of X”. The calculation of a value function from a reward function is an *example* of that motif, but it’s not the only useful example.

(As a matter of terminology, it seems to be generally accepted that the term “TD learning” can in fact apply to things that are *not* RL value functions.[^v6aq2sfj7bk] However, empirically in my own experience, as soon as I mention “TD learning”, the people I’m talking to immediately assume I *must* be talking about RL value functions. So I want to be clear here.)

**Second**, to get something closer to traditional TD learning, we’d need to replace the 2-way *switch* with a 2-way *summation*—and then the “overrides” would be analogous to rewards. Much more on “switch vs summation” in the next subsection.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/867b687b0fe9eee05de6f8ac21ca22fc1c7e1b74a41c7768.png)

Here’s a TD learning circuit that would behave similarly to what you’d see in an AI textbook. Note the purple box on the right: compared to the previous figure, I replaced the 2-way *switch* with a 2-way *summation*. More on “switch vs summation” in the next subsection.

**Third**, there are many additional ways to tweak the circuit which are frequently used in AI textbooks, and some of those may be involved in the brain circuits too. For example, we can put in time-discounting, or different emphases on false-positives vs false-negatives (see my discussion of distributional learning in a footnote of the [previous post](https://www.lesswrong.com/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor)), etc.

To keep things simple, I will be ignoring all these possibilities (including time-discounting) in the discussion below.

5.3.1 Switch (i.e., value = expected next reward) vs summation (i.e., value = expected sum of future rewards)?
--------------------------------------------------------------------------------------------------------------

The figures above show two variants of our toy model. In one, the purple box is a two-way *switch* between “defer to the short-term predictor” and some independent “ground truth”. In the other, the purple box is a two-way *summation* instead.

The *switch* version trains the short-term-predictor to predict the next ground truth, whenever it should arrive.

The *summation* version trains the short-term-predictor to predict the *sum* of future ground truth signals.

The correct answer could also be “something in between switch and summation”. Or it could even be “none of the above”.

RL papers universally use the summation version—i.e., “value is the expected sum of future rewards”. What about biology? And which is actually better?

It doesn’t always matter! Consider AlphaGo. Like every RL paper today, AlphaGo was originally formulated in the summation paradigm. But it happens to have one and only one nonzero reward signal per game, namely +1 at the end of the game if it wins, or -1 if it loses. In that case, switch vs summation makes no difference. The only difference is one of terminology:

*   In the summation case, we would say “each non-terminal move in the Go game has reward=0”.
*   In the switch case, we would say “each non-terminal move in the Go game has a reward of (null)”.

(Do you see why?)

But in other cases, it does matter. So back to the question: should it be switch or summation?

Let’s step back. What are we trying to do here?

One thing that a brain needs to do is make decisions that weigh cross-domain tradeoffs. If you’re a human, you need to decide whether to watch TV or go to the gym. If you’re some ancient worm-like creature, you need to “decide” whether to dig or to swim. Either way, this “decision” impacts energy balance, salt balance, probability of injury, probability of mating—you name it. The design goal in the decision-making algorithm is that you make the decision that maximizes inclusive genetic fitness. How might that goal be best realized?

One method involves building a *value function* that estimates the organism’s inclusive genetic fitness (compared to some arbitrary—indeed, possibly time-varying—baseline), conditional on continuing to execute a given course of action. Of course it won’t be a perfect estimate—*real* inclusive genetic fitness can only be calculated in hindsight, many generations after the fact. But once we have such a value function, however imperfect, we can plug it into an algorithm that makes decisions to maximize value (more on this in the [next post](https://www.alignmentforum.org/posts/qNZSBqLEh4qLRqgWW/intro-to-brain-like-agi-safety-6-big-picture-of-motivation)), and thus we get approximately-fitness-maximizing behavior.

So having a value function is key for making good decisions that weigh cross-domain tradeoffs. But nowhere in this story is the claim “value is the expectation of a sum of future rewards”! That’s a particular way of setting up the value-approximating algorithm, a method which might or might not be well suited to the situation at hand.

**I happen to think that brains use something closer to the switch circuit, not the summation circuit, not only for homeostatic-type predictions (like the digestive enzymes example above), but also for value functions, contrary to mainstream RL papers.** Again, I think it’s really “neither of the above” in all cases; just that it’s *closer* to switch.

Why do I favor “switch” over “summation”?

An example: sometimes I stub my toe and it hurts for 20 seconds; other times I stub my toe and it hurts for 40 seconds. But I don’t think of the latter as *twice as bad* as the former. In fact, even five minutes later, I wouldn’t remember which is which. (See the [peak-end rule](https://en.wikipedia.org/wiki/Peak%E2%80%93end_rule).) This is the kind of thing I would naturally expect from switch, but is an awkward fit for summation. It’s not *strictly* incompatible with summation; it just requires a more complicated, value-dependent reward function. As a matter of fact, if we allow the reward function to depend on value, then switch and summation can imitate each other.

Anyway, in upcoming posts, I’ll be assuming switch, not summation. I don’t think it matters very much for the big picture. I *definitely* don’t think it’s part of the “secret sauce” of animal intelligence, or anything like that. But it does affect some of the detailed descriptions.

The [next post](https://www.alignmentforum.org/posts/qNZSBqLEh4qLRqgWW/intro-to-brain-like-agi-safety-6-big-picture-of-motivation) will include more details of reinforcement learning in the brain, including how “reward prediction error” works and so on. I am bracing for lots of confused readers, who will be disoriented by the fact that I’m assuming a different relationship between value and reward than what everyone is used to. For example, in my picture, “reward” is a *synonym* for “ground truth for what the value function should be right now”—*both* should account for not only the organism’s current circumstances but also its future prospects. Sorry in advance for any confusion! I will do my best to be clear.

5.3.2 Example
-------------

After all that abstract discussion, I should probably go through a more concrete example illustrating how I think the long-term predictor works in the domain of reinforcement learning.

Suppose I’m downstairs and feeling cold. An idea pops into my head: “I’ll go put on my cozy sweater”. So I walk upstairs and put it on.

What just happened? Here’s a diagram:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/fa7988af45d571f504523737eeacbe9da0f82b89ebc900bc.png)

This is the same kind of diagram as §5.2 above, but instead of predicting a digestive enzyme signal, the blue box is predicting a special signal I call *valence*. This circuit is closely related to the value function and reward function of actor-critic reinforcement learning, as above (and more in the next post, along with [my five-post Valence series (2023)](https://www.lesswrong.com/posts/As7bjEAbNpidKx6LR/valence-series-1-introduction)).

Here the “override mode” is the pleasure from putting on the cozy sweater, which can be grounded in innate brainstem circuitry related to warmth, tactile feelings, and so on. At that point, when the sweater is on, the brainstem “knows” that something good has happened.

But the more interesting question is: *what was happening during the thirty seconds that it took me to walk upstairs?* I evidently had motivation to continue walking, or I would have stopped and turned around. But my brainstem hadn’t gotten any ground truth *yet* that there were good things happening. That’s where “defer-to-predictor mode” comes in! The brainstem, lacking strong evidence about what’s happening, sees a positive valence guess coming out of the striatum and says, in effect, “OK, sure, whatever, I’ll take your word for it.”

And now we’re in the static-context situation discussed above (§5.2.2), and therefore the short-term predictor will (after some life experience) output a good guess for the next override valence, whenever that may occur. And since the cozy sweater will be a *positive* valence override, the valence guesses preceding it will likewise be positive. And therefore I will be motivated to get the sweater, including spending 30 seconds walking up the stairs.

“…Hang on,” you might be thinking. “How was that an example of *static* context? First you were on the ground floor, and then you were walking up the stairs, then opening the drawer, and so on. That’s *changing* context, right?? §5.2.3, not §5.2.2!”

My response: Yes, *part* of the context is changing, such as my immediate sensory inputs. But I claim the most important part of the context data is *not* in fact changing, and that’s *the idea, in my head, that I’m on my way to put on a sweater*. That idea is in my head the whole time, from conceiving the idea, through walking up the stairs, right through the moment when I’m pulling on my cozy sweater. And that idea is represented by some set of cortical neurons, just like everything else in my conscious awareness, and thus it’s included among the context signals. See §5.2.1.2 above.

5.4 An array of long-term predictors involving the extended striatum & Steering Subsystem
=========================================================================================

Here’s the long-term-predictor circuit from above:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a82d467ab51cfce44cf4e3773129d8a74c81f069856e29a5.png)

Copied from above.

I can lump together the 2-way switch with the rest of the genetically-hardwired circuitry, and then rearrange the boxes a bit, and I get the following:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/689436b76ae423cb87a019c4abb687475cbd506ddcbacc49.png)

Same as above, but drawn differently.

Now, obviously digestive enzymes are just one example. Let’s draw in some more examples, add some hypothesized neuroanatomy, and include other terminology. Here’s the result:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e25cac61456fd03d6b7211eb18fdaba08187869d2242068a.png)

I claim that there is a bank of long-term-predictors, consisting of an array of short-term-predictors in the extended striatum, each with a closed-loop connection to a corresponding [Steering Subsystem](https://www.alignmentforum.org/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and) circuit. I’m calling the former (striatum) part by the name “Thought Assessors”.

Excellent! We’re halfway to my big picture of decision-making and motivation. The rest of the picture—including the “actor” part of [actor-critic reinforcement learning](https://www.freecodecamp.org/news/an-intro-to-advantage-actor-critic-methods-lets-play-sonic-the-hedgehog-86d6240171d/)—will come in the [next post](https://www.alignmentforum.org/posts/qNZSBqLEh4qLRqgWW/intro-to-brain-like-agi-safety-6-big-picture-of-motivation), and will fill in the hole in the top-left side of that diagram. (The term “Steering Subsystem” comes from [Post #3](https://www.alignmentforum.org/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and).)

Here’s one more diagram and caption for pedagogical purposes.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b68011258aabc10e07fb632c207c421ce2ba1f5b67702078.png)

Reminder: a “short-term predictor” is *one component of* a “long-term predictor”. Here’s where both those things fit into that diagram above. The *only* thing that makes it a long-term predictor is the possibility of “defer-to-predictor mode”—i.e., the [Steering Subsystem](https://www.alignmentforum.org/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and) might send a “ground truth in hindsight” signal that is not *really* “ground truth” in the normal sense, but is rather a copy of the corresponding entry on the scorecard. In other words, “defer-to-predictor mode” is like the Steering Subsystem saying to the short-term predictor: “OK sure, whatever, I’ll take your word for it”. If the Steering Subsystem regularly keeps a signal in “defer-to-predictor mode” for 10 minutes straight, then we can get predictions that anticipate the future by up to 10 minutes. Conversely, if the Steering Subsystem *never* uses “defer-to-predictor mode” for a certain signal, then we shouldn’t really be calling it a “long-term predictor” in the first place.

In the next two subsections, I will elaborate on the neuroanatomy which I’m hinting at in this diagram, and then I’ll talk about why you should believe me.

5.5 Five reasons I like this “array of long-term predictors” picture
====================================================================

5.5.1 It’s a sensible way to implement a biologically-useful capability
-----------------------------------------------------------------------

If you start producing digestive enzymes *before* eating, you’ll digest faster. If your heart starts racing *before* you see the lion, then your muscles will be primed and ready to go when you *do* see the lion. Etc.

So these kinds of predictors seem obviously useful.

Moreover, as discussed in the [previous post (§4.5.2)](https://www.alignmentforum.org/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor#4_5_2__Short_term_predictor__vs_an_RL_agent__Faster_learning_thanks_to_error_gradients), the technique I’m proposing here (based on supervised learning) seems either superior to or complementary with other ways to meet these needs.

5.5.2 It’s introspectively plausible
------------------------------------

For one thing, we *do* in fact start salivating before we eat the cracker, start feeling nervous before we see the lion, etc.

For another thing, consider the fact that all the actions I’m talking about in this post are *involuntary*: you cannot salivate on command, or dilate your pupils on command, etc., at least not in quite the same way that you can wiggle your thumb on command.

(More on voluntary actions in the [next post](https://www.alignmentforum.org/posts/qNZSBqLEh4qLRqgWW/intro-to-brain-like-agi-safety-6-big-picture-of-motivation), when I talk more about the cortex.)

I’m glossing over a bunch of complications here, but the involuntary nature of these things seems pleasingly consistent with the idea that they are being trained by their own dedicated supervisory signals, straight from the brainstem. They’re slaves to a different master, so to speak. We can kinda *trick them* into behaving in certain ways, but our control is limited and indirect (see [§6.3.3 of the next post](https://www.lesswrong.com/posts/qNZSBqLEh4qLRqgWW/intro-to-brain-like-agi-safety-6-big-picture-of-motivation#6_3_3_Thought_Generator_outputs)).

5.5.3 It’s evolutionary plausible
---------------------------------

As discussed in [§4.4 of the previous post](https://www.alignmentforum.org/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor#4_4_Extremely_simplified_toy_example_of_how_this_could_work_in_biological_neurons), the simplest short-term predictor is *extraordinarily* simple, and the simplest long-term predictor is only a bit more complicated than that. And these very simple versions are already plausibly fitness-enhancing, even in very simple animals.

Moreover, as I discussed a while back ([Dopamine-supervised learning in mammals & fruit flies (2021)](https://www.lesswrong.com/posts/GnmLRerqNrP4CThn6/dopamine-supervised-learning-in-mammals-and-fruit-flies)), there is an array of little learning modules in the fruit fly, playing a seemingly-similar role to what I’m talking about here. Those modules *also* use dopamine as a supervisory signal, and there is some genomic evidence of a homology between those circuits and the mammalian telencephalon.

5.5.4 It’s neuroscientifically plausible
----------------------------------------

The details here are out-of-scope, and somewhat more complicated than I’m making it out to be, but for (a major subset of) Thought Assessors, I believe the correspondence between my toy models and real neuroanatomy is as follows:

*   The “short-term predictor” is usually[^it6alvodeqo] a set of medium spiny neurons (a.k.a. spiny projection neurons) of the extended striatum (caudate, nucleus accumbens, lateral septum, part of the amygdala, etc.)
    *   *Most* of the extended striatum is dedicated to just a few Thought Assessors, namely the “valence guess” Thought Assessor (part of the brain’s “main” / “success-in-life” RL system), or the analogous Thought Assessors for any of the brain’s narrow-RL systems (those are off-topic for this series but see [§1.5.6 of my Valence series (2023)](https://www.lesswrong.com/posts/As7bjEAbNpidKx6LR/valence-series-1-introduction#1_5_6_Fine_print__Throughout_this_series__I_m_only_talking_about_the_brain_s__main__reinforcement_learning_system)).
    *   A smaller portion of the extended striatum—specifically, parts of the amygdala, lateral septum, and nucleus accumbens shell—comprises all the dozens-to-hundreds of “visceral” Thought Assessors, related to things like cortisol, freezing, immune system activation, and so on.
*   The “Output” signal from the short-term predictor (a.k.a. the “guess”, or “scorecard” entry) goes to one of various Steering Subsystem areas (e.g. GPi, GPe, SNr, SNc, various cell groups in the hypothalamus).
*   The “Supervisor” signal going *into* the short-term predictor is one or more dopamine neurons coming back (directly or indirectly) from the Steering Subsystem.[^f7v0v8wkkvr]

Thus, for example, my claim is that if you zoom into (say) the central amygdala, you’ll find dozens of little genetically-determined subregions, each projecting to a specific set of genetically-determined Steering Subsystem targets, and with a genetically-determined narrowly-targeted supervisory signal coming back (directly or indirectly) from often the same Steering Subsystem target. Thus, for example, maybe thus-and-such little group of amygdala neurons is *universally* associated with getting goosebumps. But each individual brain would learn within its lifetime a different set of patterns that trigger this group of goosebumps-related neurons.

Is that hypothesis consistent with the experimental evidence? My impression so far, from everything I’ve read, is “yes”, although I admit I don’t have super-solid evidence either way. More specifically:

For the valence-guess Thought Assessor, the gory details are out-of-scope, but I think my story is elegantly compatible with textbook basal ganglia circuitry, particularly the famous motif of “cortico-basal ganglia-thalamo-cortical loops”.

For the “visceral” Thought Assessors: On general priors, there’s certainly no question that the genome is capable of wiring up dozens-to-hundreds of little cell groups in dozens-to-hundreds of specific and innate ways—this is ubiquitous in the Steering Subsystem. What about more specifically? I’ve found a few things. As one small-scale example, [Lischinsky et al. (2023)](https://www.nature.com/articles/s41593-023-01475-5) studied two little embryonically-defined amygdala subpopulations, and finds that these two subpopulations wind up triggering in very different circumstances and connecting to very different downstream cells. As another example, at a larger scale, [Heimer et al. (2008)](https://www.sciencedirect.com/book/9780123742391/anatomy-of-neuropsychiatry) says that the bundle of outputs of (what I’m calling) the “visceral” Thought Assessors are just a horrific mess, going every which way, as would be expected from hundreds of cell groups projecting to hundreds of different Steering Subsystem targets:

> In the case of some macrosystem outputs, such as, for example, those from the accumbens core, the fibers course among these structures in tightly fasciculated fashion, giving off robust “bursts” of terminations only within or in the vicinity of particular structures. In other cases, good examples being the accumbens shell and extended amygdaloid outputs, the fibers are more loosely, if at all, fasciculated and have countless varicosities and short collaterals presumed to be sites of synaptic or parasynaptic transmission that involve not only defined structures but also points of indeterminate neural organization all along the course of the medial forebrain bundle. … As a consequence of their descent into this poorly defined organization, it is exceedingly difficult to conceive of precise, definitive mechanisms that outputs from the forebrain, including from the macrosystems, might engage to orchestrate purposeful, adaptive behavior from the extensive and sophisticated repertoire of autonomous, albeit restrictively programmed, hindbrain-spinal motor routines. But it is equivalently hard to dispel the notion that this is precisely what happens, and it seems advisable to presume that we are at present unable to perceive the relevant functional-anatomical relationships rather than to think they don’t exist.

As another piece of suggestive evidence, [Lammel et al. (2014)](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3778102/) mentions so-called “‘non-conventional’ VTA \[dopamine\] neurons” in “medial posterior VTA (PN and medial PBP)”. These seem to project to roughly the “visceral” Thought Assessor areas that I mentioned above, and it’s claimed that they have different firing patterns from other dopamine neurons. That seems intriguing, although I don’t have a more specific story than that right now. Relatedly, [Verharen et al. (2020)](https://doi.org/10.1016/j.conb.2020.02.002) discusses certain dopamine neurons which burst when *aversive* things happen, and four of the five regions these neurons seem to “supervise” are home to visceral Thought Assessors.[^2jlkqfz5auw] This makes sense because if the mouse freezes in terror, then that’s a positive error signal for the freeze-in-terror visceral Thought Assessor, as well as the raise-your-cortisol-level visceral Thought Assessor, and so on.

5.5.5 It offers a nice way to make sense of a wide variety of animal psychology phenomena, including Pavlovian conditioning, devaluation, and more
--------------------------------------------------------------------------------------------------------------------------------------------------

See my post [Incentive Learning vs Dead Sea Salt Experiment (2024)](https://www.lesswrong.com/posts/YQ4rSTHpHeFcAmhvi/incentive-learning-vs-dead-sea-salt-experiment) for details on that. See also [the last section of that post](https://www.lesswrong.com/posts/YQ4rSTHpHeFcAmhvi/incentive-learning-vs-dead-sea-salt-experiment#5__Bonus__Some_comments_on_how_my_model_differs_from_some_stuff_in_the_neuro_psych_literature) for some contrasts between my model and a couple others in the psychology and neuro-AI literature.

5.6 Conclusion
==============

Anyway, as usual I don’t pretend to have smoking-gun proof of my hypothesis (i.e. that the brain has an array of long-term predictors involving extended striatum-Steering Subsystem loops), and there are some bits that I’m still confused about. But considering the evidence in the previous subsection, and everything else I’ve read, I wind up feeling strongly that I’m broadly on the right track. I’m happy to discuss more in the comments. Otherwise, onward to the [next post](https://www.alignmentforum.org/posts/qNZSBqLEh4qLRqgWW/intro-to-brain-like-agi-safety-6-big-picture-of-motivation), where we will finally put everything together into a big picture of how I think motivation and decision-making work in the brain!

Changelog
=========

*July 2024:* Since the initial version, I’ve made a bunch of changes.

The biggest changes are related to neuroanatomy. I was just really confused about lots of things in 2022, but I hope I’m converging towards correct answers! In particular:

I’ve gone through several iterations of the diagrams in this post, particularly including which neuroanatomy corresponds to which boxes.

There used to be a much longer discussion of neuroanatomy in §5.4, but much of that was incorrect, and the rest got moved into the newly-rewritten §5.5.4.

I originally had a discussion of dopamine diversity. I mostly deleted that, apart from a couple sentences, having been convinced that at least one of the pieces of evidence I discussed ([Engelhart et al. 2019](https://www.nature.com/articles/s41586-019-1261-9)) was *not* related to “visceral Thought Assessors”, but rather had a different explanation. (Thanks Nathaniel Daw for the correction.) I also moved the discussion of distributional RL into a footnote of the previous post.

More minor things:

I added a note that all the diagrams in this post should be understood as the kinds of diagrams you see in FPGA design—i.e. at every instant, every block is running simultaneously, and every line (wire) is carrying a numerical value. It’s *not* like the kinds of diagrams you see in descriptions of serial program control flow, where we follow a path through the diagram, executing one thing at a time.

I’ve also changed the wording on the labels of the various diagrams a few times since the initial version. For example, where it used to say “Will lead to reward?” and “It *was* in fact leading to reward”, it now says “Valence guess” and “Actual valence”, respectively. The previous wording was OK, but I think the connotations are a bit better here, and also more consistent with my later [Valence series](https://www.lesswrong.com/posts/As7bjEAbNpidKx6LR/valence-series-1-introduction).

I added a second concrete example story (involving a cozy sweater) as §5.3.2.

I replaced a brief discussion of the “dead sea salt experiment” with a link to my more recent [post dedicated to that topic](https://www.lesswrong.com/posts/YQ4rSTHpHeFcAmhvi/incentive-learning-vs-dead-sea-salt-experiment), in §5.5.5.

*January 2026:* I added a new §5.2.1 (“preliminaries”) to hopefully explain the toy model better. (The old §5.2.1 is now §5.2.2, and the old §5.2.2 is now §5.2.3.) Plus various other minor copyedits.

[^nmsgsjxd32]: To be clear, in reality, there probably isn’t a discrete all-or-nothing 2-way switch here. There could be a “weighted average” setting, for example. Remember, this whole discussion is just a pedagogical “toy model”; I expect that reality is more complicated in various respects. 

[^d1mr9i8su2a]: I note that I’m just running through this algorithm in my head; I haven’t simulated it. I’m optimistic that I didn’t majorly screw up, i.e. that everything I’m saying about the algorithm is qualitatively true, or at least can be qualitatively true with appropriate parameter settings and perhaps other minor tweaks.  

[^t5hi9l336w]: To be clear, this transient discrepency has nothing to do with signal propagation delays. In this toy model, I declare that signal propagation is instantaneous. Rather, it’s from the 0.3 second (or whatever) time-offset internal to the short-term predictor. 

[^v6aq2sfj7bk]: Examples of using the terminology “TD learning” for something which is not related to RL reward functions include “TD networks”, and the Successor Representations literature (example), or this paper, etc. 

[^it6alvodeqo]: Not always—certain parts of cortex layer 5 are also Thought-Assessor-ish. But meanwhile, most of the cortex is doing something different, as discussed in the next post. 

[^f7v0v8wkkvr]: As in the previous post, when I say that “dopamine carries the supervisory signal”, I’m open to the possibility that dopamine is actually a closely-related signal like the error signal, or the negative error signal, or the negative supervisory signal. It really doesn’t matter for present purposes. 

[^2jlkqfz5auw]: The fifth area where that paper found dopamine neurons bursting under aversive circumstances, namely the tail of the striatum, has a different explanation I think—see here.