# [Intro to brain-like-AGI safety] 6. Big picture of motivation, decision-making, and RL

*(Last revised: January 2026. See changelog at the bottom.)*

6.1 Post summary / Table of contents
====================================

*Part of the *[*“Intro to brain-like-AGI safety” post series*](https://www.alignmentforum.org/s/HzcM2dkCq7fwXBej8)*.*

Thus far in the series, [Post #1](https://www.alignmentforum.org/posts/4basF9w9jaPZpoC8R/intro-to-brain-like-agi-safety-1-what-s-the-problem-and-why) set out some definitions and motivations (what is “brain-like AGI safety” and why should we care?), and Posts [#2](https://www.alignmentforum.org/posts/wBHSYwqssBGCnwvHg/intro-to-brain-like-agi-safety-2-learning-from-scratch-in) & [#3](https://www.alignmentforum.org/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and) split the brain into a Learning Subsystem (cortex, striatum, cerebellum, amygdala, etc.) that “learns from scratch” using learning algorithms, and a Steering Subsystem (hypothalamus, brainstem, etc.) that is mostly genetically-hardwired and executes innate species-specific instincts and reactions.

Then in [Post #4](https://www.alignmentforum.org/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor), I talked about the “short-term predictor”, a circuit which learns, via supervised learning, to predict a signal in advance of its arrival, but only by perhaps a fraction of a second. [Post #5](https://www.alignmentforum.org/posts/F759WQ8iKjqBncDki/intro-to-brain-like-agi-safety-5-the-long-term-predictor-and) then argued that if we form a closed loop involving *both* a set of short-term predictors in the extended striatum (within the Learning Subsystem) *and* a corresponding set of hardwired circuits in the Steering Subsystem, we can get a “long-term predictor”. I noted that the “long-term predictor” circuit is closely related to [temporal difference (TD) learning](https://en.wikipedia.org/wiki/Temporal_difference_learning).

Now in this post, we fill in the last ingredients—roughly the “actor” part of [actor-critic reinforcement learning](https://www.freecodecamp.org/news/an-intro-to-advantage-actor-critic-methods-lets-play-sonic-the-hedgehog-86d6240171d/) (RL)—to get a whole big picture of motivation and decision-making in the human brain. (I’m saying “human brain” to be specific, but it would be a similar story in any other mammal, and to a lesser extent in any vertebrate.)

The reason I care about motivation and decision-making is that if we eventually build brain-like AGIs (cf. [Post #1](https://www.alignmentforum.org/posts/4basF9w9jaPZpoC8R/intro-to-brain-like-agi-safety-1-what-s-the-problem-and-why)), we’ll want to build them so that they have some motivations (e.g. being helpful) and not others (e.g. escaping human control and self-reproducing around the internet). Much more on that topic in later posts.

*Teaser for upcoming posts:* The next post ([#7](https://www.alignmentforum.org/posts/zXibERtEWpKuG5XAC/intro-to-brain-like-agi-safety-7-from-hardcoded-drives-to)) will walk through a concrete example of the model in this post, where we can watch an innate drive lead to the formation of an explicit goal, and adoption and execution of a plan to accomplish it. Then starting in [Post #8](https://www.alignmentforum.org/posts/fDPsYdDtkzhBp9A8D/intro-to-brain-like-agi-safety-8-takeaways-from-neuro-1-2-on) we’ll switch gears, and from then on you can expect substantially *less* discussion of neuroscience and *more* discussion of AGI safety (with the exception of one more neuroscience post towards the end).

Unless otherwise mentioned, everything in this post is “things that I believe right now”, as opposed to neuroscience consensus. *(*[*Pro tip: there is never a neuroscience consensus*](https://x.com/steve47285/status/1707085848309645775)*.)* Relatedly, I will make minimal effort to connect my hypotheses to others in the literature (I tried a little bit in the last section of my post [“Incentive Learning vs Dead Sea Salt Experiment” (2024)](https://www.lesswrong.com/posts/YQ4rSTHpHeFcAmhvi/incentive-learning-vs-dead-sea-salt-experiment#5__Bonus__Some_comments_on_how_my_model_differs_from_some_stuff_in_the_neuro_psych_literature)), but I’m happy to chat about that in the comments section or by [email](mailto:steven.byrnes@gmail.com).

*Table of contents:*

*   In §6.2, I’ll present a big picture of motivation and decision-making in the human brain, and walk through how it works. The rest of the post will go through different parts of that picture in more detail. If you’re in a hurry, I suggest reading to the end of §6.2 and then quitting.
*   In §6.3, I’ll talk about the so-called “Thought Generator”, which corresponds more-or-less to the cortex.[^1bg9x09yd83h] (For ML readers familiar with “actor-critic model-based RL”, the Thought Generator is more-or-less a combination of the “actor” and the “model”.) I’ll talk about the inputs and outputs of this module, and briefly sketch how its algorithm relates to neuroanatomy.
*   In §6.4, I’ll talk about how values and rewards work in this picture, including the reward signal that drives learning and decision-making in the Thought Generator.
*   In §6.5, I’ll go into a bit more detail about how and why thinking and decision-making needs to involve not only simultaneous comparisons (i.e., a mechanism for generating different options in parallel and selecting the most promising one), but also sequential comparisons (i.e., thinking of something, then thinking of something else, and comparing those two thoughts). For example, you might think: “Hmm, I think I’ll go to the gym. Actually, what if I went to the café instead?” These are two sequential thoughts that you’re comparing.
*   In §6.6, I’ll comment on the common misconception that the Learning Subsystem is the home of ego-syntonic, internalized “deep desires”, whereas the Steering Subsystem is the home of ego-dystonic, externalized “primal urges”. I will advocate more generally against thinking of the two subsystems as two agents in competition; a better mental model is that the two subsystems are two interconnected gears in a single machine.

6.2 Big picture
===============

Yes, this is literally a big picture, unless you’re reading on your cell phone. You saw a chunk of it in the [previous post (§5.4)](https://www.alignmentforum.org/posts/F759WQ8iKjqBncDki/intro-to-brain-like-agi-safety-5-the-long-term-predictor-and#5_4_An_array_of_long_term_predictors_involving_the_extended_striatum___Steering_Subsystem), but now there are a few more pieces.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/98b87bbe1f7b536807f621aac60150da2959f5df61afeb24.png)

The big picture—The whole post will revolve around this diagram. Note that I’m oversimplifying in various ways, including in the bracketed neuroanatomy labels.

There’s a lot here, but don’t worry, I’ll walk through it bit by bit.

6.2.1 Relation to “two subsystems”
----------------------------------

Here’s how this diagram fits in with my “two subsystems” perspective, first discussed in [Post #3](https://www.alignmentforum.org/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8ce8c3417e38318629a94f89139eff23ce62a1dad29857d2.png)

Same as above, but the [two subsystems](https://www.alignmentforum.org/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and) are highlighted in different colors.

6.2.2 Quick run-through
-----------------------

Before getting bogged down in details later in the post, I’ll just talk through the diagram:

*1\. Thought Generator generates a thought:* The Thought Generator settles on a “thought”, out of the high-dimensional space of *every thought you can possibly think* at that moment. Note that this space of possibilities, while vast, is constrained by current sensory input, past sensory input, and everything else in your learned world-model. For example, if you’re sitting at a desk in Boston, it’s generally not possible for you to *think* that you’re scuba-diving off the coast of Madagascar. Likewise, it’s generally not possible for you to imagine a static spinning spherical octagon. But you *can* make a plan, or whistle a tune, or recall a memory, or reflect on the meaning of life, etc.

*2\. Thought Assessors grade the thought via a “scorecard”:* The Thought Assessors are a set of perhaps hundreds or thousands of “short-term predictor” circuits [(Post #4)](https://www.alignmentforum.org/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor), which I discussed more specifically in [the previous post (#5)](https://www.alignmentforum.org/posts/F759WQ8iKjqBncDki/intro-to-brain-like-agi-safety-5-the-long-term-predictor-and). Each predictor is trained to predict a different signal from the Steering Subsystem. From the perspective of a Thought Assessor, everything in the Thought Generator (not just outputs but also latent variables) is *context* ([§4.3](https://www.lesswrong.com/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor#4_3_Terminology__Context__Output__Supervisor))—information that they can use to make better predictions. Thus, if I’m thinking the thought “I’m going to eat candy right now”, a thought-assessor can predict “high probability of tasting something sweet very soon”, based *purely on the thought*—it doesn’t *need* to rely on either external behavior or sensory inputs, although those can be relevant context too.

*3\. The “scorecard” solves the interface problem between a learned-from-scratch world model and genetically-hardwired circuitry:* Remember, the current thought and situation is an insanely complicated object in a high-dimensional learned-from-scratch space of “all possible thoughts you can think”. Yet we need the relatively simple, genetically-hardwired circuitry of the Steering Subsystem to analyze the current thought, including issuing a judgment of whether the thought is overall good vs bad (see §6.4 below), and whether the thought calls for cortisol release or goosebumps or pupil-dilation, etc. The “scorecard” solves that interfacing problem! It distills any possible thought / belief / plan / etc. into a genetically-standardized form that can be plugged directly into genetically-hardcoded circuitry.

*4\. The Steering Subsystem runs some genetically-hardwired algorithm:* Its inputs are (1) the scorecard from the previous step and (2) various other information sources—pain, metabolic status, etc., all coming from its own brainstem sensory-processing system (see [§3.2.1](https://www.alignmentforum.org/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and#3_2_1_Each_subsystem_generally_needs_its_own_sensory_processor)). Its outputs include emitting hormones, motor commands, etc., as well as sending the “ground truth” supervisory signals shown in the diagram.[^49k95mpuwpr] 

*5\. The Thought Generator keeps or discards thoughts based on whether the Steering Subsystem likes them:* More specifically, there’s a ground-truth [valence](https://www.lesswrong.com/posts/As7bjEAbNpidKx6LR/valence-series-1-introduction) (or in RL terms, some weird mixture of value function and reward function, see [§5.3.1](https://www.lesswrong.com/posts/F759WQ8iKjqBncDki/intro-to-brain-like-agi-safety-5-the-long-term-predictor-and#5_3_1_Switch__i_e___value___expected_next_reward__vs_summation__i_e___value___expected_sum_of_future_rewards__)). When the valence is very positive, the current thought gets “strengthened”, sticks around, and can start controlling behavior and summoning follow-up thoughts, whereas when the valence is very negative, the current thought gets immediately discarded, and the Thought Generator summons a new thought instead.

*6\. Both the Thought Generator and the Thought Assessor* [*“learn from scratch”*](https://www.alignmentforum.org/posts/wBHSYwqssBGCnwvHg/intro-to-brain-like-agi-safety-2-learning-from-scratch-in) *over the course of a lifetime, thanks in part to these supervisory signals from the Steering Subsystem.* Specifically, the Thought Assessors learn to make better and better predictions of their “ground truth in hindsight” signal (a form of Supervised Learning—see [Post #4](https://www.alignmentforum.org/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor)), while the Thought Generator learns to generate maximally “good” (positive-valence) thoughts. (The Thought Generator learning-from-scratch process *also* involves predictive learning of sensory inputs—[§4.7](https://www.alignmentforum.org/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor#4_7__Short_term_predictor__example__2__Predictive_learning_of_sensory_inputs_in_the_cortex).)

6.3 The “Thought Generator”
===========================

6.3.1 Overview
--------------

Go back to the big-picture diagram at the top. At the top-left, we find the Thought Generator. In terms of actor-critic model-based RL, the Thought Generator is roughly a combination of “actor” + “model”, but not “critic”. (“Critic” was discussed in the previous post, and more on it below.)

At our somewhat-oversimplified level of analysis, we can think of the “thoughts” generated by the Thought Generator as a combination of *constraints* (from predictive learning of sensory inputs) and *choices* (guided by reinforcement learning). In more detail:

*   *Constraints* on the Thought Generator come from sensory input information, and ultimately from predictive learning of sensory inputs ([§4.7](https://www.alignmentforum.org/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor#4_7__Short_term_predictor__example__2__Predictive_learning_of_sensory_inputs_in_the_cortex)). For example, I *cannot *think the thought: *There is a cat on my desk and I’m looking at it right now.* There is no such cat, alas, and I can’t just will myself to see something that obviously isn’t there. I can *imagine* seeing it, but that’s not the same thought. Likewise, I can’t imagine a static spinning spherical octagon, because that’s not allowed by my generative world-model—those concepts are self-contradictory. Yet another source of constraints on the Thought Generator is what I call “involuntary attention”—like how if you’re very itchy, it’s hard to think about anything else (see table in [Valence series §3.3.5 (2023)](https://www.lesswrong.com/posts/rM8DwFKZM4eB7i2p8/valence-series-3-valence-and-beliefs#3_3_5_An_exception__I_think_anxious___obsessive__brainstorming__is_driven_by_involuntary_attention_rather_than_by_valence)).
*   But *within *those constraints, there’s more than one possible thought my brain can think at any given time. It can call up a memory, it can ponder the meaning of life, it can zone out, it can issue a command to stand up, etc. I claim that these “choices” are decided by a reinforcement learning (RL) system. This RL system is one of the main topics of this post.

6.3.2 Thought Generator inputs
------------------------------

The Thought Generator has a number of inputs, including sensory inputs and hyperparameter-shifting neuromodulators. But the main one of interest for this post is [valence](https://www.lesswrong.com/posts/As7bjEAbNpidKx6LR/valence-series-1-introduction), a.k.a. ground-truth value, a.k.a. reward. (See [previous post](https://www.lesswrong.com/posts/F759WQ8iKjqBncDki/intro-to-brain-like-agi-safety-5-the-long-term-predictor-and#5_3_1_Switch__i_e___value___expected_next_reward__vs_summation__i_e___value___expected_sum_of_future_rewards__) for why I’m equating “ground-truth value” with “reward”, contrary to how it normally works in RL.) I’ll talk about that in more detail later, but we can think of it as *an estimate of whether a thought is good or bad*, operationalized as “worth sticking with and pursuing” versus “deserving to be discarded so we can re-roll[^ooqlv9o7fbr] for a new thought”. This signal is important *both* for learning to think better thoughts in the future, *and* for thinking good thoughts right now:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ecf1bdf1ed52e96154daf397a61a2af1deabbbb065d8fdf6.png)

6.3.3 Thought Generator outputs
-------------------------------

There are meanwhile a lot of signals going *out* of the Thought Generator. Some are what we intuitively think of as “outputs”—e.g., skeletal motor commands. Other outgoing signals are, well, a bit funny…

Recall the idea of “context” from [§4.3](https://www.alignmentforum.org/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor#4_3_Terminology__Context__Output__Supervisor): The Thought Assessors are short-term predictors, and a short-term predictor can in principle grab any signal in the brain and leverage it to improve its ability to predict its target signal. So if the Thought Generator has a world-model, then *somewhere* in the world-model is a configuration of latent variable activations that encode the concept “baby kittens shivering in the cold rain”. We wouldn’t normally think of those as “output signals”—I just said in the last sentence that they’re latent variables! But as it happens, the “will lead to crying” Thought Assessor has grabbed a copy of those latent variables to use as context signals, and gradually learned through experience that these particular signals are strong predictors of me crying.

Now, as an adult, these “baby kittens in the cold rain” neurons in my Thought Generator are living a double-life:

*   They are latent variables in my world-model—i.e., they and their web of connections will help me parse an image of baby kittens in the rain, if I see one, and to reason about what would happen to them, etc.
*   Activating these neurons, e.g. via imagination, is *a way for me to call up tears on command*.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/36447fde386f5c93e10f8b9af504f37a28be04fc23ce71a5.png)

The Thought Generator (top left) has two types of outputs: the “traditional” outputs associated with voluntary behavior (green arrows) and the “funny” outputs wherein even latent variables in the model can directly impact involuntary behaviors (blue arrows).

6.4 Valence, value, and rewards
===============================

6.4.1 Teaser for [my “Valence” series (2023)](https://www.lesswrong.com/posts/As7bjEAbNpidKx6LR/valence-series-1-introduction)
------------------------------------------------------------------------------------------------------------------------------

“Valence”, the way I use the term, is a specific signal on the left side of the (somewhat oversimplified) diagram at the top. As mentioned above (§6.3.2), it signals whether a thought is good (so I should keep thinking it, and think follow-up thoughts, and take associated actions), or bad (so I should start thinking about something else instead).

For psychology readers, think of valence as somewhat more related to “motivational valence” than “hedonic valence”, although it’s not a perfect match to either.

For AI readers, think of valence as a funny mix of value-function and reward. Like reward, it can represent innate “ground truth” (e.g. pain is bad, sweet tastes are good, etc.) But like the value function, it can be forward-looking—as in the [example of the previous post](https://www.lesswrong.com/posts/F759WQ8iKjqBncDki/intro-to-brain-like-agi-safety-5-the-long-term-predictor-and#5_3_2_Example), if I’m going upstairs to get a sweater, that thought has positive valence *now* for reasons related to what’s going to happen *in the future* (I’ll be wearing my sweater).

While I think [there are multiple RL systems in the brain](https://www.lesswrong.com/posts/As7bjEAbNpidKx6LR/valence-series-1-introduction#1_5_6_Fine_print__Throughout_this_series__I_m_only_talking_about_the_brain_s__main__reinforcement_learning_system), the “valence” signal is definitionally associated with the “main” RL system, a.k.a. what I sometimes call “success-in-life RL”—the RL system that navigates cross-domain tradeoffs (e.g. is it worth suffering X amount of pain in order to get Y amount of social status?), in order to estimate what’s a good plan *all things considered*.

[My 5-post “Valence” series](https://www.lesswrong.com/posts/As7bjEAbNpidKx6LR/valence-series-1-introduction) (published in late 2023) notes that, given the obvious central importance of this “valence” brain signal‚ it should illuminate many aspects of our everyday mental life. And indeed, it does! I argue in the series that valence is critical to understanding desires, preferences, values, goals, judgements, vibes, brainstorming, motivated reasoning, the halo effect, social status, depression, mania, and more.

6.4.2 The striatum (“learned value function” / “critic”) guesses the valence, but the Steering Subsystem may choose to override
-------------------------------------------------------------------------------------------------------------------------------

There are two types of “valence” in the diagram (it looks like three, but the two red ones are the same):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f8c4b1c2d894cbe940aade92548ae62773b20bd628b0d10b.png)

Two types of “valence” in my model—“real” and “guessed”

The blue-circled signal is the valence guess from the corresponding Thought Assessor in the striatum. The red-circled signal (again, it’s one signal drawn twice) is the corresponding “ground truth” for what the valence guess *should* have been.

Just like the other “long-term predictors” discussed in [the previous post](https://www.alignmentforum.org/posts/F759WQ8iKjqBncDki/intro-to-brain-like-agi-safety-5-the-long-term-predictor-and), the long-term predictor for valence has a “defer-to-predictor mode” and an “override mode”, and the Steering Subsystem can dynamically switch between these modes. In defer-to-predictor mode, it sets the red equal to the blue, as if to say “OK, Thought Assessor, sure, I’ll take your word for it”. In override mode, it ignores the Thought Assessor’s proposal, and its own internal circuitry outputs some different value.[^g0mylfbpkdp]

**Why might the Steering Subsystem override the Thought Assessor’s valence guess?** Two factors:

*   First, the Steering Subsystem might be acting on information from *other* (non-valence) Thought Assessors. For example, in the so-called “Dead Sea Salt Experiment” (see [§2.1 of my 2024 post on that](https://www.lesswrong.com/posts/YQ4rSTHpHeFcAmhvi/incentive-learning-vs-dead-sea-salt-experiment#2_1__Dead_Sea_Salt_Experiment__3_)), the valence guess says “bad things are going to happen”, but meanwhile the Steering Subsystem is getting an “I’m about to taste salt” prediction in the context of a state of salt-deprivation. So the Steering Subsystem says to itself “Whatever is happening now is very promising; the valence guess is nonsense!”
*   Second, the Steering Subsystem might be acting on its own information sources, independent of the Learning Subsystem. In particular, the Steering Subsystem has its own sensory-processing system (see [§3.2.1](https://www.alignmentforum.org/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and#3_2_1_Each_subsystem_generally_needs_its_own_sensory_processor)), which can sense biologically-relevant cues like pain status, hunger status, taste inputs, the sight of a slithering snake, the smell of a potential mate, and so on. All these things and more can be possible bases for overruling the Thought Assessor, i.e., setting the red-circled signal to a different value than the blue-circled one.

6.5 Decisions involve not only simultaneous but also sequential comparisons of value
====================================================================================

Here’s a “simultaneous” model of decision-making, as described by [*The Hungry Brain* by Stephan Guyenet](https://www.amazon.com/Hungry-Brain-Outsmarting-Instincts-Overeat/dp/125008119X) in the context of studies on lamprey fish:

> Each region of the pallium \[= lamprey equivalent of cortex\] sends a connection to a particular region of the striatum, which (via other parts of the basal ganglia) returns a connection back to the same starting location in the pallium. This means that each region of the pallium is reciprocally connected with the striatum via a specific loop that regulates a particular action…. For example, there’s a loop for tracking prey, a loop for fleeing predators, a loop for anchoring to a rock, and so on. Each region of the pallium is constantly whispering to the striatum to let it trigger its behavior, and the striatum always says “no!” by default. In the appropriate situation, the region’s whisper becomes a shout, and the striatum allows it to use the muscles to execute its action. 

I endorse this as *part* of my model of decision-making, but only part of it. Specifically, this is one of the things that’s happening when the Thought Generator generates a thought. Different *simultaneous* possibilities are being compared.

The other part of my model is comparisons of *sequential* thoughts. You think a thought, and then you think a different thought (possibly very different, or possibly a refinement of the first thought), and the two are implicitly compared, and if the second thought is worse, it gets weakened such that a new thought can replace it (and the new thought might be the first thought re-establishing itself).

I could cite experiments for the sequential-comparison aspect of decision-making (e.g. Figure 5 of  [Herd et al. 2021](https://doi.org/10.3758/s13415-020-00842-0), which is arguing the same point as I am), but do I really need to? Introspectively, it’s obvious! You think: “Hmm, I think I’ll go to the gym. Actually, what if I went to the café instead?” You’re imagining one thing, and then another thing.

And I don’t think this is is a humans-vs-lampreys thing. I think comparison-of-sequential-thoughts is universal in vertebrates. As an illustration of what I mean:

6.5.1 Made-up example of what comparison-of-sequential-thoughts might look like in a simpler animal
---------------------------------------------------------------------------------------------------

Imagine a simple, ancient, little fish swimming along, navigating to the cave where it lives. It gets to a fork in the ~~road~~ kelp forest. Its current thought is something like “I am swimming left to my cave”. But it also has the option of turning right to go to the reef, where it often forages.

Seeing this path to the right, I claim that a new thought pops into its head: “I am swimming right and going to the reef.” This new thought / plan is immediately evaluated by the Thought Assessors and Steering Subsystem as usual, and winds up with some valence (implicitly in comparison to the old plan). If this new thought seems bad, then it gets thrown out, and the old thought (“I am swimming left to my cave”) promptly pops back into its head. The fish continues to its cave, as originally planned, without skipping a beat. (Except maybe, if you look carefully, you could notice the fish subtly twitch to the right for a fraction of a second.) Whereas if instead the new thought seems good (i.e., better than the old thought), then it gets strengthened, sticks around, and orchestrates motor commands. And thus the fish turns to the right and goes to the reef instead.

(In reality, I don’t know much about ancient fish, but rats at a fork in the ~~road~~ maze are known to imagine both possible navigation plans in succession, based on measurements of hippocampus neurons—[Johnson & Redish 2007](https://doi.org/10.1523/JNEUROSCI.3761-07.2007).)

6.5.2 Comparison-of-sequential-thoughts: why it’s necessary
-----------------------------------------------------------

In my view, thoughts are complicated. To think the thought “I will go to the café”, you’re not just activating some tiny cluster of dedicated go-to-the-café neurons. Instead, it’s a distributed pattern involving practically every part of the cortex. You can’t simultaneously think “I will go to the café” and “I will go to the gym”, because they would involve different activity patterns of the same pools of neurons. They would cross-talk. Thus, the only possibility is thinking the thoughts in sequence.

As a concrete example of what I have in mind, think of how a [Hopfield network](https://en.wikipedia.org/wiki/Hopfield_network) can’t recall twelve different memories simultaneously. It has multiple stable states, but you can only explore them sequentially, one after the other. Or think about grid cells and place cells, etc.

6.5.3 Comparison-of-sequential-thoughts: how it might have evolved
------------------------------------------------------------------

From an evolutionary perspective, I imagine that comparison-of-sequential-thoughts is a distant descendent of a very simple mechanism akin to the [run-and-tumble mechanism in swimming bacteria](https://en.wikipedia.org/wiki/Run-and-tumble_motion).

In the run-and-tumble mechanism, a bacterium swims in a straight line (“runs”), and periodically changes to a new random direction (“tumbles”). But the trick is: when the bacterium’s situation / environment is getting *better*, it tumbles *less* frequently, and when it’s getting *worse*, it tumbles *more* frequently. Thus, it winds up moving in a good direction (on average, over time).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0825c5e313b2d8a0d3c80b35e5ad0287bf6aadf0ff1a7994.png)

Starting with a simple mechanism like that, one can imagine adding progressively more bells and whistles. The palette of behavioral options can get more and more complex, eventually culminating in “every thought you can possibly think”. The methods of evaluating whether the current plan is good or bad can get faster and more accurate, eventually involving learning-algorithm-based predictors as in [the previous post](https://www.alignmentforum.org/posts/F759WQ8iKjqBncDki/intro-to-brain-like-agi-safety-5-the-long-term-predictor-and). The new behavioral options to tumble *into* can be picked via clever learning algorithms, rather than randomly. Thus, it seems to me that there’s a smooth path all the way from something-akin-to-run-and-tumble to the intricate, finely-tuned, human brain system that I’m talking about in this series. (Other discussions of run-and-tumble versus human motivation: [Karin & Alon 2022](https://doi.org/10.1371/journal.pcbi.1010340), [Travis 2022](https://osf.io/preprints/psyarxiv/wzvn9).)

6.6 Common misconceptions
=========================

6.6.1 The distinction between internalized ego-syntonic desires and externalized ego-dystonic urges is unrelated to Learning Subsystem vs. Steering Subsystem
-------------------------------------------------------------------------------------------------------------------------------------------------------------

Many people (including me) have a strong intuitive distinction between [ego-syntonic drives](https://en.wikipedia.org/wiki/Egosyntonic_and_egodystonic) that are “part of us” or “what we want”, versus [ego-dystonic drives](https://en.wikipedia.org/wiki/Egosyntonic_and_egodystonic) that feel like urges which intrude upon us from the outside.

For example, a food snob might say "I love fine chocolate", while a dieter might say "I have an urge to eat fine chocolate".

### 6.6.1.1 The explanation I like

I would claim that these two people are basically describing the same feeling, with essentially the same neuroanatomical locations and essentially the same relation to low-level brain algorithms. But the food snob is *owning* that feeling, and the dieter is *externalizing* that feeling.

These two different self-concepts go hand-in-hand with two different “higher-order preferences”: the food snob *wants to want to eat fine chocolate *while the dieter *wants to not want to eat fine chocolate*.

This leads us to a straightforward psychological explanation for why the food snob and dieter conceptualize their feelings differently, based on motivated reasoning:

*   The food snob finds it *appealing* to think of “the desire I feel for fine chocolate” as “part of who I am”. So he does.
*   The dieter finds it *aversive* to think of “the desire I feel for fine chocolate” as “part of who I am”. So he doesn’t.

I actually think that’s a good start but not quite right; for my actual opinion about what’s going on, see my [Intuitive Self-Models series (2024)](https://www.lesswrong.com/s/qhdHbCJ3PYesL9dde), especially [its §3.5.4](https://www.lesswrong.com/posts/7tNq4hiSWW9GdKjY8/intuitive-self-models-3-the-homunculus#3_5_4_Why_are_ego_dystonic_things__externalized__).

### 6.6.1.2 The explanation I don’t like 

Many people (including Jeff Hawkins, see [Post #3](https://www.lesswrong.com/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and)) notice the distinction described above, and separately, they endorse the idea (as I do) that the brain has a Learning Subsystem and Steering Subsystem (again see [Post #3](https://www.lesswrong.com/posts/hE56gYi5d68uux9oM/intro-to-brain-like-agi-safety-3-two-subsystems-learning-and)). They naturally suppose that these are the same thing, with “me and my deep desires” corresponding to the Learning Subsystem, and “urges that I don’t identify with” corresponding to the Steering Subsystem. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/19c8cc0aa8817e75f9ea991040569fa437dbe959565f6cfa.png)

Most people I talk to, including me, have separate concepts in our learned world-models for “me” and “my urges”. I claim that these concepts did *NOT* come out of veridical introspective access to our own neuroanatomy. And in particular, they do *not *correspond respectively to the Learning & Steering Subsystems.

I think this model is wrong. At the *very* least, if you want to endorse this model, then you need to reject approximately everything I’ve written in this and my previous four posts.

In *my *story, if you’re trying to abstain from chocolate, but also feel an urge to eat chocolate, then:

*   You have an urge to eat chocolate right now because the Steering Subsystem approves of the thought “I am going to eat chocolate right now”; AND
*   You’re trying to abstain from chocolate because the Steering Subsystem approves of the thought “I am abstaining from chocolate”.

(Why would the Steering Subsystem approve of the latter? It depends on the individual, but it’s probably a safe bet that social instincts are involved. I’ll talk more about social instincts in [Post #13](https://www.alignmentforum.org/posts/5F5Tz3u6kJbTNMqsb/intro-to-brain-like-agi-safety-13-symbol-grounding-and-human). If you want an example with less complicated baggage, imagine a lactose-intolerant person trying to resist the urge to eat yummy ice cream right now, because it will make them feel really sick later on. The Steering Subsystem likes plans that result in not feeling sick, and *also* likes plans that result in eating yummy ice cream.)

Here are four views on why we should believe that the Steering Subsystem is the ultimate source of not only ego-dystonic urges like hunger, but also ego-syntonic desires like friendship and justice.

*   *AI perspective:* We don’t yet know *in full detail* how model-based RL and model-based planning works in the human brain—we don’t have brain-like AGI yet. But we do at least vaguely know how these kinds of algorithms work. And we know enough to say for sure that these algorithms don’t develop prosocial motivations out of nowhere. For example, if you set the reward function of MuZero to always return 0, then the algorithm will emit random outputs forever—it won’t start fighting for justice.
*   *Rodent model perspective:* For what it’s worth, researchers have been equally successful in finding little cell groups in the rodent hypothalamus that orchestrate “antisocial” behaviors like aggression, and that orchestrate “prosocial” behaviors like parenting and sociality. I fully expect that the same holds for humans.
*   *Philosophy perspective:* Without the Steering Subsystem, the only thing the cortex can do is build a world-model from predictive learning of sensory inputs ([§4.7](https://www.lesswrong.com/posts/Y3bkJ59j4dciiLYyw/intro-to-brain-like-agi-safety-4-the-short-term-predictor#4_7__Short_term_predictor__example__2__Predictive_learning_of_sensory_inputs_in_the_cortex)). That’s “is”, not “ought”. And [“Hume’s law”](https://en.wikipedia.org/wiki/Is%E2%80%93ought_problem) says that you can’t get “ought”-statements from exclusively “is”-statements. Granted, not everyone believes in Hume’s law. But I do—see an elegant and concise argument for it [here](https://intelligence.org/2018/02/28/sam-harris-and-eliezer-yudkowsky/#:~:text=The%20question%20%E2%80%9CHow%20many%20paperclips%20result%20if%20I%20follow%20this%20policy%3F%E2%80%9D%20is%20an%20%E2%80%9Cis%E2%80%9D%20question.).
*   *My nuts-and-bolts proposals:* If you keep reading this series, along with follow-up work [Neuroscience of human social instincts: a sketch (2024)](https://www.lesswrong.com/posts/kYvbHCDeMTCTE9TAj/neuroscience-of-human-social-instincts-a-sketch), and [Social drives 1: “Sympathy Reward”, from compassion to dehumanization (2025)](https://www.lesswrong.com/posts/KuBiv9cCbZ6ALjHFw/social-drives-1-sympathy-reward-from-compassion-to), and [Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking (2025)](https://www.lesswrong.com/posts/fPxgFHfs5yHzYqgG7/social-drives-2-approval-reward-from-norm-enforcement-to), I claim to have a rather detailed step-by-step picture of how little cell groups in the Steering Subsystem ultimately lead to humans caring deeply about friendship, justice, and so on.

6.6.2 The Learning Subsystem and Steering Subsystem are not two agents
----------------------------------------------------------------------

Relatedly, another frequent error is treating either the Learning Subsystem or Steering Subsystem by itself as a kind of independent agent. This is wrong on both sides:

*   The Learning Subsystem cannot think any thoughts unless the Steering Subsystem has endorsed those thoughts as being worthy of being thunk.
*   Meanwhile, the Steering Subsystem does not understand the world, or itself. It has no explicit goals for the future. It’s just a relatively simple, hardcoded input-output machine.

As an example, the following is *entirely possible*:

1.  The Learning Subsystem generates the thought *“I am going to surgically alter my own Steering Subsystem”*.
2.  The Thought Assessors distill that thought down to the “scorecard”.
3.  The Steering Subsystem gets the scorecard and runs it through its hardcoded heuristics, and the result is: “Very good thought, go right ahead and do it!”

Why not, right? I’ll talk more about that example in later posts.

If you just read the above example, and you’re thinking to yourself “Ah! This is a case where the Learning Subsystem has outwitted the Steering Subsystem”, then *you’re still not getting it*. Maybe instead try imagining the Learning Subsystem & Steering Subsystem as two interconnected gears in a single machine.

Changelog
=========

*July 2024:* Since the initial version, I made a bunch of changes.

As in the previous post changelog, I’ve iterated on the “big picture” neuroscience diagrams several times since the initial version, as I learned more neuroscience.

For clarity, what I formerly labeled “Ground-truth value, a.k.a. reward” is now “Actual valence”, with a little note in the diagram about what this signal does. Relatedly, I added a bunch of links throughout this post to my more recent [Valence series](https://www.lesswrong.com/posts/As7bjEAbNpidKx6LR/valence-series-1-introduction), including a very brief teaser / recap of it in §6.4.1. And along the same lines, I reworded the RL discussions to use the word “reward” less and “valence” more. The word “reward” is not wrong, exactly, but it has a lot of baggage which I expect to confuse readers, as explained in the previous post and §6.4.1.

I deleted a claim that (what I am now calling) the “valence guess” signal “doesn’t have a special role in the algorithm”. I still think the thing I was trying to say is true, but the thing I was trying to say was pretty narrow and subtle, and on balance I expect that what I wrote is far more confusing than helpful. I actually think the valence guess is centrally important, especially in long-term planning, as discussed [here](https://www.lesswrong.com/posts/YQ4rSTHpHeFcAmhvi/incentive-learning-vs-dead-sea-salt-experiment#4_1_Visceral_predictions_generally_go_rapidly_to_zero_when_there_s_time__distraction__and_or_indirection_before_the_outcome__to_a_much_greater_extent_than_valence_predictions_).

I also put in the example of a “static spinning spherical octagon” to illustrate of how the “constraints” in the Thought Generator are not just sensory inputs but also our semantic knowledge about how the world works. I also mentioned [“involuntary attention”](https://www.lesswrong.com/posts/rM8DwFKZM4eB7i2p8/valence-series-3-valence-and-beliefs#3_3_5_An_exception__I_think_anxious___obsessive__brainstorming__is_driven_by_involuntary_attention_rather_than_by_valence) (e.g. itching or anxious rumination) as yet another source of “constraints”. 

I deleted some discussion of neuroanatomy (there used to be a §6.3.4), including cortico-basal ganglia thalamo-cortical loops and dopamine, because what I had written was mostly wrong, and unnecessary for the series anyway. What little I want to say on those topics mostly wound up in the previous post.

*October 2025:* Since first writing this, I have somewhat updated how I think about ego-syntonic versus ego-dystonic desires. So I changed some wording in §6.6.1.1, and added a link to my more recent discussion of that topic.

*January 2026:* More rewriting and additions to §6.6.1, plus minor copyedits throughout.

[^1bg9x09yd83h]: As usual, I’m using “cortex” in the neuroanatomical sense, which includes neocortex, hippocampus, etc.  

[^49k95mpuwpr]: The term “ground truth” here is a bit misleading, because sometimes the Steering Subsystem will just defer to the Thought Assessors, as explained in the previous post. 

[^ooqlv9o7fbr]: “Re-roll” was originally a board game term; for example, if you previously rolled the dice to randomly generate a character, then “re-roll for a new character” would mean “you roll the dice again to randomly generate a different character”. 

[^g0mylfbpkdp]: As in the previous post, I don’t really believe there is a pure dichotomy between “defer-to-predictor mode” and “override mode”. In reality, I’d bet that the Steering Subsystem can partly-but-not-entirely defer to the Thought Assessor, e.g. by taking a weighted average between the Thought Assessor and some other independent calculation.