Harley-ml/lesswrong huggingface dataset:
# Fake Updates 

### *Or: Lying To Yourself About Changing Your Mind*

  

Someone writes a hot take on Twitter, and you see red. “These morons don’t know what they’re talking about!” you think as you rain down keystrokes forming a reply that will sweep them away. Leaning back, you’re glad of a job well done. One more idiot on the internet slain. 5 minutes later, you see a response. Perhaps the idiot has written some feebleminded response, unaware that they’ve been skewered by the pointy end of your wit. “Now let’s see- oh. Dang. That’s a pretty good point. Hm.” You’ve just been destroyed on the internet.

What do you do in this scenario? One option is to cope, but that’s undignified for a genius like yourself, though you’d never dream of making intelligence a part of your identity. Leave that to Mensa. Another is to quibble over details, hiding your 99 feet of error behind their 1 foot of error. Perhaps length-wise? But no, that’s also undignified for a proud rationalist like yourself. Besides, each rebuttal that springs to mind is cast down as cope by your unruly unconscious.

If you’re like me, at this point you’ll be at risk fooling yourself into thinking you’ve updated.

Personally, I might concede that OK, the other person is right as I twist in my seat. But only grudgingly. Noticing my reluctance to update to the obviously correct point of view, I consider that I may not have given my true rejection yet. Focusing, I notice that I’ve got another objection. I listen to it and marshal the Objectively Correct arguments to bring that poor part of me into the light. Satisfied at the lack of any real response, I rejoice at being on the Right Side. Update successful.

What just happened was a Fake Update. Like a belief in belief, I can believe I updated. Yet tomorrow, I’ll go back to the position I had before the conversation. This is in spite of the fact that I was actually wrong.

Like belief in belief, Fake Updates occur when you have a non-epistemic incentive to update. Such incentives come in many guises.

Perhaps it’s because your actual rebuttal to the guy you’re arguing with sounds dumb, so you never raise it and you never actually get the chance to be shown you’re wrong. Perhaps it’s because your belief is a load-bearing part of your identity and risking it feels like you’re risking yourself. Perhaps it’s that you didn’t actually understand what someone else said but they’re too high status so you just nod your head and act like you’ve updated to please them[^q616fujwm0p], fooling even yourself.

Whatever the case, something is stopping you from propagating the info you’ve received into the rest of your beliefs. Like looking for a glass of water and seeing it’s on your desk, updates should be effortless. When they’re not, you can feel it. Learn the texture of Fake Updates. The sense that accompanies every “haha, yeah”, the tugging of your collar or twisting of your arms. Once you can feel them, you can do the dignified thing and say “I don’t believe this”

[^q616fujwm0p]: It won’t please them.
# Questions to ask when everyone is shooting themselves in the foot

*   *Why* is everyone shooting themselves in the foot? What's wrong with institutions/incentives that makes foot-shooting an attractor state? Are we in an inadequate foot-shooting equilibrium?
*   When, how, and why did we start shooting ourselves in the foot? Did it seem like a good idea at the time? Was it addressing a real problem? Did it start small and benign, like scratching an itch on the toe, and then ratchet upwards? Is there some actual still-existing problem that provides rationale or cover for foot-shooting?
*   Is there a place where people don't shoot themselves in the foot? What's different about it? Can we be more like them?
*   Have we ever *stopped* shooting ourselves in the feet before? How did that happen? Could we replicate it?
*   Does anyone justify or excuse shooting feet? On what grounds?
*   Is there a special interest group that benefits from foot-shooting? Have they captured institutions to make sure that feet keep getting shot? Could we build a coalition of the feet to provide an opposing political force against the shooting?
*   What anti-shooting / pro-foot reform could one propose, at the intersection of “small enough to happen” and “big enough to matter”?
*   How well-known is the foot-shooting? Does it just need someone to bang the drum until it becomes a meme and people decide to do something about it?

*(Originally written in response to someone who was demotivated because in his space “the solutions aren't even interesting, it's essentially begging to stop shooting yourself in the foot”)*
# What do we know about AI company employee giving?

Update (March 12): Transformer [published](https://www.transformernews.ai/p/anthropic-employees-philanthropy-billions-donations-effective-altruism-coefficient-giving-ai-safety) on this.  Their article (and comments here) note that there has already been some public discussions, which I hadn't seen.

Many Anthropic employees, especially, are sympathetic to AI safety and (will) have lots of money.  This is something that is being talked about a lot (semi-)privately, but I haven't seen any public discussion of it.    
  
I find that striking.  It seems like the topic is worthy of extensive public discussion, and it seems to me that perhaps this community is inheriting anti-helpful cultural norms against publicly discussing how individuals make use of their money.  
  
It also seems likely that many/most of AI company employees who are passionate about reducing AI risk should rapidly give much/most of their money to effective projects that would otherwise not be adequately funded.  
  
There's a lot of potential for this to do tremendous good.  There are of course things like political giving.  But I think most of this potential would come from employees having different theories of change than institutional funders, moving faster, and having higher risk appetite.  This is especially true given short timelines.  
  
A few specific thoughts:

*   I hear a lot of AI company employees give primarly to cause areas other than AI risk reduction.  It seems like donations to AI risk reduction would be much more valuable.
*   I'm concerned that many AI company employees may default to deferring to existing institutional funders who make decisions slowly and have biases.  It seems like giving faster and giving to projects that institutional funders are unwilling to support would make such donations much more valuable.
*   It seems like AI company employees typically wait for their equity to become liquid, but they could instead take out loans against that equity to accelerate their giving.  This could be very valuable given short timelines.
*   I know that AI companies have policies including matching donations for approved organizations.  It seems like influencing which organizations are elligible for matching could be very valuable, and like employees should not restrict their giving to already approved organizations.

To the extent things like the above are issues, it seems like coordination failures amongst company employees might be a large contributing factor.  Groups of AI company employees could address this by delegating relevant work to individual members who volunteer or are selected randomly.  
  
I'm fundraising for my nonprofit, Evitable, and might benefit from such things.  But my purpose in writing this is to promote public discussion that I think can benefit others in similar situations to me/Evitable.  
   
I haven't put much effort into fundraising for Evitable yet, and expect I will learn a lot more about the situation as I do.  
  
Much of the discussion here could equally well apply to individual HNWI giving more broadly.
# Cryonics Sign-Up Party

Come to sign up for cryonics!

More people have died while cryocrastinating than have actually been cryopreserved.

Signing up for cryonics will cost you £5-30/month in life insurance premiums (if you want to have it for the next 50 years) + ~$350/year in Alcor membership fees.

Yes, it’s that cheap. (This is a reaction so common that it made it into [**Yudkowsky’s post**](https://www.lesswrong.com/posts/yKXKcyoBzWtECzXrE/you-only-live-twice.))**.)**

Realistically, you will still procrastinate on signing up. So, you should come to the party: we’ll try to sign up everyone for cryonics in an evening.

Bring food! Especially frozen.

If you know people who might want to come to sign up for cryonics, please share a link with them: [https://partiful.com/e/S1zTdHG4Wjn4KYqihZky?c=PqmTBwNX](https://partiful.com/e/S1zTdHG4Wjn4KYqihZky?c=PqmTBwNX)
# How Many Parking Permits?

*EDIT: this comparison is much less clean than I thought it was: the Union Sq building has [19 garage spaces](https://www.maloneyaffordable.com/listings/prospect-union-square/#:~:text=Is%20parking%20available%3F,a%20fair%20and%20equitable%20process) reserved for affordable units. Combining these with the permits, it's 29% of parking-eligible units having a car instead of 8%.*

In [2017](https://www.jefftk.com/p/housing-without-street-parking) I wrote:

> One of the major reasons existing residents often oppose adding more housing is that as more people move in it gets harder to find on-street parking. What if we added a new category of unit that didn't come with any rights to street parking?

My city (Somerville MA) [included this](https://www.jefftk.com/p/housing-without-street-parking-implemented) in our 2019 zoning overhaul, but it does have some [exceptions](https://s3.amazonaws.com/somervillema-live/s3fs-public/parking-waiver-application-20250708.pdf):

> This policy exempts residents that may be 'choice limited', including:
> 
> *   Persons with disabilities
> *   Occupants of affordable dwelling units
> *   Residents with extenuating circumstances

While this is a compassionate approach, it means we haven't fully disconnected housing construction from parking demand. For example, there's a [proposal](https://docs.google.com/document/d/1Yhz9aVNekABVagz8ye0QWr3DNVukMcoIJ4o70GuoET0/edit?tab=t.0#heading=h.ugef1z46y0o3) to build a 500-unit parking-ineligible building in Davis Sq (which would [no longer](https://www.tuftsdaily.com/article/2026/01/copper-mill-development-submits-application-for-25-story-tower-in-davis-square) be [the end of the Burren](https://www.jefftk.com/p/tear-down-the-burren)). It's 25% affordable units, and opponents argue that if each has a driver this would be 125 additional cars competing for street parking. But would we really get that many?

A few years ago we got a similar parking-ineligible building in Union Sq, also a short walk from a subway station:

[![](https://www.jefftk.com/20-50-Prospect-Union.jpg)](https://www.jefftk.com/20-50-Prospect-Union-big.jpg)

This is 450 units, of which 20% (90) are [affordable](https://www.jefftk.com/p/misconceptions-on-affordable-housing). Ashish Shrestha submitted a records request to the city, and learned that [only seven units](https://groups.google.com/g/DavisSquareNC/c/NLeB6R5szM8/m/sNWjWCtGAAAJ) have parking permits.

While the Davis project is a little bigger, this would suggest something in the range of 10 permits, much less than feared.

This makes sense: if you're in Union or Davis, with good public transit and bike options, living without a car is pretty practical. It also saves you a lot of money, especially for folks living in affordable units.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid027wWomNgUSJnf4nKePeopC1ec9zRJbzBJkNQ9PUUpwxdnDVJRqCpvDAXHU1yGhjwvl), [lesswrong](https://lesswrong.com/posts/JqfsmGdqgRLD9bP39), [mastodon](https://mastodon.mit.edu/@jefftk/116213741542955923), [bluesky](https://bsky.app/profile/jefftk.com/post/3mgtcmcetns2w)*
# AI for Agent Foundations etc.?

What's the state of using current AIs for agent foundations research, or other theoretical AI safety work?

I'd be pretty surprised if no one has thought to do this, so I'm guessing this is just a matter of me catching up on what's going on.

I'm thinking for instance about how I saw Terence Tao talking about using Archimedes for advancing math. And I'd think that current AIs could explain their theoretical advances in ways that could convey the key insights to skilled humans. So they could maybe act as insight-searchers rather than just theorem-proving machines.

I get the sense that the current thinking is, this kind of work can't be fast enough at this point for many short timelines. That raising the alarms in public and politics is more key right now. And at first brush that basically looks right to me.

But I don't know, doing a straight AI-assisted shot at theoretical alignment work seems like it's probably worth trying, and easy enough for anyone to try, so I imagine someone is working on it. I just haven't yet heard of anyone doing this.

So, what's the current status of this kind of work?
# Sparks of RSI?

Are your long-running agents self-improving in loops with minimal prompting? Mine sure are!

I think we're seeing the first sparks of RSI here, folks. I'm expecting the frontier labs to scramble furiously to push this forward, finding and patching the meta-failure-modes. Thus, I expect next versions to be even better at this.

Here's what some other people are saying/claiming:

https://x.com/shreyasnsharma/status/2032567729560105117

https://x.com/varun_mathur/status/2032671842230501729

https://x.com/TuXinming/status/2032478765033701835

https://x.com/andrewwhite01/status/2031761577943425475

https://x.com/aramh/status/2029553870502756706

https://x.com/polynoamial/status/2029622090152956335

https://t.co/znsJlcww5r


And many more. This is just a few examples.
Not super impressive so far, but if this "task" goes the way many others have of first showing signs of progress in the 1-3% accuracy range, then rapidly shooting upwards over the next couple of model versions.... Yeah. 

Basically, I think we're in crunch time. Automated alignment time is here. Get cracking.
# 'Staying with it' Done Wrong

I was meditating today and noticed quite some over-effort happening. So I did the diligent, spiritually respectable thing: I located it in the body — "pain in my forehead" — and decided to stay with it. I even felt a small glow of pride for remembering to find it somatically instead of getting lost on the mental level.

What happened was quite disappointing: the headache locked into my attention, intensified steadily, and after a few minutes of escalating suffering I gave up and went to do something else.

I think this is a common way to misread the instruction *be with what is.* The mind notices something, names it — "pain," "tension," "sadness," "shame" — and then sincerely tries to stay with *that.* But now it is no longer staying with the living felt sense. It is staying with a conceptually frozen version: the brain's reconstruction of what this label is supposed to feel like.

Before a feeling can ease and release, it has to be allowed to *move* — to shapeshift freely. One moment it's an emotion, then it vanishes, then it reappears as pressure in the head, then as something else entirely. That, I think, is why Gendlin emphasized the intermediate handles on the felt sense that don't feel quite right yet: unless you're receiving those, you're not going to receive the truly relaxing shift. Without this intermediate aliveness, Focusing or meditative work with feelings tends to stall.

If, on the other hand, you notice it *is* stalling — the feeling has become fixed, static, stubbornly the same — that's worth treating as a signal. Something is interfering: a concerned part, a Buddhist hindrance, over-effort, aggressive awareness. Something worth investigating on its own terms.

  

  

**A little exercise from me**: Next time you do Focusing, IFS, or sit with a feeling in meditation — watch carefully what happens right after you name it. Does the naming make it more stuck, because now you have a more particular object to grip? Or was the word just a momentary touch that didn't disturb the feeling's natural unfolding?

Good luck with your practice!
# Bridge Thinking and Wall Thinking

There are a couple of frames I find useful when understanding why different people talk very differently about AI safety - the **wall**, and the **bridge**.  
  
A wall is incrementally useful. Every additional brick you add is good, and the more bricks you add the better. If you are adding a brick to the wall you are doing something good, regardless of the current state of the wall.  
  
A bridge requires a certain amount of investment. There's not much use for half a bridge. Once the bridge crosses the lake, it can be improved - but until you get a working bridge, you have nothing.  
  
A solid example of wall thinking is [the image in this thread by Chris Olah](https://threadreaderapp.com/thread/1666482929772666880.html). Any approach around “eating marginal probability” involves a wall frame. Another example is the theory of change of the standards work I've done for [Inspect Evals](https://github.com/UKGovernmentBEIS/inspect_evals), which I would summarise as “Other fields like aviation and rocketry have solid safety standards and paradigms. We need to build this for evaluations - it’s the kind of thing that a mature AI safety field needs to have.” This theory doesn’t have a full story of how it helps save the world end-to-end, but it doesn’t have to, under the wall frame - it just has to be pointing in the right direction and be broadly helpful.  
  
A good example of bridge thinking is the MIRI approach - asking openly and outright for an [international treaty](https://arxiv.org/abs/2511.10783). In my understanding, MIRI are not asking for the most they think they can get away with. They are asking for what they think is necessary to solve the problem, and believe anything less is insufficient. In lieu of p(doom), [Eliezer asks](https://www.lesswrong.com/posts/4mBaixwf4k8jk7fG4/yudkowsky-on-don-t-use-p-doom) “What is the minimum necessary and sufficient policy that you think would prevent extinction?” This is bridge thinking - we need to achieve a certain outcome X, and <X won’t do. Anything that has no chance of achieving X or greater is unhelpful at best and counterproductive at worst, And to figure out what X is needed, you need to have a solid idea of what your high level goal is from the start and how a given course of action gets you there.  
  
From the wall perspective, bridge thinkers are ignoring or denigrating important marginal or unglamorous work in favor of swinging for the fences. From the bridge perspective, wall thinkers are doing things that are not helpful and will end up rounding down to zero.  
  
I have found this a very useful way to understand why some people in AI safety are proposing very different ideas than me.
# Brown math department postdoctoral position

My friend (and former coauthor) [Eric Larson](https://www.math.brown.edu/elarso19/) is a professor at the Brown Math Department. The dept has (unexpectedly) opened a rapid-turnaround postdoc position on [Math + AI](https://www.mathjobs.org/jobs/list/28192). **The deadline is March 31** (in two weeks). The department has clarified that this includes any research involving Math and AI (not just AI for Math as the app text may suggest). The applicant must have a Math PhD. We're posting here in order to see if there are any safety-aligned academics who would be interested in applying for this position in this short timescale.
# The Psychopathy Spectrum

![image.png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1773783393/lexical_client_uploads/ugyfgouqhm27tdlwbfiz.png)

The term “psychopathy” is a mess, so I've written a sequence to tease apart all the different meanings along several dimensions.

[**Article 1: The Problem**](https://impartial-priorities.org/p/psychopathy-the-problem)**.** An introduction to why it's so opaque to talk about “psychopathy,” the different dimensions of the confusion, and my taxonomy in short.

[**Article 2: The Substrate**](https://impartial-priorities.org/p/psychopathy-the-substrate)**.** What we know about the genetic and neurological foundations of psychopathy, and how to think about primary versus secondary presentations.

[**Article 3: The Shaping**](https://impartial-priorities.org/p/psychopathy-the-shaping)**.** How different types of adversity lead to different outcomes, and why the same genetic loading can produce a functional person or a criminal depending on context.

[**Article 4: The Self**](https://impartial-priorities.org/p/psychopathy-the-self)**.** The different ways the psychopathic self can be organized, including the autonomy dimension and the relationship between psychopathy and narcissism (what I’ve called [sovereignism](https://impartial-priorities.org/p/narcissism-echoism-and-sovereignism)).

[**Article 5: The Mechanics**](https://impartial-priorities.org/p/psychopathy-the-mechanics)**.** An overview of the different ways empathy can break down, from perceptual failures to simulation failures to affective inversion. (This connects to my earlier work on [the sadism spectrum](https://impartial-priorities.org/p/the-sadism-spectrum-and-how-to-access).)

[**Article 6: The Types**](https://impartial-priorities.org/p/psychopathy-the-types)**.** Common profiles that tend to co-occur, with recognizable presentations that readers may identify with.

[**Article 7: The Choice**](https://impartial-priorities.org/p/psychopathy-the-choice)**.** An assessment of what recovery means for different presentations and the trade-offs involved.
# LessOnline ticket sales are live! (Earlybird pricing until April 7)

LessOnline is back in 2026, its third year running. As usual, it will take place at Lighthaven in Berkeley, CA. Tickets are live at [**less.online**](https://less.online?ref=lwap)

**When:** June 5-7, 2026  
**Where:** Lighthaven, Berkeley, CA  
**Who:** bloggers who are wrong on the internet, and people who like their writing  
**What else?** Manifest is the following weekend (June 12-14), with Summer Camp at Lighthaven running in between. Ticket bundles available.

**How much?** LessOnline tickets are currently 550USD and will increase to 675USD on April 7. All-Access tickets are 1250USD (going up to 1500USD). See the site for the rest of the options.

**I have further questions!** See the [FAQ](https://less.online/#faq?ref=lwap) on the main site.

![typetown_good_bloom_faded_outline.jpg](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1773710931/lexical_client_uploads/um5jxl2hatpku39d6m7v.jpg)

***What is LessOnline?***

A Festival of Writers Who are Wrong on the Internet
---------------------------------------------------

> LessOnline is a festival celebrating truth-seeking, optimization, and blogging. It's an opportunity to meet people you've only ever known by their LessWrong username or Substack handle... or hang out with them irl again if you met them at the first or second LessOnline!
# Principles of Intelligence is hiring

**New roles, core roles, big roles, small roles - we've got it all!**

Roles are remote except the RM position, with a preference for long-term employees to be in the Bay (US) or London (UK); deadline to apply for all is March 22nd!  

[Urgent Contract Hires: Research Engineer and Research Scientist](https://princint.ai/contract-urgent-hire-pibbsss-ambitious-mech-interp-amis-research-division/)

*   These roles will support the PrincInt Ambitious Mech Interp(AMI)’s research division as it sprints for a NeurIPS submission. This is a temporary role for research engineers and scientists to help us get papers on the following topics across the line:
*   *   Developing new unsupervised training methods used to reverse engineer current-day systems (i.e., SAEs, Weight-Based Decomposition Methods)
    *   Explore training-time uses of interpretability tools for better safety guarantees
    *   Design new neural network architectures that are interpretable by design
*   Candidates who excellently perform here will be also in consideration for the full-time hire

[Full-Time Hires: Research Engineer and Research Scientist](https://princint.ai/now-hiring-research-engineers-and-research-scientists-at-principles-of-intelligence/)

*   Similar to the above, and working on the same bet but more broadly, these roles are for those looking to permanently work for us rather than help us with just the NeurIPS submission.

[PIBBSS Fellowship Program Director](https://princint.ai/now-hiring-fellowship-program-director-pd-at-principles-of-intelligence/)

*   This is a big role that will join the core team at Principles of Intelligence. This person will lead the fellowship end-to-end: set strategy, run summer/winter selection cycles, execute the program, and supervise a global cohort of fellows and affiliates working on AI alignment research.

[Technical Research Manager (Iliad Fellowship 2026 & PIBBSS Fellowship 2026)](https://princint.ai/technical-research-manager-iliad-fellowship-2026-pibbss-fellowship-2026/)

*   In-person in London only!
*   We’re looking for a *Research Manager* to greatly own the planning, execution, and evaluation of the summer 2026 Iliad Fellowship and the [2026 PIBBSS Fellowship program](https://princint.ai/programs/fellowship/). You need RM experience, and understanding of ML/Math needed to supervise our Fellows.

**Opportunities with our Partners**

[Iliad Intensive + Iliad Fellowship](https://www.lesswrong.com/posts/b9bhm2iypgkCNppv4/announcement-iliad-intensive-iliad-fellowship)

*   The Iliad Intensive will cover taught coursework, serving as a widely comprehensive introduction to the field of technical AI alignment. The Iliad Fellowship will cover mentored research; it will support mentored research fellows for three months, giving them adequate time to generate substantial research outputs. Both programs will take place entirely in-person, in London.
# "Lost in the Middle" Replicates 

I was able to replicate some of the results from Liu et al's "[Lost in the Middle](https://arxiv.org/abs/2307.03172)" paper using a quantized Llama-2 7B model.

I used the multi-document question answering task from Liu et al. The model receives a question along with 10 retrieved documents (Wikipedia passages), exactly one of which contains the answer. The model must read all documents and produce the correct answer.  
  
I used the exact dataset published by Liu et al., based on Natural Questions (NQ-Open) - a collection of real Google search queries with short factual answers derived from Wikipedia. For the 10-document setting, the dataset provides 2,655 test examples at each of three gold document positions:  
  
Position 0 - the answer-containing document is placed first  
Position 4 - the answer-containing document is placed in the middle (5th of 10)  
Position 9 - the answer-containing document is placed last  
  
Each test example consists of the same question and the same 10 documents; only the position of the gold (answer-containing) document varies. The 9 distractor documents are real Wikipedia passages retrieved for the same query that do not contain the answer.  

![image.png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1773850046/lexical_client_uploads/w7kdogljbdzclh0sn1a0.png)![image.png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1773850103/lexical_client_uploads/yvspulpvlxit8fgkcesw.png)

  
  
The "U-Shaped" effect where the model did worse at answering questions based on information contained in Position 4 did replicate, though personally I'd describe it more as a Nike Swoosh. Note that the Liu et al. paper did a 20 document test for some of the models, which I did not bother to replicate as the primary purpose of this replication was just to establish a baseline for my own internal use.
# Metagaming matters for training, evaluation, and oversight

Following up on our [previous work on verbalized eval awareness](https://www.antischeming.ai/):

![image.png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1773867099/lexical_client_uploads/rq1iaggzapjr3bduu7rc.png)

we are sharing a post investigating the emergence of *metagaming* reasoning in a frontier training run.

1.  Metagaming is a more general, and in our experience a more useful concept, than evaluation awareness.
2.  It arises in frontier training runs and does not require training on honeypot environments.
3.  Verbalization of metagaming can go down over the course of training.

We also share some quantitative analyses, qualitative examples, and upcoming work.
# What should we think about shard theory in light of chain-of-thought agents?

LawrenceC proposed that the nine main theses of shard theory are as follows:

> *   Agents are well modeled as being made of shards---contextually activated decision influences.
> *   Shards generally care about concepts inside the agent's world model, as opposed to pure sensory experiences or maximizing reward.
> *   Active shards bid for plans in a way shaped by reinforcement learning.
> *   The optimization target is poorly modeled by the reward function.
> *   Agentic shards will seize power.
> *   Value formation is very path dependent and relatively architecture independent.
> *   We can reliably shape an agent's final values by changing the reward schedule.
> *   "Goal misgeneralization" is not a problem for AI alignment.
> *   Shard theory is a good model of human value formation.

Back in the day, discussion of this was quite abstract, almost by necessity, however now chain-of-thought provides us with much more (albeit imperfect) insight into how models reason.

It feels like the time is ripe to re-evaluate this theory. Does anyone have any takes how this pans out?
# Subscriber count graphs of popular youtubers on ASI risk

2026-03-19

Disclaimer
 - **Quick Note**
 - target audience - potential youtubers on ASI risk

Main
 - **It seems likely to me that these are among the most important graphs in human history, and whether our species lives or dies depends on these graphs.**
 - The most important thing to track is not the absolute number of subscribers as a function of time, but the slope in number of subcribers as a function of time. Youtube channels, like software apps, follow exponential growth curves. Most people are not used to thinking in terms of exponentials. Even a mediocre channel could grow if you make enough videos. A good channel will grow more quickly with lesser videos required.
 - I am tracking only youtube not twitter or substack, because the highest view counts are orders of magnitude higher. As of 2025-03, MrBeast has over 400 million subscribers. Top substack accounts have less than 10M subscribers. I am aware that all views are not equal, and views of powerful or technical people are worth more, for the AI pause political movement.
 - Statistics obtained from viewstats by MrBeast on 2026-03-19 (Viewstats is also good for tracking outlier videos)

Rational Animations

![Rational Animations](https://samuelshadrach.com/raw/non_text_non_video/viewstats_images/rationalanimations.jpeg "Rational Animations")

AI in context

![AI in context](https://samuelshadrach.com/raw/non_text_non_video/viewstats_images/ai_in_context.jpeg "AI in context")

The AI Risk Network - John Sherman

![The AI Risk Network](https://samuelshadrach.com/raw/non_text_non_video/viewstats_images/theairisknetwork.jpeg "The AI Risk Network")

AI Species - Drew Spartz

![AI Species](https://samuelshadrach.com/raw/non_text_non_video/viewstats_images/aispecies.jpeg "AI species")

Doom Debates - Liron Shapira

![Doom Debates](https://samuelshadrach.com/raw/non_text_non_video/viewstats_images/doomdebates.jpeg "Doom Debates")

Lethal intelligence - Michael Zafiris

could not obtain graph - 162K subs

Rob Miles

![Rob Miles](https://samuelshadrach.com/raw/non_text_non_video/viewstats_images/robmilesai.jpeg "Rob Miles")

Siliconversations - Liam

![Siliconversations](https://samuelshadrach.com/raw/non_text_non_video/viewstats_images/siliconversations.jpeg "Siliconversations")

The Inside View - Michael Trazzi

![The Inside View](https://samuelshadrach.com/raw/non_text_non_video/viewstats_images/theinsideview.jpeg "The Inside View")
# OpenAI: How we monitor internal coding agents for misalignment


Sharing some of the monitoring work I've been doing at OpenAI: How we monitor internal coding agents for misalignment.  
OpenAI now monitors 99.9% of internal coding traffic for signs of misalignment using our most powerful models. Today, that monitor is GPT-5.4 Thinking. It gets access to the full conversation context, that is everything the agent saw, and everything the agent did, including tool calls and CoT. Higher severity cases are sent for human review within 30 minutes.  
Some examples of misaligned behavior we’ve found:  
\- Encoding commands in base64 to circumvent monitors  
\- Calling other versions of themselves to bypass restrictions  
\- Attempting to upload files to the public internet  
\- Trying to prompt inject users.  
We haven’t yet detected any real world examples of sabotage, scheming or sandbagging.  
  
Our goal is to establish robust monitoring practices internally, strengthen them through iteration, and ultimately help make similar safeguards standard across the industry.
# "The AI Doc" is coming out March 26

On Thursday, March 26th, a major new AI documentary is coming out: ***The AI Doc: Or How I Became an Apocaloptimist***. Tickets are [**on sale now**](https://www.focusfeatures.com/the-ai-doc-or-how-i-became-an-apocaloptimist#watch).

The movie is excellent, and MIRI staff I've spoken with generally believe it belongs in the same tier as *If Anyone Builds It, Everyone Dies* as an extremely valuable way to alert policymakers and the general public about AI risk, especially if it smashes the box office.

When IABIED was coming out, the community did an incredible job of helping the book succeed; without all of your help, we might never have gotten on the *New York Times* bestseller list. MIRI staff think that the community could potentially play a similarly big role in helping *The AI Doc* succeed, and thereby help these ideas go mainstream.

(Note: Two MIRI staff were interviewed for the film, but we weren’t involved in its production. We just like it.)

The most valuable thing most people can do is maximize **opening-weekend success**. Buy [tickets](https://www.focusfeatures.com/the-ai-doc-or-how-i-became-an-apocaloptimist#watch) to see the movie now; poke friends and family members to do the same. This will cause more theaters to pick up the movie, ensure it stays in theaters for longer, and broadly increase the film’s exposure, the chance of international releases, etc.

You could also consider:

*   Hosting a viewing party and bringing a larger group to the theater to see the movie together.
*   Buying out a theater. If you’re interested in this, the producers have a [contact form](https://docs.google.com/forms/d/e/1FAIpQLSdFDziMgjNAfUGp7bXfkiD425FQnRH5Hazuoja97wNPOdbgdg/viewform) for this purpose. Also consider [contacting the MIRI team](mailto:contact@intelligence.org) if you’d like support from us for things like this, including potential financial support.
# China declares AGI development to be a part of 5-year plan

The CCP writes in its 15th 5-year plan that it will.

> Encourage innovation in multimodal, agentic, embodied, and swarm intelligence technologies, and explore development paths for general artificial intelligence.

This is translated from the original:

> 鼓励多模态、智能体、具身智能、群体智能等技术创新，探索通用人工智能发展路径。

Source: [https://www.spp.gov.cn/spp/tt/202603/t20260313_723954.shtml](https://www.spp.gov.cn/spp/tt/202603/t20260313_723954.shtml)

The English-language commentary I found does not have much more to say about this, e.g.: [https://triviumchina.com/2026/03/06/15th-five-year-plan-puts-ai-at-center-of-digital-economy-agenda/](https://triviumchina.com/2026/03/06/15th-five-year-plan-puts-ai-at-center-of-digital-economy-agenda/)

Given that they gave less than half a sentence in a 140-page document to the most important invention in the history of mankind, it seems likely the authors don't really understand what this means. Concerning nonetheless
# Contra Dances Should Avoid Saturdays

There are a lot of great musicians who don't live near you, and if you hold your dance on a Saturday it's much harder to put together a tour that brings them to you. Consider a Friday evening or Sunday afternoon, or even a weekly evening slot?

Looking at the 330 contra dances tracked by [TryContra](https://www.trycontra.com/), which I think is just about all of them, there's a very clear scheduling pattern:

[![Mon 12; Tue 7; Wed 15; Thr 10; Fri 71; Sat 187; Sun 28](https://www.jefftk.com/contra-dances-by-day-of-week.png)](https://www.jefftk.com/contra-dances-by-day-of-week-big.png)

There are more dances on Saturdays than the rest of the week put together. This makes sense: people are mostly off, and they're mostly off the next morning too. If you consider each dance in isolation, Saturday is often going to be the best choice.

The picture changes, however, when you consider [tours](https://www.jefftk.com/p/touring-as-a-contra-band). I live in Boston, and it doesn't make sense for me to drive 8hr round trip to NYC or Belfast ME to play a single evening. If I can make the weekend of it, though, and play Fri-Sat-Sun, the ratio of driving to playing gets a lot better. Similarly, a 12hr round trip to Philly or 16hr round trip to DC don't work on their own, but they're possible as part of being able to play Wed-Thr-Fri-Sat-Sun in Princeton, Philly, DC, Bethlehem/Chatham/Lancaster, and NYC.

If you're thinking about starting a new series, consider that picking a different day can help you convince bands and callers to come visit. I think Sunday afternoons in particular are underrated: in addition to helping attract touring bands there are a lot of people who have to get up early, more time to drive home means it's possible for people to attend from a larger radius, and there's tons of time left for afters.

It can also be worth explicitly coordinating schedules with dances that are 1-3hr away, and offering a group of dates to a band. Scheduling tours is a pain, but if a group of dances that are normally too far away offered a Fri-Sat-Sun I think many more musicians and callers would consider it.

I wouldn't want to move to a world without Saturday night dances, or one where dances tended not to have any local talent, but I think we're pretty far from this world. Consider prioritizing the cross-pollination benefits of bringing callers and musicians from a bit further off?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02uACgnCskfdBQ21UKYyra3qRxQ7Q1f8hTjqPpddZEVZJg2Zkm68c4PckFVjT8Kw1pl), [lesswrong](https://lesswrong.com/posts/XDGdHhb9xM4gHxPGY), [mastodon](https://mastodon.mit.edu/@jefftk/116281809669232752), [bluesky](https://bsky.app/profile/jefftk.com/post/3mhrjx7lwf22j)*
# Coming of Age: Chapters 1 and 2

I am starting to release my fiction book. I am telling here, because it is a kind of rational fiction. Or, more precisely, I would frame the genre as a combination of:

*   Rational fiction as it is commonly defined.
*   Depiction of rationalist culture as a cultural phenomenon.
*   Optimistic proud-human-spirit golden-age-style sci-fi, but very hard.
*   Depiction of Eastern European (or, to be more precise, Ukrainian) science olympiad culture.

Not a utopia, but what I call an *antiantiutopia*: a world from whose point of view our world looks like an antiutopia.

So you won't find there a lot of mind games and bayesian thinking embedded in the plot, but you will find many rationalist memes and references to existing culture. And I hope that, for a particular kind of people, this imagined world would be a pleasant place to be in.

The setting:

*   Our time, but a parallel timeline that diverged from ours in the mid-20th century. This is a timeline where history has gone well. They have their equivalent of "[What happened in 1971?](https://wtfhappenedin1971.com)" but in the positive direction. Different pro-progress ideologies have basically won there (rationalism, transhumanism, general pro-science stance).
*   It starts with slow pace, not much action, a lot of worldbuilding and multiverse-nostalgia about a better world; philosophy and culture. Action will come, but like after 5 chapters.
*   Central topic: seriousness, civilizational adulthood, and cosmic responsibility.

Note on AI usage:

Around 60% of the book was written by the following process: I dictate the text to AI, then AI removes all the interjections and crooked phrases and fixes broken sentences when there are some, and then I polish the final text.

Another 40% are just written by me directly.

I will publish on Royal Road 1-2 chapters each week. I will also make weekly posts with the links here on LessWrong, unless I see there is no interest from the community.

As for now, 2 chapters are published:

1.  Chapter 1: [https://www.royalroad.com/fiction/158269/coming-of-age/chapter/3170876/an-improbable-morning](https://www.royalroad.com/fiction/158269/coming-of-age/chapter/3170876/an-improbable-morning)
2.  Chapter 2: [https://www.royalroad.com/fiction/158269/coming-of-age/chapter/3186842/san-andreas](https://www.royalroad.com/fiction/158269/coming-of-age/chapter/3186842/san-andreas)
# An Informal Definition of Goals for Embedded Agents

*This post was written as part of research done at MATS 9.0 under the mentorship of Richard Ngo.*

You can conceptualise [Embedded agents](https://arxiv.org/abs/1902.09469) as inducing a partition[^a53mmytkal] of the world into "the agent", "the external world", and the dynamics that mediate their interaction; the dynamics include observations and actions. The agent has beliefs, which can be thought of as a generative model of the world. This contains a generative [self-model](https://www.lesswrong.com/s/qhdHbCJ3PYesL9dde) of "the agent" and its relationship to the world through intermediate dynamics[^ovwygdg1u9i].

An agent's self-model contains probable events that are statistically dependent on her actions. They are likely, but only if she acts to make them happen. These events are her goals.

  

[^a53mmytkal]: See Demski 2025 and Critch 2022 for mathematical treatments of partitions 

[^ovwygdg1u9i]: This model may or may not be interpretable.
# $1 billion is not enough; OpenAI Foundation must start spending tens of billions each year

OpenAI is now a public benefit corporation, with a charter that demands they use AGI for the benefit of all, and do so safely. To justify this structure to the Attorneys General of Delaware and California, they split off the nonprofit OpenAI Foundation, and instead of full ownership, gave it 27% equity, worth well over $150 billion - what some have called [the largest theft in human history](https://thezvi.substack.com/p/openai-moves-to-complete-potentially). They said at the time they would donate a small fraction of the value, $25 billion, eventually.

This week, [they announced a team, and a commitment](https://openaifoundation.org/news/update-on-the-openai-foundation); giving "at least $1 billion" over the coming year; [in the full post](https://forum.effectivealtruism.org/posts/ME7yYXondbFpTCrri/usd1-billion-is-not-enough-openai-foundation-must-start), I argue that given their values and expectations, everyone should agree that this is far too little, moving far too slowly.
# Uncertain Updates: March 2026

[The book](https://fundamentaluncertainty.com/) is almost done!

I finished the second editing, and I’m now into copy editing. That’s also almost done, with just the last two chapters to go. Which means that, sometime in the next month, the book will finally, after a bit over 4 years, be complete!

I’ve decided to go the independent publishing route. The book market has changed a lot, even just since I started writing, and my book is at high risk of being too niche to support a run with a big, commercial publisher. Thankfully, in that same time, independent publishing has lost a lot of its stink as the book market has transformed away from mass distribution, so it’s no longer a low-status move to go independent. Thus, I’ll be retaining the copyright, publishing the book free online for anyone to read, and also have print and audiobook versions for those who would like that available for sale with most major book retailers.

Very excited to get the final version of the book into everyone’s hands, and then to get onto my next big project. Expect a full announcement on it soon, but to tease you now, it’s a conference, it’s about epistemics, and vibes, and it’ll be happening a little later this year. More details soon!
# Label By Usable Volume

I always look at unit prices: how much do I get for my dollar? But that assumes I can use all of it. The manufacturer puts "12oz" whether I'll be able to get the full 12oz or only 6oz. L'Oreal was selling lotions where:

> these Liquid Cosmetic Product containers only dispense between as little as 43 percent to 81 percent of the container's advertised contents. — [Critcher et al. v. L'Oreal](https://www.classaction.org/media/crticher-et-al-v-loreal-usa-in.pdf)

Even though these containers would often dispense less than half of the advertised volume, L'Oreal [won the case](https://law.justia.com/cases/federal/appellate-courts/ca2/19-2474/19-2474-2020-05-11.html): the law says the amount listed on the container means the amount in the container, not the amount you'll be able to get out of the container. But it doesn't have to be that way. What should our laws say?

We should update our labeling laws to require manufacturers to use the amount a consumer could reasonably extract. If you have a wide mouth transparent container with smooth insides, a rubber scraper can get it all. If you have a narrow mouth squeeze bottle, then only count what squeezes out. Maybe manufacturers would shift to more efficient packaging, or maybe consumers would accept higher unit cost for more convenience. The important thing is aligning incentives: pay for what you can use.

There is actually one area where we do this already: medicine. Because it seriously matters that when the doctor prescribes 10ml you receive 10ml, they are required to measure losses and adjust for them. If we could only do this in one part of the economy I agree medication is good choice, but why don't we do this everywhere?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid025TapVj6Ro9gB4cgx8ECWALgD4sSYZvBrLjfNptS5NF5TjphETWmqkrTtWDbGyfHul), [mastodon](https://mastodon.mit.edu/@jefftk/116293107177846897), [bluesky](https://bsky.app/profile/jefftk.com/post/3mhwkidytxs2t)*