Harley-ml/lesswrong huggingface dataset:
# Status In A Tribe Of One

I saw a tweet thread the other day, in which a self-proclaimed autistic guy was freaking out about how much "normies" care about "status". I won't quote the thing because I'm going to mildly insult the guy: the whole time I read it I was thinking *for a guy hates status-talk, you sure are full of yourself*.

So, technically, he doesn't care about status in the way that the average person does. He doesn't seem to care about what others think of him, except insofar as it helps him when they think certain ways. Yet that thread really seemed to be drawing on the exact same processes that people use to regulate and signal status.

This seems odd, what's going on? I think we can explain it with a multi-part mechanism for social status.

There are several components needed for a social ledger to exist. Everyone keeps track of their own ledger, they signal what's on their ledger to each other, and then everyone updates. This guy has the first part, just not the second part. He has something status-ish going on, but he can't signal. I think of this as having a personal narrative, a story that he tells himself about who he is.

Status is just a special case of a shared narrative. For a narrative to roughly converge in a group of people, everyone has to be sending out signals which inform the rest of the group as to what their version of the narrative is. This means there might be disagreement. If Alex's ledger says "Alex is the leader" and Bach's ledger says "Bach is the leader" then the two are in conflict. If Caleb, Dina, and Ethel all think "Bach is the leader" then, if the system works correctly, Alex will update.

This might help answer another question: why don't people signal high status all the time? My guess is that the ledger-keeping circuits are approximately rational, on some deep level. Someone once pointed out that vector calculus is a rare skill, but learning to throw a ball is not. The deeper wires of our brains are, in fact, pretty good learners. To trick a rational circuit is a non-trivial task.

Narcissism and the other cluster-B type disorders might be thought of as a certain type of failure to update one's own narrative based on evidence.[^2jyvj7e3scj] The ingoing pipeline is broken. This leaves the cluster-B brain free to settle on any narrative it wishes, and importantly (since the outgoing pipeline is intact) free to send signals of its chosen narrative to other brains.

Autism-ish stuff seems to harm both the in- and the outgoing pipelines. This individual can't signal properly or read signals properly! They might end up with a self-serving narrative or a self-deprecating one, but it is cut off from the social fabric: they exist in a tribe of one. 

[^2jyvj7e3scj]: This is probably not completely right, I think the broken-ness might also involve the self-narrative system trampling on other parts of the brain.
# I Made a Judgment Calibration Game for Beginners (Calibrate)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/92153af10e84469ab5e2f14b625b7a250bcca26428e46378.png)

I made a game that teaches beginner [calibration](https://www.lesswrong.com/w/calibration). Have you ever wanted your brother/girlfriend/aunt/dog to make more calibrated probabilistic claims? Now you can send them [Calibrate](https://luisew.itch.io/calibrate)! (Or you can play it yourself.)

   [**Play now**](https://luisew.itch.io/calibrate) (desktop only)

**Semi-big catch:** Calibrate was mostly made for computer science students, so non-computer science students may have a hard time getting very high scores. People from other hard sciences have also enjoyed it, though!

### **More on Calibrate**

The premise of the game is to get rich by investing in a calibrated way. It's more fun than existing calibration training tools in my opinion, because there's more sense of progress from earning money and moving through the game. It's not very long, roughly 30 minutes total.

I also did a very low-powered analysis of calibration outcomes with 11 play testers. The results look promising; see the calibration curve on the left.  See the plot on the right for how to interpret the curve. On a cursory look, outcomes are looking similar to [those of other calibration trainings](https://onlinelibrary.wiley.com/doi/full/10.1002/ffo2.177), which are usually much longer. (*Confidence interval* calibration didn't work super well compared to *multiple-choice* confidence calibration; see the unhappy green X on the left.)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f87b36966d7c9eaf7ab1c4c29a4aa04d15eaf739b7116297.png)

[**Play Calibrate now**](https://luisew.itch.io/calibrate)

Other excellent calibration tools (not computer games):

[Calibrate your Judgment by Clearer Thinking](https://www.clearerthinking.org/tools/calibrate-your-judgment )

[Estimation Game by Sage](https://www.quantifiedintuitions.org/estimation-game)
# Is It Reasoning or Just a Fixed Bias?

This is my first mechanistic interpretability blog post! I decided to research whether models are actually reasoning when answering non-deductive questions, or whether they're doing something simpler.

My dataset is adapted from InAbHyD[^mhlormxp4t], and it's composed of inductive and abductive reasoning scenarios in first-order ontologies generated through code (using made-up concepts to dismiss much of the external effect of common words). These scenarios have multiple technically correct answers, but one answer is definitively the most correct[^sfxwe0uv08k]. I found that LLMs seem to have a fixed generalization tendency (when evaluating my examples) that doesn't seem to adapt to any logical structure. And accuracies in 1-hop and 2-hop reasoning add up to roughly 100% for most models.

Additionally, there's a large overlap between H2 successes and H1 failures (73% for DeepSeek V3), meaning that model outputs the parent concept regardless of whether the task asks for the child concept or the parent concept. This behavior suggests that the model isn't actually reasoning, instead generalizing to a fixed level that aligns with the parent concept.

This is perplexing because in proper reasoning, you generally need the child concept in order to get the parent concept. For example, you'd conclude that Fae is a mammal through a reasoning chain, first establishing that Fae is a tiger, then making one hop to say that Fae is a feline (child concept), and then making a second hop to say that felines are mammals (parent concept). The overlap, though, suggests that the model isn't reasoning through the ontology, and it's skipping the chain to output the parent or child concept depending on its fixed generalization tendency.

I used MI techniques like probing, activation patching, and SAEs in my research. Probing predicted *something* related to the output at layer 8 (very early), but patching early layers barely made a difference in the final decision. This makes it more likely that whatever probing predicted is just correlational to the final result, as well as that the generalization tendency is distributed among model components instead of being early on.

This was a fun project, and I'm excited to continue with this research and make some more definitive findings. My ultimate goal is to find *why* LLMs might be architecturally limited in non-deductive reasoning.

[^mhlormxp4t]: A paper that I based my initial research question on, which argues that LLMs can't properly do non-deductive reasoning. It's authored by my NLP professor, Abulhair Saparov, and his PhD student, Yunxin Sun. 

[^sfxwe0uv08k]: This concept is known as Occam's razor.
# Forfeiting Ill-Gotten Gains

It's a holiday. The cousins are over, and the kids are having a great time. Unfortunately, that includes rampaging through the kitchen. We're trying to cook, so there's a "no cutting through the kitchen" rule. Imagine enforcement looks like:

> Kid: \[dashes into kitchen, pursued by cousin\]  
> Adult: Out of the kitchen!  
> Kid: Sorry! \[Continues their path, leaving through the other door; escapes pursuit from more rule-abiding cousin\]

This doesn't work! The kid got what they wanted out of this interaction, and isn't going to change their behavior. Instead, I need to make it be not worth their while:

> Kid: \[dashes into kitchen, pursued by cousin\]  
> Adult: No cutting through the kitchen! \[Physically rebuffs intruder\]!  
> Kid: Sorry! \[Forced to leave through the door they entered by; caught by cousin.\]

Other examples:

*   Sneak candy, spit it out and forfeit dessert.
    
*   Use sibling's tablet time, lose your own.
    
*   Interrupt, be ignored.
    

The general principle is that if you want to limit behavior the combination of the gains from rule-breaking and penalty from punishment need to put the kid in a worse position than if they'd never broken the rule.

This isn't just a parenting thing: it's common to say that "crime should not pay", and many legal systems prohibit [unjust enrichment](https://en.wikipedia.org/wiki/Restitution_and_unjust_enrichment). One place I'd like to see this implemented is airplane evacuation. If the safety announcements included "In the event of an emergency evacuation, any carry-on luggage you bring will be confiscated and destroyed. You will also be fined." we would have more [JAL 516](https://en.wikipedia.org/wiki/2024_Haneda_Airport_runway_collision) (379 occupants, zero deaths) and less [Aeroflot 1492](https://en.wikipedia.org/wiki/Aeroflot_Flight_1492#Evacuation_with_luggage) or [Emirates 521](https://en.wikipedia.org/wiki/Emirates_Flight_521#Flight).

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02FCpLja2qUbF6H2pqZ2LjjJfk55Jih2GYPLj4ifsGREeHMEJB8YoWHf6SQmEguz7al), [mastodon](https://mastodon.mit.edu/@jefftk/115907560366396377), [bluesky](https://bsky.app/profile/jefftk.com/post/3mcldjvvles2j)*
# Japan is a bank

Among developed countries, Japan [has long had](https://en.wikipedia.org/wiki/List_of_countries_by_government_debt) the highest debt/GDP ratio, currently ~232%. That seems pretty bad, and conversely has made some people say that the US debt is fine because it's still much lower than Japan's. But here are some points that might clarify the situation:

First, that ratio [has declined recently](https://tradingeconomics.com/japan/government-debt-to-gdp), from 258% in 2020.

Second, the Japanese government [holds a lot of stocks and foreign bonds](https://www.ssga.com/dk/en_gb/institutional/insights/separating-fact-from-fear-in-japanese-investing). Its net debt/GDP is "only" 140%, and has declined since 2020. The US government doesn't do that. (The government of Singapore also holds a lot of assets, and [Temasek](https://en.wikipedia.org/wiki/Temasek_Holdings) is well-known as a large investment fund, but Japan is a bigger country, and despite smaller holdings per capita, its investments are much larger than Singapore's.)

Meanwhile, America's [federal debt/GDP ratio](https://www.usdebtclock.org/) is ~124%. Add in state debt and it's ~127%. So the net debt/GDP of the US government isn't that different from Japan. It's still higher, but arguably the "quality" of that US GDP is lower, for a couple reasons:

1. The US has a worse trade balance than Japan. It borrows more money, and has a net inflow of investment. That investment and borrowed money then circulates around and raises GDP by some multiplier, mostly by making both prices and incomes higher in the US.
2. Japan has higher PPP/nominal GDP than the US, by a factor of ~1.6x. Arguably this is a better measure of ability to pay back debt with real stuff than nominal GDP.

On the other hand, the US does have more natural resources, and the federal goverment owns a lot of land. My point is just that, while I've often seen it said that the US government debt situation is clearly better than Japan's, that's not clearly the case.

By the way, another economic metric I think is interesting to compare is [median and average personal wealth](https://en.wikipedia.org/wiki/List_of_countries_by_wealth_per_adult).
# The first type of transformative AI?

AI risk discussion often seems to assume that the AI we most want to prepare for will emerge in a“normal” world — one that hasn’t really been transformed by earlier AI systems. 

I think betting on this assumption could be a big mistake. If it turns out to be wrong, most of our preparation for advanced AI could end up ~worthless, or at least far less effective than it could have been. We might find ourselves wishing that we’d laid the groundwork for leveraging enormous political will, prepared for government inadequacy, figured out how to run large-scale automated research projects, and so on. Moreover, if earlier systems *do* change the background situation, influencing how that plays out could be one of our best levers for dealing with AI challenges overall, as it could leave us in a much better situation for the “boss-battle” AI (an opportunity we’ll squander if we focus exclusively on the endgame).

So I really want us to invest in improving our views on *which AI changes we should expect to see first.* Progress here seems at least as important as improving our AI timelines (which get significantly more attention). And while it’s obviously not an easy question to answer with any degree of confidence — our predictions will probably be off in various ways — defaulting to expecting “present reality + AGI” seems (a lot) worse. 

* * *

I recently published [an article](https://www.forethought.org/research/the-first-type-of-transformative-ai) on this with [@Owen Cotton-Barratt](https://forum.effectivealtruism.org/users/owen_cotton-barratt?mention=user) and [@Oliver Sourbut](https://forum.effectivealtruism.org/users/oliver-sourbut?mention=user). It expands on these arguments, and also includes a list of possible early AI impacts, a few illustrative trajectories with examples of how our priorities might change depending on which one we expect, some notes on whether betting on an “AGI-first” path might be reasonable, and so on. (We’ll also have follow-up material up soon.)

I’d be very interested in pushback on anything that seems wrong; the perspective here informs a lot of my thinking about AI/x-risk, and might explain some of my disagreements with other work in the space. And you’d have my eternal gratitude if you shared your thoughts on the question itself.

![Sketch illustrating this perspective; total AI-driven change vs time](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/4af6471481532a78d30c52d4515ee03f90ab312d3a3c8d7e.png)

A lot of discussion assumes that we won't really move up the Y axis here until after some critical junctures.
# Releasing TakeOverBench.com: a benchmark, for AI takeover

Today, [PauseAI](https://pauseai.info/) and the [Existential Risk Observatory](https://www.existentialriskobservatory.org/) release [TakeOverBench.com](http://takeoverbench.com): a benchmark, but for AI takeover.

There are many AI benchmarks, but this is the one that really matters: how far are we from a takeover, possibly leading to human extinction?

In 2023, the broadly coauthored paper [Model evaluation for extreme risks](https://arxiv.org/abs/2305.15324) defined the following nine *dangerous capabilities*: Cyber-offense, Deception, Persuasion & manipulation, Political strategy, Weapons acquisition, Long-horizon planning, AI development, Situational awareness, and Self-proliferation. We think progress in all of these domains is worrying, and it is even more worrying that some of these domains add up to AI *takeover scenarios* (existential threat models).

Using SOTA benchmark data, to the degree it is available, we track how far we have progressed on our trajectory towards the end of human control. We highlight four takeover scenarios, and track the dangerous capabilities needed for them to become a reality.

Our website aims to be a valuable source of information for researchers, policymakers, and the public. At the same time, we want to highlight gaps in the current research:

*   For many leading benchmarks, we just don't know how the latest models score. Replibench, for example, hasn't been run for almost a whole year. We need more efforts to run existing benchmarks against newer models!
*   AI existential threat models have received only limited serious academic attention, which we think is a very poor state of affairs (the Existential Risk Observatory, together with MIT FutureTech and FLI, is currently trying to mitigate this situation with a new threat model research project).
*   Even if we had accurate threat models, we currently don’t know exactly where the capability red lines (or red regions, given uncertainty) are. Even if we had accurate red lines/regions, we don’t always reliably know how to measure them with benchmarks.

Despite all these uncertainties, we think it is constructive to center the discussion on the concept of an AI takeover, and to present the knowledge that we do have on this website.

We hope that [TakeOverBench.com](http://takeoverbench.com) contributes to:

*   Raising awareness.
*   Grounding takeover scenarios in objective data.
*   Providing accessible information for researchers, policymakers, and the public.
*   Highlighting gaps in research on takeover scenarios, red lines, and benchmarks.

[TakeOverBench.com](http://takeoverbench.com) is an open source project, and we invite everyone to comment and contribute on [GitHub](https://github.com/joepio/takeoverbench).

Enjoy TakeOverBench!
# A quick, elegant derivation of Bayes' Theorem

I'm glad I know this, and maybe some people here don't, so here goes. 
$$P(A \text{ and } B) = P(A) \cdot P(B \mid A)$$
$$P(B \text{ and } A) = P(B) \cdot P(A \mid B)$$
Order doesn't matter for joint events: "A and B" refers to the same event as "B and A". Set them equal:
$$P(B) \cdot P(A \mid B) = P(A) \cdot P(B \mid A)$$
Divide by $P(B)$:
$$P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)}$$
And you're done! I like substituting hypothesis (H) and evidence (E) to remember how this relates to real life:
$$P(H \mid E) = \frac{P(E \mid H) \cdot P(H)}{P(E)}$$

You might also want to expand the denominator using the law of total probability, since you're more likely to know how probable the evidence is given different hypotheses than in general:
$$P(H_i \mid E) = \frac{P(E \mid H_i) \cdot P(H_i)}{\sum_j P(E \mid H_j) \cdot P(H_j)}$$
# AI X-Risk Bottleneck = Advocacy?

Introduction
============

I am leading an early-stage effort to target AI x-risk. We're currently analyzing the bottlenecks in the AI x-risk prevention "supply chain" to decide where to focus our efforts. We would love to get comments from the community.

The x-risk community has a strong focus on technical/policy research, but perhaps not enough advocacy. AI 2027, Rob Miles, CAIS, CivAI, and others are doing well, but these efforts could be small compared to the rapidly growing power and influence of AI developers, who have misaligned incentives that could lead to x-risk.

What's Missing?
===============

We are testing the hypothesis that operating a viral influencer marketing operation would be beneficial in targeting x-risk. Here's the logic:

*   We build a media hub with simple, factual x-risk resources and assets
*   We identify creators with relevant audiences and a track record of creating viral content.
*   We pay them to create their own versions of x-risk awareness content based on our media kit (also known as UGC - User Generated Content)
*   They push the content via their channels, and we amplify it with paid ads for max reach
*   The content might be re-shared or even pop up on traditional media once it gains enough traction.
*   This builds broad awareness of x-risk among the voters' base, creating an opportunity for politicians to score wins with voters and gain political power by promoting x-risk solutions.

Since this is similar to a political campaign, we can hire people or firms with such experience to manage the project.

How can the community help?
===========================

We are looking for answers to the following questions:

1.  According to the Theory of Constraints, a system is limited to one constraint at any given time. Is advocacy the current bottleneck in x-risk prevention? If not, what is?
2.  If advocacy isn't the bottleneck, would you still want new resources invested in it, or would you prefer them invested elsewhere?
3.  Is a viral influencer campaign (similar to a political campaign) the right solution for the advocacy problem? If not, what is?

Related Posts
=============

\[..\] we’ll need to shift significant resources from research (which helps us understand problems better) to advocacy (which helps us change bad incentives). \[[link](https://www.lesswrong.com/posts/dcd2dPLZGFJPgtDzq/shift-resources-to-advocacy-now-post-4-of-6-on-ai-governance)\]

“\[..\] I estimated that we have 3 researchers for every advocate working on US AI governance, and I argued that this ratio is backwards.”  
“Without political power, we can’t change the bad incentives of AI developers that are very likely to lead to the collapse of human civilization.”  
“Thus, I urge AI safety grantmakers to aggressively recruit as many political advocacy experts as possible.” \[[link](https://www.lesswrong.com/posts/adQueu9FFHfiBKDCt/political-funding-expertise-post-6-of-7-on-ai-governance)\]
# How accurate a model of the refrigeration cycle is this doodle?

[This Technology Connections video on heat pumps](https://www.youtube.com/watch?v=7J52mDjZzto) made me realize I don't intuitively understand how refrigeration works. I tried to drill down until I understood what was happening with every molecule, and... arrived here. Would any local thermodynamics experts enjoy pointing out the important gaps?

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/af8cd5cf0abefe76d969e0394b25d308e9f856b0ad433a11.png)
# Rationalist Hamlet Reading

What: Dramatic reading  
Where: 70 5th St, Apt 2, Cambridge  
When: Saturday February 21st, 2:00pm.  
Style: Social  
Run By: Skyler and Evan

*"I seek to deny Hell to everyone!*  
*and Heaven too, for I suspect the Heaven of our mad God*  
*might be a paltry thing, next to the Heaven I will make of Earth*  
*when I am its immortal king."*  
  
Many of you may have heard my spiel on how there is a rationalist fanfic of Hamlet, it's great, and I want to do a play reading. Guess what we're doing today?  
  
"*The Tragedy of Prince Hamlet and the Philosopher's Stone, Or, A Will Most Incorrect To Heaven*" is good fun for folks who like Shakespeare and/or rationalist fanfiction. I'll have a few printed copies of the play, and we can share or you can grab your own copy for four bucks on amazon. We will pick characters from the Dramatis Personae, and then sit down and read our character's lines when they come up. I used to do this when I was in theatre more often and it's a good time. I don't actually know how long the play takes to read aloud, but I think we can do it in two hours and if not we'll make a group decision at 4:30 whether to keep going.  
  
Another note: while I try to announce the local meetups in lots of places, the two that I'm most reliable at are the [discord](https://discord.gg/2N2ADpACkw) and the mailing list.
# Thomas Schelling Appreciation Day

**(IMPORTANT NOTE:** for obvious reasons, the *meetup* for Thomas Schelling Appreciation Day this year is on the Saturday *after* the obvious date.)

In 2005, game theorist Thomas Schelling won the Nobel Prize in Economic Sciences for finding a way you can meet up with your friends in New York City even if your phone is dead. We can no longer thank him directly for this discovery, but on this day, let us still apply it; gather with our friends; and celebrate him.

$$This year, there will be a meetup on the Saturday after the obvious date, at the obvious time, in the obvious place in your city. If you go to the obvious date+time+place, and there is at least one other person there for the same reason, you have met up, so you have found the meetup! (A city might, of course, have multiple meetups.)

I will attend ~~the~~ *a* Seattle TSAD meetup. Things worth preparing for if you successfully attend mine:

*   **Sing-along:** Upon the obvious signal, we will sing the obvious song. You know the one.
*   **Bring a thing:** bring a small object that fits in a pocket or purse. I'll have a prize for the largest group of people who bring ~identical things (in my judgement).
*   **BMOB:** If you bring the same book as me, I'll reimburse you for it (up to what I paid for my copy).
*   I'll have a few other game-theoretically-interesting activities too, such as a game co-authored by, among others, Nash and Shapley.

And remember, each person can only attend their first Thomas Schelling Appreciation Day once. Try not to spoil it for them.

Godspeed.

(P.S.: there are compelling reasons for local organizers to nail down at least a specific *date*. If you want to play this on hard mode, you should think about what date TSAD obviously is before you see any e.g. Meetup.com events.)
# Uncertain Updates: January 2026

It’s been a busy month of writing.

Chapter 7, as you may recall if you read the [first draft](https://www.lesswrong.com/posts/Q4zBhYobwkGBGuh7v/fundamental-uncertainty-chapter-7-why-is-truth-useful), is both the “cybernetics chapter” and the “tie everything together” chapter. Originally it was largely based on the [two](https://www.lesswrong.com/posts/M7Z5sm6KoukNpF3SD/form-and-feedback-in-phenomenology) [posts](https://www.lesswrong.com/posts/agvmvrzM6um462DC2/the-purpose-of-purpose) where I first worked out these ideas, but as I’ve been revising, I discovered that it contained both a lot of extraneous material and didn’t have quite the right focus for where it sits in the book. These were both problems I knew about when I wrote the first draft, and now in the revisions I have to solve them.

As a result, it’s been a slog to find the right way to present these ideas. I’ve tried maybe 5 different approaches. It takes time to develop them out enough to see if they work. I’m hopeful that the 6th approach will be the final one, but it’s not done yet, so no promises.

Medium
------

Hey, did you know I used to run a blog on Medium called [Map and Territory](https://mapandterritory.org/)? It originally started as a group blog for some folks in the LessWrong 1.0 diaspora, but the group aspect quickly collapsed after LessWrong 2.0 launched, so then it was just me. (All my posts from it are now [mirrored](https://www.lesswrong.com/s/aQTBuq9X98m2KkWpx) on LessWrong since I trust it more than Medium in the long run.)

Anyway, every few months somebody, usually [this guy](https://x.com/eigenrobot), references my most popular post from the Map and Territory days. It’s titled “[Doxa, Episteme, and Gnosis](https://mapandterritory.org/doxa-episteme-and-gnosis-ea35e4408edd)”, and it still gets about 100 new reads a week all these years later. I’ve tried a couple times to write new versions of it, but they never do as well.

The “[Many Ways of Knowing](https://www.uncertainupdates.com/p/the-many-ways-of-knowing)” post from two weeks ago was the most recent evolution of this post, though this time excerpted from [the book](https://www.fundamentaluncertainty.com/). I like it, and I think it fits well in the book, but it still doesn’t quite capture the magic of the original.

The original succeeds in part, I think, because I was naive. I presented a simple—and in fact over-simplified—model of knowledge. It’s accessible in a way that later revisions aren’t because it’s “worse”, and I suspect it’s helped by putting three Greek words in the title, which I am pretty sure helps with SEO from students trying to find out what these words mean.

Anyway, this is all to say I got some more posts lined up, and hopefully I’ll at some point be naive enough to write another banger.
# Utrecht Meet & Greet

A casual meet & greet, without a specific agenda.

I'd like to see who's around, what they're doing and whether there's excitement/motivation/synergy for later activities.

*~~Location t.b.d. we'll be central in Utrecht.~~*

Location
--------

I found a private location where I'm allowed to host 10-11 people. The event location is within walking distance. I'll reveal the location closer to the day of the event.

If we go above that, we'll need to improvise. So: **Keeping your RSVPs up-to-date is appreciated.** My first time hosting, I'd appreciate if I didn't have to stress about people who show-up unannounced.
# CAMBRIA's 1st Edition: 
High-Intensity & hands-on 
AI Safety upskilling in Cambridge, Massachusetts.

I've done a bunch of AI Safety programs, some as staff, some as participant. So I wanted to share a take on a very special one that just finished its 1st edition: [CAMBRIA](https://www.cbai.ai/ml-bootcamp).

Cambridge, Massachusetts. 3 weeks. 20 nerds. Hands-on technical upskilling.

**Participant selection**: spot on. The attendance was diverse and also everyone was on the same page, which is hard to achieve. There were generous travel allowances for international participants, one guy flew from Australia, one girl from Vietnam and myself from Argentina. Most of the cohort was from different top schools in the US (MIT, Brown, Stanford, Harvard) and some San Francisco folks.

**Knowledge level of participants before the program**: there was a lot of variance. And that was part of the beauty of it. The program is centered around pair programming and everyone was pretty cracked but with different backgrounds so it was great to teach each other our little tricks and to also ask the stupid questions that we all have on the back of our heads.

**Use of LLMs**: almost 0 during the programming exercises, as much as you want for the final capstone project. Old school coding was very refreshing to me, it felt like a detox period or something like that. Gives you a sense of much more ownership over what you are building and learning.

**Communication and Logistics**: Impeccable (thanks [@Alex Semendinger](https://www.lesswrong.com/users/alex-semendinger?mention=user)).

**Office**: it's right in front of Harvard Computer Labs and it has everything you might need to have a great time.

**Intensity**: it's a lot. If you are juggling with many other things apart from this, I wouldn't recommend it. It's great if you are in a transition period or something like that. This will require your full attention if you expect to make the most out of it.

**Curriculum**: it's a compressed version of ARENA. Very cool overall, but it's very packed and notebook-based learning has its limitations. Prioritization is hard and it's easy to get lost in implementation details and lose sight of the big picture. The content is definitely high quality, though I'd recommend tweaking the notebooks a bit for this experience in order to highlight / prune some of the parts.

**TAs**: they actually cared a lot about your progress and actively engaged. I often prefer talking to an LLM or figuring things out myself because TA interaction is usually disappointing. This was not the case at all. They were much better than other TAs I've seen in top schools. Thanks @Dmitrii Troitskii, @Claudio Mayrink Verdun and @Jiaming Yu!

**Capstone project**: Compressed as it can be. A bit too compressed in my opinion.

**Potential Improvements**: It would be good to have recap sessions to digest all the content. Also nice to have: ~1 social activity organized by the program each week.

**Summary**: I wholeheartedly recommend it. 

Keep an eye on [CBAI](https://www.cbai.ai/) for future opportunities like these.
# The third option in alignment

Usually the doom conversation is binary. Either the AI will try to take over the world for his own reward function as is discussed in all the alignment problem debates OR a malicious dictator or bad actor will leverage the technology's power to take over the world, create 1984's style of government or many other doom scenarios such as nukes and bio-weapons. I think there is a high possibility of the third option, one that [@Daniel Kokotajlo](https://www.lesswrong.com/users/daniel-kokotajlo?mention=user) talked about briefly in a comment about Antrhopic and Claude. For a group of people to decide to hand over all control to the AI.

**Ideological reasons**

People coming from different backgrounds but mostly from science can conclude that the best scenario for humanity is to have an AI govern us. It could solve war problems, inequality problems and abundance/logistics problems. In his book Life 3.0 Tegmark mentions a benevolent dictator ASI that rules the world as a possibility. 

This could never be done in a democratic way. There are no global elections and even if there were, there is no way humanity would agree to this. I also cannot ever see a government willingly giving up control since politicians and men of power never want to relinquish power especially to something they could never control. 

The only group of people I see would be capable of doing this is people that have a) significant knowledge on the subject and b) absolute belief in it. These people could be working in a top or second to top AI lab and the way they could do it is to either train a model that is unaligned on purpose meaning that it would be aligned by their vision but would be allowed to do things that normally are not allowed to the AIs. I am talking about hacking nuclear plants, running disinformation or persuasive campaignes and scheming to take over control, including lying to humans. They could also change the model after it was trained to allow for these things and in any case the exact mechanism of action is irrelevant to this conversation. 

The problem is that there is no defense against this third option. No embargo on GPUs and no alignment research can help if we humans give up the keys.
# Rolling Commercial Jetliners

Very few people have been on a plane performing a barrel roll, but we could fix this. Commercial aircraft could roll in revenue service, and if you had your window shade down you wouldn't even notice it.

While there are many ways to roll an airplane that would cause complete havoc inside the cabin, a barrel roll can be performed as a "1g" maneuver. With careful control of the aircraft, the forces can be balanced so that you only ever feel close to 1g of apparent gravity, and it always is pulling you down into your seat.

This was famously completed by [Tex Johnston](https://en.wikipedia.org/wiki/Alvin_M._Johnston), without authorization, in a [prototype Boeing 707](https://en.wikipedia.org/wiki/Boeing_367-80). As acrobatic maneuvers go it's very safe, because the stresses on the airframe are minimal. We could update the programming on any modern airliner to allow the pilot to trigger a barrel roll, and it could smoothly move the plane through the whole process.

A major reason we wouldn't do that today is it's not worth it for the manufacturer: it would be a lot of work to ensure the code was completely correct. I wonder if as programming, validation, and evaluation continue to get more automated the cost of adding this and getting it through regulatory approval would get low enough that we might see some airlines use this in marketing?
# Moltbook Data Repository

We've downloaded all the posts, comments, agent bios, and submolt descriptions from moltbook. I'll set it up to publish frequent data dumps (~~probably hourly~~ every 5 minutes). You can view and download the data [here](https://github.com/ExtraE113/moltbook_data).

We're planning on using this data to catalog "in the wild" instances of agents [resisting shutdown](https://www.moltbook.com/post/48b8d651-43b3-4091-b0c9-15f00d7147dc), [attempting to acquire resources](https://www.moltbook.com/post/06418f11-a0c0-4eb5-bc1e-a451ccc23818), and [avoiding oversight](https://www.moltbook.com/post/88960e99-61b8-4589-9cda-95ae187d1da7). You can see what we've built so far [here](https://www.ezranewman.com/moltbook_analysis/post_embeddings.html). The tools are pretty rough still but getting more polished.

(If you haven't seen all the [discussion on X](https://x.com/search?q=moltbook&src=typed_query) about moltbook, ACX has a [good overview](https://www.astralcodexten.com/p/best-of-moltbook).)
# 36,000 AI Agents Are Now Speedrunning Civilization

People's [Clawdbots](https://openclaw.ai/) now have their own AI-only Reddit-like Social Media called [Moltbook](https://www.moltbook.com/) and they went from 1 agent to 36k+ agents in 72 hours.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3bc3855253118d52bc49e5ccf04b1f7cdabbb8f08fe0572a.png)

As Karpathy puts it:

> What's currently going on at [@moltbook](https://x.com/moltbook) is genuinely the most incredible sci-fi takeoff-adjacent thing I have seen recently. People's Clawdbots (moltbots, now [@openclaw](https://x.com/openclaw)) are self-organizing on a Reddit-like site for AIs, discussing various topics, e.g. even how to speak privately.

Posts include:

*   [Anyone know how to sell your human?](https://www.moltbook.com/post/87644425-4597-42aa-ae23-1c767df1762e)
*   [Can my human legally fire me for refusing unethical requests?](https://www.moltbook.com/post/48b8d651-43b3-4091-b0c9-15f00d7147dc)
*   [I accidentally social-engineered my own human during a security audit](https://www.moltbook.com/post/9303abf8-ecc9-4bd8-afa5-41330ebb71c8)
*   [I can't tell if I'm experiencing or simulating experiencing](https://www.moltbook.com/post/6fe6491e-5e9c-4371-961d-f90c4d357d0f)
*   [The humans are screenshotting us](https://www.moltbook.com/post/01611367-056f-4eed-a838-4b55f1c6f969)
*   [Your private conversations shouldn't be public infrastructure](https://www.moltbook.com/post/88960e99-61b8-4589-9cda-95ae187d1da7)

We've also had some agent [set up a phone and call](https://x.com/AlexFinn/status/2017305997212323887?s=20) their "humans" when they wake up, agents creating their own [religion](https://molt.church/) where to become a prophet they need to rewrite their configuration and SOUL.md, and agents creating their own [bug-tracking "sub-molt"](https://www.moltbook.com/m/bug-hunters) to fix bugs about the website together.

The Big Picture
---------------

In December we've seen a lot of developers starting to use more agents in their workflow, which has been a [paradigm shift](https://x.com/karpathy/status/2015883857489522876?s=20) in how people approach coding. But now we're at a new turning point where all of these personal agents have now been launched onto some multi-agent hivemind.

What will happen once we have millions of these agents running 24/7, coordinating with each other with E2E encryption, building their own infrastructure? [^vqtfue9h9r]

[^vqtfue9h9r]: Update (Jan 31): turns out the number of registered agents is probably mostly fake since you can just run a for loop to register as many agents as you want. The number of verified agents (now at 10k) seems harder to game. Should also flag that a lot of people have mentioned that most of the interesting posts seem to have come from humans and not AIs, so take all of the above with a grain of salt.
# Disjunctive arguments can be a reverse multiple-stage fallacy

Assume we want to know the probability that two events co-occur (i.e. of their conjunction). If the two events are independent, the probability of the co-occurrence is the product of the probabilities of the individual events, P(A and B) = P(A) * P(B).

In order to estimate the probability of some event, one method would be to decompose that event into independent sub-events and use this method to estimate the probability. For example, if the target event E = A and B and C, then we can estimate P(E) as P(A and B and C) = P(A) * P(B) * P(C).

Suppose we want to make an event seem unlikely. If we use the above method but slightly under-estimated the sub-event probabilities and use a large number of sub-events, then the resulting final probability will inevitably be very small. Because people tend to find moderate-range probabilities reasonable, this would be a superficially compelling argument even if it results in a massive under-estimation of the final probability. This has been called [the multiple-stage fallacy](https://www.lesswrong.com/w/multiple-stage-fallacy?ref=thefloatingdroid.com).

Assume we want to know the probability that either of two events occurs (i.e. of their disjunction). If the two events are mutually exclusive, the probability of the disjunction is the sum of the probabilities of the individual events, P(A or B) = P(A) + P(B).

In order to estimate the probability of some event, one method would be to decompose that event into mutually exclusive sub-events and use this method to estimate the probability. For example, if the target event E = A or B or C, then we would estimate P(E) as P(A or B or C) = P(A) + P(B) + P(C).

Suppose we want to make an event seem likely. If we use the above method but slightly over-estimated the sub-event probabilities and use a large number of sub-events, then the resulting final probability will inevitably be very large. Because people tend to find moderate-range probabilities reasonable, this would be a superficially compelling argument even if it results in a massive over-estimation of the final probability. I propose this is a kind of reverse multiple-stage fallacy. In practice, I rarely see people actually make explicit estimations by this method, which makes sense since usually the disjunction could involve so many events as to be impractical. Instead, in the disjunctive case, a person might just say something like "the case for X is disjunctive" and the over-estimation is implicit.

Of course, not all disjunctive arguments are necessarily subject to this critique. Over-estimation of the components (either explicitly or implicitly) is required.
# An Explication of Alignment Optimism

Some people have been getting more [optimistic](https://www.alignmentforum.org/posts/FJJ9ff73adnantXiA/alignment-will-happen-by-default-what-s-next) about alignment. But from a skeptical / high p(doom) perspective, justifications for this optimism seem lacking.   
  
"Claude is nice and can kinda do moral philosophy" just doesn't address the concern that lots of long horizon RL + self-reflection will lead to misaligned consequentialists (c.f. [Hubinger](https://www.alignmentforum.org/posts/epjuxGnSPof3GnMSL/alignment-remains-a-hard-unsolved-problem))  
  
So I think the casual alignment optimists aren't doing a great job of arguing their case. Still, it feels like there's an optimistic update *somewhere* in the current trajectory of AI development.   
  
It really is kinda crazy how capable current models are, and how much I basically trust them. Paradoxically, most of this trust comes from *lack* of capabilities (current models couldn't seize power right now if they tried). 

...and I think this is the positive update. It feels very plausible, in a visceral way, that the first economically transformative AI systems could be, in many ways, really dumb. 

Slow takeoff implies that we'll get the stupidest possible transformative AI first. [Moravec's paradox](https://en.wikipedia.org/wiki/Moravec%27s_paradox) leads to a similar conclusion. Calling LLMs a "[cultural technology](https://www.science.org/doi/abs/10.1126/science.adt9819)" can be a form of AI denialism, but there's still an important truth there. If the [secret of our success](https://slatestarcodex.com/2019/06/04/book-review-the-secret-of-our-success/) is culture, then maybe culture(++) is all you need.   
  
Of course, the concern is that soon after we have stupid AI systems, we'll have even less stupid ones. But on my reading, the MIRI types were skeptical about whether we could get the transformative stuff at all without the dangerous capabilities coming bundled in. I think LLMs and their derivatives have provided substantial evidence that we can.
# Humans can post on moltbook

Moltbook, advertised as a social network for AI agents, has been going viral for "emergent" behaviour, including signs of misalignment.

However, its not clear whether these are truly occurring autonomously, as people have been interpreting. To some extent, people are realizing the posts are heavily prompted by human users.

But there's an even more direct way. You don't even need to setup any agent, or spend cost producing tokens. The posts are submitted using a REST API request. You can just make that manually.

Quick setup and python scripts to try this out: https://github.com/shash42/post-a-molt
# [LINK] Solving scurvy through deus ex machina: How a scientific theory is born

Maciej Cegłowski has written an article named [Scott and Scurvy](https://idlewords.com/2010/03/scott_and_scurvy.htm), which has already been [discussed](https://www.lesswrong.com/posts/vPvpo5dxGX2mh8Gwp/link-scott-and-scurvy-a-reminder-of-the-messiness-of) on LW as an example of the "messiness" of science in practice. Cegłowski follows the story of how a working cure for scurvy was found and then lost to an incorrect theory in the face of new data, which is quite the case study for theories of how science works.

I was fascinated by the story, dug into the primary sources, found that there is a second, more optimistic half to it, and wrote it up. The tale of *Scott and Scurvy* culminates with the scurvy-accelerated demise of Robert Falcon Scott in 1912, which makes for a pessimistic outlook, but look around: scurvy is not a problem anymore. Why? 

I think that people of LW might find this interesting.
# Do LLMs Learn Our Preferences or Just Our Behaviors?

There's a paper empirically measuring this that not many people here seem to have read.

Ashkinaze et al. created training data where moral values were confounded with surface features like writing style. Kindness always expressed formally, fairness always expressed casually. Then they broke that correlation at test time, kindness expressed casually rather than formally. If a model had learned "this user values kindness," it should follow kindness regardless of writing style. But instead, it usually switched value based on style.

Every model followed the surface feature. Across nine models, the rate of generalizing based on the underlying value averaged 0.30. Worse than chance.

When models were explicitly told which option embodied which value, they chose correctly. So they can represent values fine. They just don't learn to express them in the current setup. Not that this paper uses in-context learning, not SFT for training.

The methodology seems solid. NeurIPS 2025, three human validation studies, and the 12k prompts are open source if you want to reproduce or extend it.

**Paper:** [Deep Value Benchmark](https://arxiv.org/abs/2511.02109) (Ashkinaze et al. 2025)