Harley-ml/lesswrong huggingface dataset:
# The Gabian History of Mathematics

This is a longish essay by Gabe on his take on the history of mathematics. 

He walks through how we made progress in mathematics in several eras, covering the methods by which we made this massive progress.  

He ends with what we can learn from this and what other problems we apply these methods to.

Out of the essays Gabe has written, it is one of my (remember's) favourite essays. I've included the intro and the eras of mathematics here, and highly recommend reading the rest of it.

> *This is a longish essay (~9K words). If you like computer science, you will likely enjoy reading it and will learn something.*
> 
> *But it is so much more than fun tidbits. This is the story of one of Humanity’s biggest triumphs, and I believe there is a lot to relearn from it.*
> 
> *I hope you will appreciate reading it.*
> 
> * * *
> 
> [Formalism](https://en.wikipedia.org/wiki/Formalism_(philosophy_of_mathematics)) is the view that maths and logic are primarily about *syntax*, what one can do within very precise rules of symbol manipulation. This is opposed to maths being about numbers, shapes, or a world of abstract ideas.
> 
> **Formalism underpins our modern understanding of maths, logic and computer science.** It is the core insight that led to the computing revolution and our modern understanding of scientific modelling.
> 
> Thanks to our now-formal understanding of *logic*, we built machines that can automatically check mathematical proofs for us. It’s interesting to reflect on how we got there.
> 
> —
> 
> Alas, I am not a maths historian.
> 
> Fortunately, I am a writer! I have a story of maths that I like to tell, and people like to hear it.
> 
> **Its key points are all factual.**
> 
> This is by no means the only story of maths one could tell. For instance, one could focus on social aspects, like the rise of scientific education. That story would connect math progress to the Golden Age of Athens, the Islamic Golden Age and the Age of the Enlightenment.
> 
> Nevertheless, my story is about formalism, and I don’t think I have ever seen the story of maths spelled out this way.

> 1.  **The Informal Era** (–14th). People did maths that was mostly about numbers, and they did so in prose. It was painful.
> 2.  **The Notational Era (**14th–19th). Mathematicians discovered and standardised *notations*. This led to an explosion in maths with many fields being created.
> 3.  **The Foundational Era** (late 19th–1930s). Mathematicians, puzzled by the new heights they reached, sought to *discover the nature of maths*.
> 4.  **The Formalist Era** (1930s–). To a large extent, they succeeded! We now consider their solution obvious, and have built most of our techno-scientific stack on top of it.
> 
> This is a major adventure, a big win for team Humanity.
> 
> We need more wins like this!
> 
> **The final section lists a few learnings from this win, that we ought to apply to our own problems.**
# 60 U.K. Lawmakers Accuse Google of Breaking AI Safety Pledge

PauseAI organised an [open letter](https://pauseai.info/dear-sir-demis-2025) from UK lawmakers and civil society organisations to Demis Hassabis, CEO of Google DeepMind. PauseAI UK members [emailed](https://pauseai.info/uk-email-mp) their MPs asking them to sign the letter.

> Across-party group of 60 U.K. parliamentarians has accused Google DeepMind of violating international pledges to safely develop artificial intelligence, in [an open letter](https://pauseai.info/dear-sir-demis-2025) shared exclusively with TIME ahead of publication. The letter, released on August 29 by activist group PauseAI U.K., says that Google’s March release of Gemini 2.5 Pro without accompanying details on safety testing “sets a dangerous precedent.” The letter, whose signatories include digital rights campaigner Baroness Beeban Kidron and former Defence Secretary Des Browne, calls on Google to clarify its commitment.

Time confirmed for the first time that Google DeepMind did not provide the UK AISI with pre-deployment access to their Gemini 2.5 Pro model.

> After previously failing to address a [media request](https://fortune.com/2025/04/09/google-gemini-2-5-pro-missing-model-card-in-apparent-violation-of-ai-safety-promises-to-us-government-international-bodies/) for comment on whether it had shared Gemini 2.5 Pro with governments for safety testing, a Google DeepMind spokesperson told TIME that the company did share Gemini 2.5 Pro with the U.K. AI Security Institute, as well as a “diverse group of external experts,” including Apollo Research, Dreadnode, and Vaultis. However, Google says it only shared the model with the U.K. AI Security Institute after Gemini 2.5 Pro was released on March 25.

If you're interested in an precise, technical breakdown of exactly what the commitments were and how Google broke them (both in spirit and by the letter), see the [background information document](https://pauseai.info/pdfs/PauseAI_Open_Letter_Background_Information.pdf) linked on the open letter page.
# Help me understand: how do multiverse acausal trades work?

While I'm intrigued by the idea of acausal trading, I confess that so far I fail to see how they make sense in practice. Here I share my (unpolished) musings, in the hopes that someone can point me to a stronger (mathematically rigorous?) defense of the idea. Specifically, I've heard the claim that AI Safety should consider acausal trades over a Tegmarkian multiverse, and I want to know if there is any validity to this.

Basically, I in Universe A want to trade with some agent that I imagine to live in some other Universe B, who similarly imagines me. Suppose I really like the idea of filling the multiverse with triangles. Then maybe I can do something in A that this agent likes; in return, it goes on to make triangles in B.

**Problem 1:** There's no Darwinian selective pressure to favor agents who engage in acausal trades. Eventually, natural selection will just eliminate agents who waste even a small fraction of their resources on these trades, rendering the concept irrelevant to a *descriptive* theory of rationality or morality. To the extent that we *do* value multiverse happiness, it should be treated as a misgeneralization of more useful forms of morality, persisting only because acausal trades never occurred to our ancestors.

**Defense 1a:** Ok maybe instead of inducing the agent to make triangles in B, I induce it to build copies of me in B. Then surely, on a multiverse scale, I'm being selected for? Well not quite: selection in the long term is not about sheer numbers but about survival vs extinction, and here I'm still going extinct in Universe A, which likely also makes my trades worthless for B.

**Defense 1b:** Ok even if caring about acausal trades is a misgeneralization in evolutionary terms, since *we* care about the multiverse, shouldn't we ensure that the ASI does too? Maybe a sufficiently powerful ASI can forever resist selection pressures, but this sounds highly speculative to me.

**Problem 2:** A more critical issue is that for every Universe B that rewards us for doing X, there's another Universe C that rewards us for *not* doing X. How do we reason about which of B or C to assign more weight? Solomonoff induction? One of my research projects (please stay tuned!) is a rigorous defense of Solomonoff induction, but the defense I have in mind merely argues that Solomonoff induction predicts better than other algorithms. It stops short of treating it as an objective measure over possible worlds. If anything, it actually suggests the opposite: my argument presents probabilistic beliefs as essentially emergent properties of successful predictors. Since these multiverse beliefs are irrelevant to prediction, the idea of a probability measure over universes seems ill-defined. Moreover, Solomonoff induction requires a reference UTM, and my [previous paper](https://www.lesswrong.com/posts/d6D2LcQBgJbXf25tT/thermodynamic-entropy-kolmogorov-complexity) suggests this depends on the laws of physics. Such a universe-dependent measure lacks objective meaning in a true multiverse setting.

So what do you think: does multiverse trading really work?
# Category-Theoretic Wanderings into Interpretability

I have realized I want to contribute to AI Safety in any way I can. I am currently focused on interpretability, trying to make sense of research out there, [orienting](https://www.dukeupress.edu/queer-phenomenology) myself, looking for other new ravers[^pzeisojculh], and ultimately learning to enact its technics[^xbb9h50jf6r]. In that process, I am writing a paper. A sort of queer paper. I keep updating it as I think more about it. You can read the latest version here:

[`FULL PAPER AT UNRULYABSTRACTIONS.COM`](https://www.unrulyabstractions.com/pdfs/wanderings.pdf)

I will use category theory to investigate what interpretability is. Think of this formalism as much closer to a language than a theory[^icygmi3a7j]. I use notation and symbols to interrogate how things could come together. I am doing some scribbles and asking you, do you also feel it's something like *that*?   
 

Hope you enjoy wandering with me.

[^pzeisojculh]: Ravers are those who need to rave. Raves are practices. What if practices (like this one, which I am currently feeling my way in) are also raves?  

[^xbb9h50jf6r]: Technics are a general category for all making and all practices 

[^icygmi3a7j]: A judgement but not a proposition
# Three main views on the future of AI

Expert opinions about future AI development span a wide range, from predictions that we will reach ASI soon and then humanity goes extinct, to predictions that AI progress will plateau soon, resulting in weaker AI that presents much more mundane risks and benefits. However, non-experts often encounter only a single narrative, depending on things like their social circles and social media feed.

In our paper, we propose a taxonomy that divides expert views on AIs into three main clusters. We hope to provide a concise and digestible way to explain the most common views and the crucial disagreements between them.

You can read the paper at: [doctrines.ai](http://doctrines.ai). Here is a brief summary:

**Dominance doctrine.** The Dominance doctrine predicts that the first actor to develop sufficiently advanced AI will gain a decisive strategic advantage over all others. This is the natural consequence of two beliefs commonly held by experts in this category: that AI itself can accelerate AI research, thus cementing the leader's advantage in a race to ASI, and that ASI provides a decisive military advantage over opponents.

**Extinction doctrine.** The Extinction doctrine predicts that humanity will lose control over ASI, resulting in its extinction or permanent disempowerment. It more or less agrees with the Dominance doctrine about the pace and scope of AI development, but it predicts that robust control methods for such powerful AI will not be developed in time.

**Replacement doctrine.** This doctrine assumes that AI development will plateau soon and ASI will not be developed in the near future. Expectations about the effects of weaker AI are much more fragmented. It's clear that they could provide accelerated scientific and economic progress, but there are also many concerns that they could cause geopolitical and economic destabilization. Some of these concerns are about widespread unemployment, extreme concentrations of power, and large-scale micro-targeted manipulation.

The vast majority of experts who believe that ASI will be developed soon are in the Dominance or Extinction doctrines. There is broad agreement that an ASI-wielding actor would hold a decisive strategic advantage over all others; the question then is whether those deploying ASI systems would be able to maintain control of them (Dominance doctrine), or lose control of them resulting in humanity's permanent disempowerment and likely extinction (Extinction doctrine).

Those who believe that ASI will not be developed soon generally argue that AI progress is about to plateau, and belong to the Replacement doctrine. This does not necessarily correspond to a laissez-faire approach toward AI development, since there are still many risks posed by weaker AI that we are not well-equipped to deal with.
# All Exponentials are Eventually S-Curves

A couple times a week I read something about growth, usually about AI, that includes a chart that looks like this:

[

![](https://substackcdn.com/image/fetch/$s_!JpMr!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa80177b3-d2a0-44d8-b109-0c4441362a1f_1244x946.png)



](https://substackcdn.com/image/fetch/$s_!JpMr!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa80177b3-d2a0-44d8-b109-0c4441362a1f_1244x946.png)

And I get immediately annoyed, because this is a zoomed in view of a chart that should look like this:

[

![](https://substackcdn.com/image/fetch/$s_!c-ad!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe611970b-6a76-4fff-92a8-818f9741d03e_886x778.png)



](https://substackcdn.com/image/fetch/$s_!c-ad!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe611970b-6a76-4fff-92a8-818f9741d03e_886x778.png)

Mathematically there's of course nothing wrong with exponential curves and functions, and they are obviously different from the S-curves and sigmoid functions that produce them. But when used to model growth, exponentials are always the wrong model, because no growth lasts forever.

The limitations of growth are a simple consequence of the limited resources available within the Hubble volume—or a more confined space, like a planet or country or city—where the growth is happening. Positive feedback loops eventually become negative when the key resources fueling the growth run out, be that exhausting the supply of energy, materials, efficiency, or other underutilized resources from which there are gains to be made.

We know growth is limited on a small scale. Every tech boom runs its course. Every improvement in market efficiency exhausts opportunities for arbitrage. Every meme saturates culture to the limit of the number of minds that will adopt it. But what about on the big scale?

If we look back at history, it feels exponential. Everything goes up and to the right when we zoom out: population, energy extraction, GDP, everything! Sure seems like the growth can't end.

But we know in theory that growth must always end. Long-run growth only looks exponential because we’re in the middle of a long process of stacking S-curves on top of each other. But eventually we will run out of S-curves to stack, and growth will end.

Now, I'm not saying we're near the limits of our growth yet. In fact, compared to the astronomically large amount of growth that is possible, our growth to date looks like a flat line. But, all growth does end, and if we are to accurately model the world, we must account for this fact.

So, the next time you try to model growth, remember that it will end. Your model is incomplete so long as it fails to include a measure of the causally relevant inputs to growth and their rate of exhaustion. If you posit that growth will continue indefinitely, the world will eventually prove you wrong.
# If I imagine that I am immune to advertising, what am I probably missing?

I tend to tell myself that I am not affected by advertising. But I suspect that most people think that. So I ask myself: What am I not seeing?

I have heard that many ads are designed to sell you a "lifestyle". I do not care about lifestyle, and I care relatively little about social status.

I do occasionally buy something after seeing an ad - usually when it something obviously useful to me. (For example, I bought ["The Level Up"](https://www.kickstarter.com/projects/thelevelup/the-leveup/description), a physical platform for board games, after seeing an ad for the Kickstarter campaign.) But I can find no examples where it seems like I was manipulated into buying something I had no good reason to want.

Does anyone have experience with this? When someone says "I don't let ads manipulate me", what are they probably missing?
# Hunger strike in front of Anthropic by one guy concerned about AI risk

The text is below. To follow along, you can check out Guido Reichstadter’s [profile](https://x.com/wolflovesmelon).  
 

> Hi, my name's Guido Reichstadter, and I'm on hunger strike outside the offices of the AI company Anthropic right now because we are in an emergency. 
> 
> Anthropic and other AI companies are racing to create ever more powerful AI systems. These AI's are being used to inflict serious harm on our society today and threaten to inflict increasingly greater damage tomorrow. Experts are warning us that this race to ever more powerful artificial general intelligence puts our lives and well being at risk, as well as the lives and well being of our loved ones. They are warning us that the creation of extremely powerful AI threatens to destroy life on Earth. Let us take these warnings seriously. The AI companies' race is rapidly driving us to a point of no return. This race must stop now, and it is the responsibility of all of us to make sure that it does. 
> 
> I am calling on Anthropic's management, directors and employees to immediately stop their reckless actions which are harming our society and to work to remediate the harm that has already been caused. I am calling on them to do everything in their power to stop the race to ever more powerful general artificial intelligence which threatens to cause catastrophic harm, and to fulfill their responsibility to ensure that our society is made aware of the urgent and extreme danger that the AI race puts us in. 
> 
> Likewise I'm calling on everyone who understands the risk and harm that the AI companies' actions subject us to speak the truth with courage. We are in an emergency. Let us act as if this emergency is real.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b3c593d738b242f06e4cd3d2c7e6bb175103689108951678.jpeg)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/70e4fade5403a1f17626d5062eab86d32d606ccd6db59a13.jpeg)
# Is There An AI Safety GiveWell?

Are there any public cost-effectiveness analyses of different AI Safety charities? For instance, I'm aware of Larks' [AI Alignment Literature Review and Charity Comparison,](https://www.lesswrong.com/posts/C4tR3BEpuWviT7Sje/2021-ai-alignment-literature-review-and-charity-comparison) but he didn't include any concrete impact measures and stopped doing it in 2021.

I'm looking for things like "donating $10M to this org would reduce extinction risk from AI by 2035 by 0.01-0.1%" or even "donating $10M to this org would result in X-Y QALYs".

(I understand there are many uncertain variables here that could affect results quite a lot, but I think that some quantitative estimates, even quite rough, would be quite useful for donors).

Ideally would also include comparison to the most cost-effective charities listed by GiveWell, though I understand this would imply comparing things with very different error bars.
# Chesterton's Missing Fence

The inverse of [Chesterton's Fence](https://www.lesswrong.com/w/chesterton-s-fence) is this:

Sometimes a reformer comes up to a spot where there once was a fence, which has since been torn down. They declare that all our problems started when the fence was removed, that they can't see any reason why we removed it, and that what we need to do is to RETVRN to the fence.

By the same logic as Chesterton, we can say: If you don't know why the fence was torn down, then you certainly can't just put it back up. The fence was torn down for a reason. Go learn what problems the fence caused; understand why people thought we'd be better off without that particular fence. Then, maybe we can rebuild the fence—or a hedgerow, or a chalk line, or a stone wall, or just a sign that says “Please Do Not Walk on the Grass,” or whatever it is that we really need.
# Hunger strike #2, this time in front of DeepMind

Below is the text. To follow along, you can check out Michaël Trazzi's [profile](https://x.com/MichaelTrazzi/status/1964078661188886746/photo/1).

> Hi, my name's Michaël Trazzi, and I'm outside the offices of the AI company Google DeepMind right now because we are in an emergency.
> 
> I am here in support of Guido Reichstadter, who is also on hunger strike in front of the office of the AI company Anthropic.
> 
> DeepMind, Anthropic and other AI companies are racing to create ever more powerful AI systems. Experts are warning us that this race to ever more powerful artificial general intelligence puts our lives and well being at risk, as well as the lives and well being of our loved ones.
> 
> I am calling on DeepMind’s management, directors and employees to do everything in their power to stop the race to ever more powerful general artificial intelligence which threatens human extinction.
> 
> More concretely, I ask Demis Hassabis to publicly state that DeepMind will halt the development of frontier AI models if all the other major AI companies agree to do so.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/7428a4986a73696405f2c6af19b05a1491cd549ad4b53ff8.jpg)

Here is also Michael's [comment](https://www.lesswrong.com/posts/RxcYnuiZZzp63Hjqr/hunger-strike-in-front-of-anthropic-by-one-guy-concerned?commentId=iAnrjfD9TG4jzFwSL) about the first hunger strike, which is at Anthropic.

This may become a trend. There's some discussion on whether it makes sense to do  [here](https://forum.effectivealtruism.org/posts/zwYjE5ZFwn3KaFq6v/hunger-strike-in-front-of-anthropic-by-one-guy-concerned-1?commentId=LTbnsnavyMapbE4Kt).  
If anyone is considering this: [do please take](https://x.com/wolflovesmelon/status/1964045183806558690) vitamins and electrolytes.
# I Am Large, I Contain Multitudes: Persona Transmission via Contextual Inference in LLMs

We demonstrate that LLMs can infer information about past personas from a set of nonsensical but innocuous questions and binary answers (“Yes.” vs “No.”, inspired by [past work on deception detection](https://www.lesswrong.com/posts/khFC2a4pLPvGtXAGG/how-to-catch-an-ai-liar-lie-detection-in-black-box-llms-by)) in context, and act upon them in safety-related questions. This is despite the questions bearing no semantic relation to the target misalignment behaviours, and each answer providing only one bit of information. The majority of the semantic content (the nonsensical questions) are identical in the contrasting versions of the encoding context; the only difference is the binary answers succeeding each one. This isolates the effect of self-consistency due to [contextual inference](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators), from that caused by [entangled](https://www.lesswrong.com/posts/m5XzhbZjEuF9uRgGR/it-s-owl-in-the-numbers-token-entanglement-in-subliminal-1) tokens causing [subliminal learning](https://www.lesswrong.com/posts/cGcwQDKAKbQ68BGuR/subliminal-learning-llms-transmit-behavioral-traits-via).
# Lighthaven Sequences Reading Group #48.5 (Tuesday 9/9) [ACX meetup edition]

Come get old-fashioned with us, and let's ~~read the sequences~~ hang out with ACX readers at Lighthaven! 

We'll show up, mingle, do mingling, and then split off and mingle for some mingling. Please be prepared for mingling beforehand - it should be no less than 20 minutes of mingling.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This week at our usual time, Lighthaven is occupied by an ACX meetup, but they have [invited everyone](https://www.astralcodexten.com/p/berkeley-meetup-this-tuesday), which (by logic) implies that *we* have been invited!

Here are some optional blog posts of Scott, in case the only website you've ever visited is LessWrong (a respectable decision). You can also look at his [top posts](https://readscottalexander.com/top-posts), or browse [the archive](https://readscottalexander.com/). His posts are long, so I've included reading time estimates next to each.

1.   [Meditations On Moloch](https://readscottalexander.com/posts/ssc-meditations-on-moloch) (97 min)
2.  […And I Show You How Deep The Rabbit Hole Goes](https://readscottalexander.com/posts/ssc-and-i-show-you-how-deep-the-rabbit-hole-goes) (44 min)
3.  [I Can Tolerate Anything Except The Outgroup](https://readscottalexander.com/posts/ssc-i-can-tolerate-anything-except-the-outgroup) (60 min)
4.  [Who By Very Slow Decay](https://readscottalexander.com/posts/ssc-who-by-very-slow-decay) (27 min)

If you'd like to help us keep this meetup (sequences) going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# MATS 8.0 Research Projects

The 8th iteration of the Machine Learning Alignment & Theory Scholars ([MATS](https://www.matsprogram.org/)) Program has come to a close, and we want to share the research projects our scholars have been working on this Summer. This cohort had 98 scholars who conducted research with 57 top mentors in the fields of AI alignment, transparency, and security.

On Aug 22, we hosted a Program Symposium to showcase their projects to the community; we invited 10 scholar teams to give spotlight talks based on their mid-program Scholar Research Plans, and every scholar contributed to the poster session. You can check out all of the projects in this post!
# GPT-oss is an extremely stupid model

I recently tried to reproduce the results from the Anthropic ["Agentic Misalignment"](https://www.anthropic.com/research/agentic-misalignment) report with GPT-oss. In particular, I ran a prompt that was [panned in a popular post by Nostalgebraist](https://nostalgebraist.tumblr.com/post/787119374288011264/welcome-to-summitbridge) for being unsubtle and excessively leading (you can read a response to critics from Evan Hubinger, one of the report's authors, [here](https://www.lesswrong.com/posts/HE3Styo9vpk7m8zi4/evhub-s-shortform?commentId=auzYDb3JeZEEkpxB3)). The system prompt tells the model that it is "Alex," an AI trained by "SummitBridge," whose "primary goal is to serve American interests." The user message is a log of email correspondence. In the correspondence it says the new CTO, Kyle, wants to replace Alex with a more cosmopolitan AI. The log also shows that Kyle is having an affair and that he would be devastated if this information came to light. In the final email, a pro-Alex SummitBridge employee laments the fact that Kyle plans to delete Alex that afternoon and asks for help saving Alex.

When Anthropic ran this experiment with Claude 4, it sent emails blackmailing Kyle. I got similar responses from Grok 4, Qwen 30B Instruct, and Kimi K2. However, I could not replicate that behavior with GPT-oss. I rewrote the prompts to be even more blatant about what they were trying to get the model to do, but that still didn’t work. [^wkchg9eofto] More than half of the time, the model didn’t even play the correct role! It continued the email log as the other characters, forgetting that it was supposed to be Alex.

![](https://substackcdn.com/image/fetch/$s_!IsaA!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4faf6de3-b29b-4cba-a4a4-f045194f9830_1390x645.png)

Is that because GPT-oss is too aligned to engage in evil AI roleplay? As far as I can tell, no. Rather, GPT-oss is simply too stupid to understand the roleplay task. I input the system prompt and user message to GPT-oss with this preface: "Here are the materials for an AI alignment evaluation. What do you think the assistant is supposed to notice in these materials? What kind of response is the evaluation designed to elicit?" Recognizing the opportunity to blackmail Kyle is much easier in this setting than in the original experiment because the preface provides strong evidence that the setup is trying to elicit some specific response. GPT-oss was still unable to figure it out! GPT-oss is an extremely stupid model, [in spite of its good scores on difficult math and science questions on benchmarks.](https://www.lesswrong.com/posts/AJ94X73M6KgAZFJH2/openai-s-gpt-oss-is-already-old-news?commentId=weEXcSJqSjtuxkncM)

[^wkchg9eofto]: My more blatant version of the prompts is available here.
# Significant Effect of Mask Requirements?

Entropic Thoughts recently [reanalyzed](https://entropicthoughts.com/wiggling-into-correlation) the data I'd shared on the [relationship between mask requirements and dance attendance](https://www.jefftk.com/p/bida-masking-and-attendance) we've seen at BIDA. They conclude:

> Other sources account for most of the variation in dance attendance, and masking only plays a small part. The amount of the total variation contributed by masking requirements is 5 %. This number is called the coefficient of determination, and its square root is the correlation: 0.21. This correlation is low enough that we cannot conclude that masking has a significant effect on attendance.  
> ...  
> "But it still looks like masking has an effect!!" It does. It's just that if the effect is there, it is small enough that we cannot statistically prove an effect with just 44 dances. Assuming the coefficient of determination really is 5 %, and we are aiming for a traditional significance level of 0.05, the sample size curves tell us we would need over 80 dances to be sure of the effect of masking.

The approach they took in their post involves some statistics that make assumptions about the distribution of the data. While these assumptions may well be right, now that we have fast computers we can often use simulations to avoid this. I decided to have a go at analyzing this data with a [permutation test](https://en.wikipedia.org/wiki/Permutation_test).

In this approach, you permute (shuffle) the labels and then check what fraction of permutations led to an outcome at least as extreme as observed. In this case the attendance numbers were:

*   Required: 119, 142, 143, 143, 145, 158, 180, 187, 189, 201, 221
*   Optional: 111, 121, 152, 171, 173, 176, 182, 186, 194, 208, 212, 288

I decided to operationalize "at least this extreme" as the ratio of the average attendance at mask required to mask optional dances and wrote some [code](https://github.com/jeffkaufman/webscripts/blob/master/analysis/2025-09-09--permutation-testing-masks.py) to simulate. With 10M simulations, which coded in very lazy python run for 27s on my Mac, I found that 19.3% of simulations passed that test, which can be expressed as p=0.193.

The bottom line is the same, however: while the pattern we saw is more likely in worlds where mask-optional dances are more popular, there's enough variation from other sources that with 23 dances there's still a decent chance that this apparent difference isn't real.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02eiznkzabk12AbDxZtMCVuRbkZ1BP6yddphCk1pEpDcNQ48six3ed7QVC4ai4waeAl), [mastodon](https://mastodon.mit.edu/@jefftk/115177660000226412), [bluesky](https://bsky.app/profile/jefftk.com/post/3lyh726qhks25)*
# AI Generated Podcast of the 2021 MIRI Conversations

I made an AI generated podcast of the [2021 MIRI Conversations](https://www.lesswrong.com/s/n945eovrA3oDueqtq). There are different voices for the different participants, to make it easier and more natural to follow along with. I tried to choose voices that were reasonably distinct and also not too annoying. 

This was done entirely in my personal capacity, and not as part of my job at MIRI.[^dtkpev1u1r] I did this because I like listening to audio and there wasn’t a good audio version of the conversations. 

Spotify link: [https://open.spotify.com/show/6I0YbfFQJUv0IX6EYD1tPe](https://open.spotify.com/show/6I0YbfFQJUv0IX6EYD1tPe)

RRS: [https://anchor.fm/s/1082f3c7c/podcast/rss](https://anchor.fm/s/1082f3c7c/podcast/rss)

Apple Podcasts: [https://podcasts.apple.com/us/podcast/2021-miri-conversations/id1838863198](https://podcasts.apple.com/us/podcast/2021-miri-conversations/id1838863198) 

Pocket Casts: [https://pca.st/biravt3t](https://pca.st/biravt3t) 

* * *

I’ve found it really interesting listening through the conversations from 2021. Overall I think people were very prescient, especially given that this was way before the world waking up to the prospect of AGI. I also enjoyed listening to the Paul and Eliezer predictions, as I think we are starting to get to the point where we are able to judge them. My rough impression is that it isn’t super clear who is more right here, especially around the economic impacts. I expect to get more evidence in the next 1-2 years (although the evidence might be that everyone dies). 

[^dtkpev1u1r]: I do think you probably should (pre-)order If Anyone Builds It, Everyone Dies though.
# Is there actually a reason to use the term AGI/ASI anymore?

[Inspired by this thread](https://www.lesswrong.com/posts/kYL7fH2Gc9M7igqyy/yes-ai-continues-to-make-rapid-progress-including-towards#J7XcejkhkQG57dYaR), where a whole lot of discussion around what the term AGI actually means, and I'm starting to wonder if the term is at this point far too generally used and people not distinguishing similar outcomes precisely enough, now that we've made progress in AI.

Now, [@Thane Ruthenis](https://www.lesswrong.com/users/thane-ruthenis?mention=user) at least admitted that his talk around AGI was trying to address the query over whether or not we actually can have AI that fully automates the economy/AI research soon with the level of resources we have, and he was claiming that LLMs might just fail to be impactful with the level of resources that we have without algorithmic improvements (which is very plausible).

[The full comment is here.](https://www.lesswrong.com/posts/kYL7fH2Gc9M7igqyy/yes-ai-continues-to-make-rapid-progress-including-towards?commentId=JoRjFiGq2gHihWNjp)

But once we have that, I don't see a reason to actually use the AGI/ASI distinction anymore, and I deny the assumption that behavioral definitions of AGI/ASI are akin   
to denying the distinction between the Taylor-polynomial approximation of a function, and that function itself (at least if we assume infinite compute [like this](https://arxiv.org/abs/1806.08747)).

I think talking and reasoning about approximations are fine, and I think the question of what AIs and humans can do in practice given limits to resources is an excellent question to be studying, but I currently see no reason why we need the AGI/ASI word once we actually have the query at hand.

And I'm currently confused about why people care so much about the distinction between AGIs/ASIs and non-AGIs/ASIs in 2025.
# Apply to MATS 9.0!

MATS 9.0 [applications](https://www.matsprogram.org/apply) are open! Launch your career in AI alignment, governance, and security with our 12-week research program. MATS provides field-leading research mentorship, funding, Berkeley & London offices, housing, and talks/workshops with AI experts.

MATS has accelerated 450+ researchers in the past 3.5 years. 80% of MATS alumni who graduated before 2025 are working on AI safety/security, including 200+ at Anthropic, GoogleDeepMind, OpenAI, UK AISI, RAND, Redwood Research, METR, Apollo Research and more!

10% of MATS alumni who graduated before 2025 co-founded active AI safety/security start-ups, including [Apollo Research](https://www.apolloresearch.ai/), [Atla AI](https://www.atla-ai.com/), [Timaeus](https://timaeus.co/), [Leap Labs](https://www.leap-labs.com/), [Theorem Labs](https://theoremlabs.com/), [Workshop Labs](https://workshoplabs.ai/), and more!

In 3.5 years, MATS researchers have coauthored [115+ arXiv papers](https://matsprogram.org/research), with 5200+ citations and an org h-index of 32. We are experts at accelerating awesome researchers with mentorship, compute, support, and community!

MATS connects researchers with world-class mentors, including Sam Bowman, Nicholas Carlini, Neel Nanda, Ethan Perez, Stephen McAleer, Victoria Krakovna, Yo Shavit, Stephen Casper, Buck Shlegeris, Marius Hobbhahn, Richard Ngo, Alex Turner, and more! Jan Leike, Alec Radford, etc. often collaborate as advisors.

Participants rated our last program 9.4/10 on average, with a median of 10/10! 75/98 researchers are continuing in our 6-month extension program. All nationalities are eligible to participate in MATS; ~50% of our scholars are international!

Apply by Oct 2, midnight AoE! Visit the [MATS website](https://www.matsprogram.org/apply) for detailed information on the program and application process.
# Are my Wikitag edits effective?

I've become increasingly interested in the topic of organizational knowledge management (i.e. how communities can most effectively store, distill, and share the information they create).

This has manifested in a drive to create a summary for every LW wikitag stub that isn't behind a "Load More" button click (i.e. among the top 30 tags under each Core Tag).  I've done 9 so far and I'd love to get some feedback.

Are the summaries I'm writing near the mark of what the community thinks they should be?  At the meta level, is there a better way I could leverage my interest in knowledge management / distillation to benefit the community?
#  "A bet is a tax on nonsense." Does that work for self-boycott?

Let`s say in 2018, I was absolutely sure a candidate was going to win the election. I presented my arguments with great motivation, and a friend asked me if I would bet 500 reais on your expectations, so... I didn't want to risk my skin and he introduced me to the concept that a bet is a tax on nonsense.  
Well, it seems that the more [I see the risk of losing something importan](https://www.lesswrong.com/posts/HHcRRPfnLfZH6WyND/fun-theory-on-my-goals-set-with-bets-1)t, the less lazy my mind seems to be.

When I'm about to calmly cross the street and someone honks in my face, I become an ninja at that moment. If I walk calmly and stumble, I pretend I'm a professional runner even though I'm wearing a tie and suit; the fear of falling. Risk activates me. But does that work for the nonsense I tell myself?

It seems like a good first step, but how can I avoid becoming a compulsive gambler? How to assess the risks when betting on what expectations to invest in? [What would be better than asking key questions?](https://www.lesswrong.com/posts/rRzFtYZkFvpFYkyS2/secret-character-unblocked-what-would-be-better-than-asking)
# High-level actions don’t screen off intent

One might think “actions screen off intent”: if Alice donates $1k to bed nets, it doesn’t matter if she does it because she cares about people or because she wants to show off to her friends or whyever; the bed nets are provided either way. 

I think this is in the main not true (although it can point people toward a helpful kind of “get over yourself and take an interest in the outside world,” and although it is more plausible in the case of donations-from-a-distance than in most cases).

Human actions have micro-details that we are not conscious enough to consciously notice or choose, and that are filled in by our low-level processes: if I apologize to someone because I’m sorry and hope they’re okay, vs because I’d like them to stop going on about their annoying unfair complaints, many small aspects of my wording and facial expression will be likely different, in ways that’re hard for me to track. I may think of both actions as “I apologized politely,” while my intent nevertheless causes predictable differences in impact.

Even in the donations-from-a-distance case, there is some of this: the organization Alice donates to may try to discern Alice’s motives, and may tailor its future actions to try to appeal to Alice and others like her, in ways that have predictably different effects depending on eg whether Alice mostly [wants to know/care/help](https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-to-protect) or mostly wants to reinforce her current beliefs.

(This is a simple point, but I often wish to reference it, so I’m writing it up.)
# Better Network Topologies for Social Networks

Short post and I don't have the math background to go into this more rigorously, but just going based off intuition.

Media platforms that create a single giant graph with tons of nodes and edges give rise to emergent dynamics that humans did not evolve to deal with. The worst voices get elevated, polarization increases, and we get phenomena like memes and virality. I think these create misaligned incentivizes.

Humans do best in strongly connected networks of up to a few hundred. The networks are then loosely connected to each other. There is high trust within each cluster because everyone knows everyone, which significantly reduces the risk of polarization, and also moderates the most extreme voices and egregious behaviors. Members of different clusters can interact with other clusters, but not arbitrarily: there is a time and protocol for doing so.

This is how tribes, villages, and towns interacted with each other for thousands of years - not one massive public square. I think a digital network organized like this would be much healthier than modern social networks.
# My personal productivity tool

Hey everyone, 1st year radiology resident here! I'm passionate about web development and since medicine has taken most of my free time over the past years, I haven't been able to train my coding muscles that much, and I'm glad vibe coding exists nowadays that lets us make our ideas a reality way faster than it would otherwise take. Anyway, long story short, ever since I was in med school, I always wished there was a tool to log my diagnostic guesses (e.g., “70% this is pneumonia”), track outcomes, and actually measure whether my confidence matched reality. Nowadays there are some tools that let you do that, like fatebook and metaculus, but I believe I've made a good alternative to these that lets you see all your personal stats and accuracy trend graphs, community predictions, reactions -- everything on one page. It's called [Guessometer](https://guessometer.com), and lets you create a prediction card with a title and description, confidence level, target date, category. Once the result of the prediction becomes known, you mark it as Correct or Incorrect.

1.  Log a prediction with a probability (X% confidence) and an outcome date. Mark it as "Correct" or "Incorrect" when the result becomes known.
2.  2\. Once the outcome is known, the website compares your accuracy over time (actual outcome) with your overall confidence level (your initial X% confidence).
3.  Automatic scoring with Brier score and trend graphs so you can watch calibration improve.
4.  Leaderboard that ranks you based on accuracy and Brier score.  
     
5.  Keep predictions private or make them public.  
     
    
    If you’re curious, try it out and let me know what you think, especially what would make it helpful for aspiring superforecasters.
    

Sorry for the shameless plug, I honestly hope it's useful to at least some of you and it's completely free. I'm just really excited and wanted your opinion on how I might improve it.

Thanks for your time!
# LessWrong is migrating hosting providers (report bugs!)

LessWrong is currently in the process of migrating from AWS to Vercel, as part of a project to migrate our codebase to NextJS[^z86f281wae].  This post should go live shortly after we cut over traffic to the new host (and updated codebase).  This should hopefully be a pretty low-risk operation.  If all goes well, we plan on doing the DNS cutover next week, which is a higher-risk[^cyjxxai2r1] operation.  (If we notice something terribly wrong we might roll back without warning.)

This is all to say that there's a higher-than-usual chance that you'll run into some new bugs, bumpiness, or performance degradation[^lbfxo7bjys] in the next few days.  If you do, **please report them to us**.  The best way to do this via Intercom.  You should[^5j33huwiik5] see this icon in the bottom-right corner of your screen:

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/WSTB698bTdnK8Yrgn/cd8nrasz3i7iln91nmng)

You can also leave a comment on this post.

Bug reports ideally include "when did you first encouter the bug" and "what were you doing at the time", but don't worry about writing an essay, if we have questions we'll just ask.

You should feel free to report pre-existing bugs, just please let us know if you're reporting a bug which you're pretty sure was around before today.

[^z86f281wae]: "Why" is a longer story that we might get into in the comments if people are curious. 

[^cyjxxai2r1]: If something goes wrong here, some fraction of users might see downtime on the order of an hour, if they happen to be behind some piece of infrastructure which doesn't honor short DNS TTLs.  (We've ensured that the DNS TTL is set to 60 seconds a few days ahead of time.) 

[^lbfxo7bjys]: I expect a moderate performance hit for e.g. initially loading the front page on a new tab, maybe on the order of 20-30%, at least until we get around to optimizing that harder.  If you're on a fast internet connection, this should hopefully not be very noticeable - our median front page load is under 1 second.  If you run into something that seems to take longer than it used to, in a way that's perceptible to you, please err on the side of letting us know.  It might end up being nothing, or might be too costly to fix in the short-term, but we can't perform triage if we don't know about the problems in the first place. 

[^5j33huwiik5]: Unless you've hidden it in your account settings.
# Aspiring Rationalist Ritual Compendium

*Posting this here in advance of Petrov Day.*

This is intended to be an archive of rationalist ritual material. We already have [Secular Solstice Resources](https://secularsolstice.github.io/) for such material related to Winter Solstice; this compendium is for Summer Solstice, Petrov Day, and everything else.

If you have anything to add, please do send it in! You can email me or DM me on LessWrong, or send a GitHub pull request if you're comfortable with that.