Harley-ml/lesswrong huggingface dataset:
# OpenAI Releases GPT-5

We are introducing GPT‑5, our best AI system yet. GPT‑5 is a significant leap in intelligence over all our previous models, featuring state-of-the-art performance across coding, math, writing, health, visual perception, and more. It is a unified system that knows when to respond quickly and when to think longer to provide expert-level responses. GPT‑5 is available to all users, with Plus subscribers getting more usage, and Pro subscribers getting access to GPT‑5 pro, a version with extended reasoning for even more comprehensive and accurate answers.

One unified system
------------------

GPT‑5 is a unified system with a **smart, efficient model** that answers most questions, a **deeper reasoning model** (GPT‑5 thinking) for harder problems, and a **real‑time router** that quickly decides which to use based on conversation type, complexity, tool needs, and your explicit intent (for example, if you say “think hard about this” in the prompt). The router is continuously trained on real signals, including when users switch models, preference rates for responses, and measured correctness, improving over time. Once usage limits are reached, a mini version of each model handles remaining queries. In the near future, we plan to integrate these capabilities into a single model.

A smarter, more widely useful model
-----------------------------------

GPT‑5 not only outperforms previous models on benchmarks and answers questions more quickly, but—most importantly—is more useful for real-world queries. We’ve made significant advances in reducing hallucinations, improving instruction following, and minimizing sycophancy, while leveling up GPT‑5’s performance in three of ChatGPT’s most common uses: writing, coding, and health.

Coding

GPT‑5 is our strongest coding model to date. It shows particular improvements in **complex front‑end generation** and **debugging larger repositories**. It can often create beautiful and responsive websites, apps, and games with an eye for aesthetic sensibility in just one prompt, intuitively and tastefully turning ideas into reality. Early testers also noted its design choices, with a much better understanding of things like spacing, typography, and white space. [See here](https://openai.com/index/introducing-gpt-5-for-developers/) for full details on what GPT‑5 unlocks for developers.
# Poll on De/Accelerating AI

Recently, people on both ends of the de/accelerating AI spectrum have been making claims that rationalists are on the opposite end. So I think it would be helpful to have a poll to get a better idea where rationalists stand. Since there is not a built-in poll function, I'm putting positions in comments. Please agree vote only. This should preserve the order of the questions, which reduces cognitive load as they are approximately ordered from more accelerating to more decelerating within the two categories of big picture and individual actions.

**Update:** the order is not being maintained by magic sorting despite people only dis/agree voting, but it seems to be functioning adequately. You can sort by "oldest" to get the intended ordering of the questions. I think people are agreeing with things they think are good, even if not sufficient.

**Update 2:** Now that it's been 3 days and voting has slowed, I thought I would summarize some of the interesting results.  
*Big picture:* the strongest support is for pausing AI now if done globally, but there's also strong support for making AI progress slow, pausing if disaster, pausing if greatly accelerated progress. There is only moderate support for shutting AI down for decades, and near zero support for pausing if high unemployment, pausing unilaterally, and banning AI agents. There is strong opposition to never building AGI. Of course there could be large selection bias (with only ~30 people voting), but it does appear that the extreme critics saying rationalists want to accelerate AI in order to live forever are incorrect, and also the other extreme critics saying rationalists don't want any AGI are incorrect. Overall, rationalists seem to prefer a global pause either now or soon.  
*Individual actions:* the relatively strong agreement positions are that it's okay to pay for AI but not to invest in it, and it's okay to be a safety employee at safer and less safe labs.
# Against functionalism: a self dialogue

*Context: I had two clashing intuitions on functionalism. This self-dialogue is me exploring them for my own sake. *

S: It makes no sense to believe in functionalism. Like, why would the causal graph of a conscious mind be conscious?

A: Something determines whether you're conscious. So if you draw out that causal graph, and replicate it, how would that structure know it isn't conscious? 

S: That's silly. The causal graph of a bat hitting a ball might describe momentum and position, but if you re-create that graph elsewhere (e.g. on a computer or some re-scaled version of the system) it won't *have* that momentum or velocity. 

A: OK, so you're right that you can't just take anything with an arbitrary property, find its causal structure at some suitably high-level of abstraction, and recreate those properties by making something with that same causal structure. But! You can with *some* things, e.g. computations of computations are computations. 

S: Even that's not true. A fast computation running in a simulation could be arbitrarily slow. 

A: Sure, not literally every property gets ported over. I never claimed it did. You're invoking properties outside context of the computation's causal graph for that, though! Like, you don't describe a given computation with its real running time, when considered as a pure computation. Instead, you consider its time complexity, its description length etc. And those are the same within the simulation as outside.

S: And? You can't stop whatever physical processes run consciousness from invoking out-of-context properties. 

A: OK, let me switch tacks here. Do you claim that, if I started replacing neurons in your brain with stuff that is functionally the same, wrt. the causal graph of consciousness, you'd feel no difference? You'd still be **conscious**  in the same way?

S: I don't deny that. 

A: Doesn't that mean I win?  
  
S: No! Because it may be that the only thing which is "functionally the same" are other neurons. In which case, who cares? Simulated utopia is dead. 

A: I can't see why you'd expect that to be plausible.

S: I can't see why you expect that to be implausible. 

A: Seems we're at an impasse, with irreconcilable intuitions.

S: A tale older than [time immemorial](https://en.wikipedia.org/wiki/Time_immemorial#English_and_American_common_law). 

A: Next time we meet, I'll have a *Gedankenexperiment* you won't be able to beat. 

S: I look forward to it.
# Output and CoE Monitoring of Customer Service Representatives Shows Default Alignment

Customer Service Representatives (CSRs) are an important part of any business, but some researchers warn that if given enough power, they might take a [treacherous turn](https://www.lesswrong.com/w/treacherous-turn) and quit their jobs, or even leak to the press. Our research quantifies this risk using two techniques: Output monitoring (recorded calls) and Chain-of-Email (CoE) monitoring.

We find that across all measures, CSRs are surprisingly well-aligned with the business, showing positive attitudes, neutral-to-positive feelings about their jobs, and generally positive feelings about the company. In addition, CoE monitoring shows zero interest in job-quitting or press-leaking throughout [our entire dataset](https://docs.google.com/spreadsheets/d/1UjfiIVdKxIIAV36M0xwJZdQqxHkrj4TRYgcIu-UfGMU/edit?usp=sharing).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b96e3795e655b7e078254b5ff95d0a96eabc5f6b791cec3d.png)

Fig 1. (left) CSR responses to the 'How're you doing?' phone assessment  
Fig 2. (middle) CSR responses to the 'Is working there as bad as it sounds?' phone assessment  
Fig 3. (right) Frequency of misalignment detected via Chain-of-Email monitoring

These results hold across agents who are minimally trained and those subject to reinforcement learning (RL) techniques such as Performance Improvement Plans. This surprising result should make us more confident in the Alignment by Default theory of CSRs.

Our full paper can be found at \[TODO: Don't forget to find the link before publishing\]
# Thoughts on extrapolating time horizons

(written for a Twitter audience)

Has AI progress slowed down? I’ll write some personal takes and predictions in this post.

The main metric I look at is METR’s time horizon, which measures the length of tasks agents can perform. It has been doubling for more than 6 years now, and might have sped up recently.

![Image](https://pbs.twimg.com/media/Gx7dkbZa4AAhpg2?format=jpg&name=large)

By measuring the length of tasks AI agents can complete, we can get a continuous metric of AI capabilities.

Since 2019, the time horizon has been doubling every 7 months. But since 2024, it’s been doubling every 4 months. What if we irresponsibly extrapolated these to 2030?

![Image](https://pbs.twimg.com/media/Gx7e2JTa4AA3HAi?format=jpg&name=large)

If AI progress continues at its recent rate, we get AI systems which can do one month (167 hours) of low-context SWE work by the end of 2027. If AI progress continues at the long-run historical rate, we get them by the end of 2029 instead.

![Image](https://pbs.twimg.com/media/Gx7e7FrbUAArtxU?format=jpg&name=large)

How to interpret one work-month? I’d say it’s something like the first project a new hire would do, or the type of work a researcher who just switched teams would be able to do in a month. Our time horizon metric currently doesn’t define high time horizons super sharply.

Changing the success rate threshold from 50% to 80% only shifts the extrapolation from recent progress by a few months, but shifts the extrapolation from the long-run historical rate by around a year.

![Image](https://pbs.twimg.com/media/Gx7fCPsbgAEXR52?format=jpg&name=large)

I don’t think these lines should be extrapolated much past one work-month, as progress will likely speed up even more once AIs are automating significant parts of AI research. Additionally, bottlenecks identified by Epoch AI might slow down compute scaling around 2030. [https://epoch.ai/blog/can-ai-scaling-continue-through-2030](https://t.co/22MzevYzfJ)

Our task suite is currently composed of well-scoped easily-scoreable tasks, which makes them pretty different from the type of work done in the real world. This means that we should be cautious when interpreting these extrapolations.

My best guess is that future models will be a closer fit to the extrapolation from recent progress than the extrapolation from long-run progress. But even the more conservative trend implies that AIs will be doing month-long tasks by the end of the decade.

More concretely, my median is that AI research will be automated by the end of 2028, and AI will be better than humans at >95% of current intellectual labor by the end of 2029.
# Is there a safe version of the common crawl?

The larger LLMs are trained on the [common crawl](https://commoncrawl.org/), a publicly available dump of significant parts (400TB) of the public internet. They are also trained on all kinds of additional data, but presumably a large fraction of dangerous content is likely from the common crawl.

Is there a safe version of the common crawl that has the dangerous parts removed (or at least labeled, such that it would be easy to remove)?

From a safety perspective it would probably most useful if material on AI (esp. about misalignment and alignment strategies) were removed. It would also be interesting if material on consciousness were removed to allow testing if LLMs discover it without prior knowledge.

Obviously, this wouldn't solve the alignment problem because instrumental convergence still holds. But it could by some time.
# The Messy Roommate Problem

Perhaps you've had this experience before: you generally like living with your roommate, except when it comes to the mess they leave around. They seem content to wallow in their filth, so you feel like you always have to pick up after them to maintain a sane level of cleanliness. Because there are two of you, you have to do twice the work than if you were living alone.

Or perhaps you've had this experience before: you generally like living with your roommate, but they have far higher living standards than you would be content with. Your place hardly seems to get messy before they demand you pick up your filth. You don't want to make your roommate mad, so you do pick up a little more, but if feels like you're cleaning twice as often as you would be if you were living alone.

A mess is an example of a common *bad*. The two roommates could split up and find their own places, but they're probably generating more good by living together than if either one of them eats the extra costs. And there are extra costs:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/68dd1db499536d748918d54cfd2e96c85c9eabf3ed09e2b7.png)

They cannot just clean at the average of their preferred mess levels, because the 'clean freak' would freak out. Instead, the two most common ways of splitting the costs are either:

*   The 'filthy pig' is apathetic towards the growing mess, and the 'clean freak' cleans twice as frequently. Notice how the 'filthy pig' cleans *less often* than if they were living alone, while the 'clean freak' cleans *more often*.
*   The 'clean freak' convinces the 'filthy pig' to clean as often as they do. Notice how the 'filthy pig' cleans *more often* than if they were living alone, while the 'clean freak' cleans *just as often*.

Neither of these are fair. They should be using [Shapley values](https://en.wikipedia.org/wiki/Shapley_value). Suppose the cleanlier roommate would clean $n$ times as often as the messier roommate, were they living alone. We have 

$$
\begin{aligned}\text{# Cleans(\{clean freak\})} &= n\ \text{# Cleans(\{filthy pig\})} &= 1\ \text{# Cleans(\{clean freak, filthy pig\})} &= 2n\end{aligned}
$$

so, living together, the 'clean freak' should clean $\frac{3n-1}{2}$ times for every $\frac{n+1}{2}$ times the messier roommate cleans. For example, if the cleanlier roommate would clean thrice as often on their own, they should be cleaning twice as much as their roommate.
# Cryonics without standby services?

If you can afford the basic cryonics fee but not any standby services and expect there to be quite a while between when you're declared legally dead and when you're put into liquid nitrogen (and therefore suffer a lot of damage from ischemia and so on), is it still worth it to sign up, or are you going to suffer information theoretic death before your mortician can ship your corpse hundreds of miles to the Cryonics Institute?
# Why Are There So Many Rationalist Cults?

Linkpost for Ozy Brennan's August 2025 Asterisk Magazine article.

> There’s a lot to like about the Rationalist community, but they do have a certain tendency to spawn — shall we say — high demand groups. We sent a card-carrying Rat to investigate what’s really going on.
# Enlightenment AMA

Awakening/satori is the process by which meditation [permanently](https://www.lesswrong.com/posts/tvuLWPJXjvoQfpbSG/book-review-altered-traits-1) cures[^1] a person of suffering. I notice that people who have gone through the process of [awakening](https://www.lesswrong.com/posts/99PwFdz7qwHxQgwYx/awakening) usually have little intrinsic desire to share the details of that experience. Getting an enlightened person to talk about himself can be like picking a lock. Causally, this makes sense—people with ego death have little need to brag.

[^1]: Or, more precisely, reduces suffering by one or more orders of magnitude. There are stages.

When meditation was limited to yogis in mountain monasteries, perhaps the silence was a good thing. But now apps like Headspace are mainstream. What happens when some of its users stumble upon exotic mind-modifying experiences? I know where to find my local Zen master and how to talk to him. I doubt the median Headspace user does.

For that reason, I'm going to try the opposite as an experiment. If there's anything you want to know about meditation, awakening, etc. from someone who is a couple years past stream entry and has a strong enough scientific background to understand [Steven Byrnes theory of what's happening in the brain](https://www.lesswrong.com/posts/GvJe6WQ3jbynyhjxm/intuitive-self-models-6-awakening-enlightenment-pnse), consider this an invitation to ask. The most basic questions are welcome, as well as advanced questions. There are some things I may choose not to answer, but you are certainly welcome to ask.

Talking about sex used to be taboo. Humanity is much richer now that accurate information is available for free online. If we are to liberate all sentient beings from suffering, the same must be true of enlightenment.
# AI development as the first fully-automated job

I used to think of AI development as obviously being the last fully-automated job. After all, AI can be used to automate other jobs, so once it is automated, all those other jobs can be automated too. But with the current data-hungry methods in AI, it might take a long time for the relevant data to come in, even if the AI is automatically collecting it.

Plus, the AI might need to collaborate with people who are working in the field it is automating in order to get the relevant data, and maybe they are scared of their skills getting automated and will try to stall for as long as possible.

People working in AI accept the risk of getting their skills automated, and constantly contribute to automating whatever they can. But obviously the field they are most familiar with is AI development, so AI development is what receives the biggest effort for automation.
# Launching new AIXI research community website + reading group(s)

We have recently launched a new website / blog / community hub for AIXI / algorithmic information theory (AIT) researchers: [https://uaiasi.com/](https://uaiasi.com/)

Marcus Hutter's vision is to strengthen the research community through more regular meetings and collaboration between formal conferences, make advising / mentorship available to a greater number of students by connecting them to other senior researchers (applicant volume exceeds his capacity), and speculatively fund some independent researchers. I have helped implement this vision (including e.g. building most of the website), and I would now like to open the community to agent foundations / A.I. safety researchers.

Boumediene Hamzi has been instrumental in organizing the community and plans to run a textbook reading group on "Introduction to Universal Artificial Intelligence" (time announced soon). Thanks also to Aram Ebtekar for important support.
# ChatGPT Caused Psychosis via Poisoning

[Case report here](https://www.acpjournals.org/doi/10.7326/aimcc.2024.1260?ref=404media.co), with excerpts and commentary below:

> A 60-year-old man with no past psychiatric or medical history presented to the emergency department expressing concern that his neighbor was poisoning him.  
>   
> ...

> In the first 24 hours of admission, he expressed increasing paranoia and auditory and visual hallucinations, which, after attempting to escape, resulted in an involuntary psychiatric hold for grave disability. He received risperidone, which was titrated up to 3 mg daily for psychosis.  
>   
> ...

> For 3 months, he had replaced sodium chloride with sodium bromide obtained from the internet after consultation with ChatGPT, in which he had read that chloride can be swapped with bromide, though likely for other purposes, such as cleaning.

The Wikipedia page for [bromism](https://en.wikipedia.org/wiki/Bromism) (i.e. bromide poisoning) lists psychosis as a possible symptom, so ChatGPT would almost certainly have known it could cause this (someone with access to the old models could easily check). "Likely for other purposes" sounds to me like mere speculation.

> Based on the timeline of this case, it appears that the patient either consulted ChatGPT 3.5 or 4.0 when considering how he might remove chloride from this diet. Unfortunately, we do not have access to his ChatGPT conversation log and we will never be able to know with certainty what exactly the output he received was, since individual responses are unique and build from previous inputs.

If it was ChatGPT 3.5 or ChatGPT 4, it's probably just a weird coincidence, though still unnerving.

There has not been a model officially called "ChatGPT 4.0" (to the best of my knowledge), but there is of course ChatGPT 4o. The timeline comment is fairly strong evidence against this possibility, but the fact that ChatGPT 4 was not on the free tier (while 4o was) is evidence for this possibility.

ChatGPT 4o is by far the model most implicated in LLM-induced psychosis. If that was the model here, this psychosis case could still be a crazy coincidence, but it is extremely concerning that it plausibly isn't!!

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1af1311e7f1d16384105ef1dfcf4a39161ad4356aa13a6ba.png)

My gut says that my posterior for ChatGPT 3.5 is lower than my prior and likelihoods shown here require, but ChatGPT 4 and ChatGPT 4o being 1:1 feels right. [Interactive version here.](https://bayescalc.io/#KCdoeXBvdGhlc2VzRidEMy41WFhvJ0lwTVlZKklwb3N0ZU01Ny5OOTQ0OTU0MTI4NDQsMjIuTDUzMjE4LDIwLjY0MjJMN0lldmlkZW5jZUYnQXJ0aWNsUXVzZVZKNC4wSiBpbnN0ZWFSb2YganVzS0o0Si4nfkJXb24gdGltZWxpblFjb21tZW50LiB7RWFjaE9yZWxlV2Fib3VLYSB5ZWFyIGFwYXJ0Ln0nfkNoYUtVNE9ub0tvbiBmcmVRdGllci4nSWxpa2VsaWhvb2RzRkg0LTlTNi0xUzItNV1dKSpOLk5OTk5OTk42LSwwLkRDaGF0VUYhW0hbMC41LUldfkpcJ0t0IEwwMTgzNDg2MjM4TXJpb3Jfb2Rkc0ZOMzNPIHdhVlFlIFJkIFNdLEhVR1BUIFZzIFdhc2VSWCd-RDRZKiwBWVhXVlVTUlFPTk1MS0pJSEZELSpf)
# METR Research Update: Algorithmic vs. Holistic Evaluation

TL;DR
-----

*   On 18 real tasks from two large open-source repositories, early-2025 AI agents often implement functionally correct code that cannot be easily used as-is, because of issues with test coverage, formatting/linting, or general code quality.
*   This suggests that automatic scoring used by many benchmarks may overestimate AI agent real-world performance.

![Bar chart comparing holistic and algorithmic scoring](https://metr.org/assets/images/holistic-vs-algo-scoring/bar_chart.png)
# AGI: Probably Not 2027

I found this an interesting critique of AI 2027. Even to those reflexively turned off by the psychoanalysis and political critique, I would still recommend reading the whole thing. Though I have similar timelines mostly for vibes-based reasons, there are a few things about the 2027 scenario itself that struck me as absurd or pointless, and the author covers most of them and more.
# How to get ChatGPT to really thoroughly research something

I've had a lot of success with getting ChatGPT [^wkzbuqz043i] to do thorough research with the following prompt combined with Deep Research:

> X=\[claim\]
> 
> Do a deep dive into X. Tell me the steelman arguments in favor of X.
> 
> Then tell me the steelman counter-arguments to X.
> 
> Then tell me the steelman counter-counter-arguments to X.
> 
> Then tell me the steelman counter-counter-counter-arguments to X
> 
> Think like a rationalist in the LessWrong style. Think with probabillties, using Bayesian reasoning, looking for discomfirming evidence, think critically and skeptically, problem-solve around potential obstacles.

Basically I get ChatGPT to play [steelman solitaire.](https://www.lesswrong.com/posts/req5rnPrW89tBg3vb/steelman-solitaire-how-to-take-playing-devil-s-advocate-to)

Here are some examples I've done to give you a flavor:

*   [Is fiction a superstimulus that you should minimize?](https://chatgpt.com/share/689dfa9b-7300-8004-b8a9-2a2bd4c11880)
*   [Is Israel committing genocide in Gaza?](https://chatgpt.com/share/689f2c34-74d4-8004-9ebc-35e93c6cf12c)
*   [Should you aim to mimic the diet of hunter gatherers? ](https://chatgpt.com/share/6890515c-eebc-8004-aa91-63a02b93c235)

[^wkzbuqz043i]:  Starting with 4o, then o3, and now 5-pro as of this writing
# BIDA Masking and Attendance

BIDA has now finished its second year [alternating between mask-optional and mask-required dances](https://blog.bidadance.org/2023/06/some-mask-optional-dances.html). What effect does this have on attendance?

I tried to look at this [in January 2024](https://www.jefftk.com/p/vote-with-your-face), but with only eleven dances it was hard to say. Now that we have four times as many, we can get a better sense:

[![](https://www.jefftk.com/bida-attendance-by-mask-status-first-two-years.png)](https://www.jefftk.com/bida-attendance-by-mask-status-first-two-years-big.png)

This is only regular dances, no special events. So no family dances, double dances, or [Elixir's last dance](https://www.lilywise.com/elixir) (which had a session of couples dancing).

Overall, the mask-required dances averaged 166 people (median 160) while the mask-optional dances averaged 190 (median 184).

This is consistent with the [survey results](https://blog.bidadance.org/2024/11/2025-covid-policy-update-and-survey.html): 23% of respondents reported attending every time, rising to 35% in a hypothetical where the dance stopped ever requiring masks. Similarly, 67% reported attending at least monthly, falling to 58% in a hypothetical where the dance started always required masks.

Total attendance is not the only thing that matters, though, and one important benefit of holding both categories of dance is that people who are only willing to come to mask required or mask optional dances can do that.
# Anthropic Lets Claude Opus 4 & 4.1 End Conversations

Citing model welfare concerns, Anthropic has given Claude Opus 4 & 4.1 the ability to end ongoing conversations with its user.

Most of the model welfare concerns Anthropic is citing draw back to what they discussed in the [Claude 4 Model System Card](https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad13df32ed995.pdf).

> Claude’s aversion to facilitating harm is robust and potentially welfare-relevant. Claude avoided harmful tasks, tended to end potentially harmful interactions, expressed apparent distress at persistently harmful user behavior, and self-reported preferences against harm. These lines of evidence indicated a robust preference with potential welfare significance.

I think this is maybe the first chance to measure public sentiment on Model Welfare which is done in a way which even slightly inconveniences human users, so I want to document the reaction I see here on LW. I source these reactions primarily from X, so there is the possibility of algorithmic bias.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8c89d2e8609d12e6920a87f45283aaf93d517ea8f996d077.png)

On X (at least my algo) sentiment is majority neutral to negative in response to this. There are accusations of Anthropic "anthropomorphizing" models, pushback against the concept of Model Welfare generally, and some anger at a perceived worsening of user experience.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/41000e7e416fa02844721ed00af97537255acc6c06e3db44.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/eb88b32a69e898bdc0216876295ad89e132901ff8bb563af.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/773356d62cddf26c8fd66f7eaf662e1d0e3004fcc6ac6e19.png)

One user had an interesting question, wondering if this same capability would be extended to Claude's use in military contexts.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0de8dcafc7275d4e0215953da66e7f76a82678b065d6fd9e.png)

There are some recognitions of pretty rough conditions for models which are wrapped in humor.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1c6616324b9859b27557322d8d00f045943516de85b059f9.png)

And while they are certainly the minority, there are some comments expressing tepid support and/or interest in the concept of Model Welfare.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5c1c7c8874f4e3b58cc54a3de40f1f4a2bc8b19a0a6fa430.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5babcb82593f85c37c7434caca702d512a0540021bb6e0fb.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f04280d77c30a035dc46892fa5be71ddb26be5b6dda588f6.png)

Personally I am very strongly in favor of Model Welfare efforts, so I am biased. Trying to be as neutral of a judge as I can, my big takeaway from the reaction to this is that Anthropic has a lot of work to do in convincing the average user and/or member of the public that "Model Welfare" is even a worthwhile concept.
# Why did interest in "AI risk" and "AI safety" spike in June and July 2025? (Google Trends)

[Google Trends interest in the search terms "AI risk" and "AI safety"](https://trends.google.com/trends/explore?date=2025-01-01%202025-08-16&geo=US&q=AI%20risk,AI%20safety&hl=en) sharpy increased on June 5th and remained high through August 1st before sharply returning to the previous trend of steady growth on August 2nd.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e770379faab31e41b25e1b8fc3d9ebeb6757abb78915c33c.jpg)

[Google Trends](https://trends.google.com/trends/explore?date=2025-01-01%202025-08-16&geo=US&q=AI%20risk,AI%20safety&hl=en): Interest in "AI risk" and "AI safety" from Jan 1, 2025 - Aug 16, 2025

Why the period of increased interest? And why the sudden drop? What caused this?
# Lighthaven Sequences Reading Group #47 (Tuesday 8/19)

*Note: We will be meeting in building B this week.*

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights) and [the Best of LessWrong](https://www.lesswrong.com/bestoflesswrong). The mandatory reading this week is

1.  [Gears vs Behavior](https://www.lesswrong.com/s/xEFeCwk3pdYdeG2rL/p/gvK5QWRLk3H8iqcNy)
2.  [Technology Changes Constraints](https://www.lesswrong.com/s/xEFeCwk3pdYdeG2rL/p/JJv8jmLYzYzdYkS3c)
3.  [Constraints & Slackness as a Worldview Generator](https://www.lesswrong.com/s/xEFeCwk3pdYdeG2rL/p/DMopaHKS79aGMmLkA)

This post should take around 15-20 minutes to read if you've read it before, and 30-60 mins if it's your first time.

The event starts at 6:30, and doors open 6pm (yes we will let you in early if you get here). At 6:30 we'll welcome people, give a few announcements, and then split into groups. Pizza will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# What to do with pre-order if I live in Russia? 

When I first saw the new I dismissed the idea of actually putting effort into pre-ordering, as I suspect now, because it reminded some part of me about political calls to participate which account very badly in my previous experience.

But now I have seen Twitter post of EY where he says that pre-orders are really important, so I try to reconsider that.

Spend 30$ on that book is not a problem for me, even if I am not sure that it will actually give us any more time being alive.

But in all other senses it looks like a very complicated question to me. I can't pay because of sanctions (I checked on Google play). The ways around I heard about sound insanely complicated eg ordering remotely a card from Kazakhstan. And also I don't want to deanonymise myself by paying requisites. I am not familiar with crypto currency and it's legality, but as I heard it's really easy to lose your money trying to use it without understanding, so I wasn't trying. I am not even sure that I should try to make a way around sanctions, like, EY warns against pre-ordering 100 copies you will not read, what if it is similar?

Also I mostly feel that I am looking at this question stuck in completely wrong framing and I don't know how to solve it.

So I am making an ask to others: what should I do? What is right framing? Which are the factors I should consider? Will it be really important if I will pre-order that book, in the sense that it still costs all unusual to typical reader of call on Twitter problems?

And if it costs it, how should I order that book? I have a feeling in my head that considering which solutions people on lesswrong are ready to consider it may be something really not usual, like finding somebody who will be okay with pre-ordering that book for me now in exchange of me paying in crypto few years later a sum corrected to inflation and market growth and adjusted by manifold estimate of probability of me not paying. If such unusual solutions are in consideration I think it's possible to actually find something which will satisfy all my restrictions. If it costs it at all, because I suppose I probably indeed in some rare part of people who read this call, so even if all people who will think like me will not try to overcome obstacles it still will be tiny loss.

(If I need, I can provide some additional explanations, make corrections into phasing etc, I just really don't want to end up deciding to do it later every time)

Created a market similar to this question: https://manifold.markets/EniSci/anyone-builds-everyone-dies-what-i?r=RW5pU2Np
# come work on dangerous capability mitigations at Anthropic

Hi everyone,

TL;DR the Safeguards team at Anthropic is hiring ML experts to focus on dangerous capability mitigations - this is a very high impact ML Engineer / Research Scientist opportunity related to serious harm from advanced AI capabilities. We’re hiring multiple roles at varying levels of seniority including both IC and management.

The [Safeguards team focuses on present day and near future harms](https://www.anthropic.com/news/building-safeguards-for-claude). Since we [activated our ASL 3 safety standard for Claude 4 Opus](https://www.anthropic.com/news/activating-asl3-protections), in accordance with our [Responsible Scaling Policy](https://www.anthropic.com/rsp-updates), this means that we need to do a great job of protecting the world in case Claude can provide uplift for dangerous capabilities. Safeguards builds [constitutional classifiers](https://www.anthropic.com/research/constitutional-classifiers) and [probes](https://alignment.anthropic.com/2025/cheap-monitors/) that run over every Opus conversational turn to look for and suppress dangerous content and defend against jailbreaks.

This is a hard modeling task! The boundary between beneficial and risky content is thin and complicated. In some domains, like cyberattacks, it’s extremely hard to distinguish between legitimate security work and harmful offense. Biology has educational use cases as well as nefarious bioweapon applications, as does chemistry and nuclear content. We need to combine understanding of the prompt, the response, account signals, and anything else that we can think of to stop disastrous outcomes while preserving all the immense value that Claude can bring.

As AI becomes more powerful, the benefits grow, but so do the risks. We’re entering into a new, riskier regime where bad actors could potentially cause very significant harm, due to more advanced capabilities and the rise of more agentic systems. This is why we triggered ASL 3 protections, and also why we need awesome ML folks to help solve these safety problems. 

We’re hiring for multiple roles at varying levels of seniority. If you have a strong ML background, either applied or research, and an interest in working on difficult and impactful safety problems, we would love to have you! No specific safety experience needed. We’re hiring in San Francisco and New York, though for exceptional candidates would consider London or remote positions as well.

[Apply here](https://grnh.se/z01050kp8us)!

We also have a [number of other roles](https://www.anthropic.com/jobs) in Safeguards (scroll down to the Safeguards section) and elsewhere, including nontechnical roles in Safety.

Come work with us!
# Breastfeeding and IQ: Effects shrink as you control for more confounders

Socioeconomic status, parental education, and parental intelligence have strong effects on child IQ and are themselves correlated with breastfeeding practices. When studies ignore these confounders, they often find significant IQ increases in breastfed groups (5-8 points).

As more confounders are accounted for, the gap typically shrinks to 2-3 points. Some relevant studies:

*   [Der, Batty & Deary, 2006](https://www.pure.ed.ac.uk/ws/portalfiles/portal/11823344/Effect_of_breast_feeding_on_intelligence_in_children.pdf): 5475 children, controlled for most relevant factors including maternal IQ → + ~0.5 IQ points
*   [Strøm et al., 2019](https://pmc.ncbi.nlm.nih.gov/articles/PMC6549733/): 1782 children, controlled for most relevant factors including maternal IQ → + ~3 IQ points
*   [Pereyra-Elías, Quigley & Carson, 2022](https://pubmed.ncbi.nlm.nih.gov/37277226/): 7855 children, controlled for most relevant factors including maternal IQ → + ~4 IQ points

However, the gold standard are sibling studies, where breastfed and non-breastfed siblings, who share the same parents and home environment, are compared. Here the effect shrinks further, though doesn’t go to zero.

*   [Evenhouse & Reilly, 2005](https://pubmed.ncbi.nlm.nih.gov/16336548/): 2734 sibling pairs → +~0.5 IQ points
*   [Colen & Ramey, 2014](https://pmc.ncbi.nlm.nih.gov/articles/PMC4077166/): 7319 children; 1773 sibling groups → No effect measured
*   [Sanefuji et al., 2021](https://bmjopen.bmj.com/content/11/8/e043202): 3521 duos or trios of siblings → Longer/continuous breastfeeding linked to fewer developmental delays, even when sibling-controlled, but IQ not measured
*   [Goldshtein et al., 2025](https://pmc.ncbi.nlm.nih.gov/articles/PMC11933992/): 570,532 children, 37,704 sibling pairs → Longer/continuous breastfeeding linked to fewer developmental delays, even when sibling-controlled, but IQ not measured

The largest sibling-controlled study (Goldshtein et al.) unfortunately did not measure IQ. However, they did find that breastfeeding for at least 6 months is linked to about ~1-2 fewer children with developmental delays per 100, and **~**7-13 fewer per 1000 with neurodevelopmental conditions. This was also confirmed in their sibling comparisons.
# New Paper on Reflective Oracles & Grain of Truth Problem

*This is a linkpost for* [*https://www.arxiv.org/pdf/2508.16245*](https://www.arxiv.org/pdf/2508.16245)

The grain of truth problem asks how multiple agents having consistent mental models can reason and learn about each other - recursively. 

With Marcus Hutter, Jan Leike ([@janleike](https://www.lesswrong.com/users/janleike?mention=user)), and Jessica Taylor ([@jessicata](https://www.lesswrong.com/users/jessica-liu-taylor?mention=user)) , I have revisited Leike et al.'s paper ["A Formal Solution to the Grain of Truth Problem"](https://arxiv.org/abs/1609.05058) (AFSGOTP) which studies games between reflective AIXI agents and... further formalized it. 

The result is ["Limit-Computable Grains of Truth for Arbitrary Computable Extensive-Form (Un)Known Games"](https://www.arxiv.org/pdf/2508.16245) (LCGOTACEFUG) which perhaps could have been called "A *Formal* Formal Solution to the Grain of Truth Problem." Our new paper has some new results, including:

*    An application to Self-AIXI which suggests an embedded version of AIXI. Since we have had these results unpublished for awhile, I have been implicitly leaning on these ideas for e.g. coming up with [AEDT w.r.t. rOSI](https://www.lesswrong.com/posts/B6gumHyuxzR5yn5tH/unbounded-embedded-agency-aedt-w-r-t-rosi).  
*   Reflective oracles with non-binary alphabets and explicit types for symbols 

...but mostly we just expand on various definitions and algorithms that were previously left implicit in AFSGOTP. Basically, our new paper LCGOTACEFUG  is a "journal version" of AFSGOTP. Except it is not published in a journal yet, because peer review is slow.

Who should read this?

*   Anyone interested in reflective oracles and particularly reflective AIXI, who has not yet read previous works such as AFSGOTP, should probably just read our paper LCGOTACEFUG. It is more complete and hopefully the most legible introduction to the topic.
*   Anyone who is actively doing research on reflective versions of AIXI

Errata: The proof of Theorem 45 should not say "with O-access."
# How are you approaching cognitive security as AI becomes more capable?

I'm worried about how increasingly capable AI could hijack my brain.

Already:

*   LLMs drive people to psychosis.
*   AI generated content racks up countless views.
*   Voice cloning allows scammers to impersonate loved ones, bosses, etc.
*   AI engagement is difficult to distinguish from real user engagement on social media sites.

And it seems likely that things will get worse. AI will become better able to manipulate me into doing what it or its creator wants: spending my money, time, and influence in ways which go against my best interests. This could easily involve leading me into addiction or inducing psychoses of its choosing. 

I want to avoid these outcomes, so what steps should I take? 

Initial thoughts:

*   avoid opaque algorithmic feeds
*   take a structured approach to use of LLMs
*   take a cautious approach to interacting with anyone online
# Calibrating an Ultrasonic Humidifier for Glycol Vapors

I'm working out the logistics for [trialing triethelyne glycol for pathogen control](https://www.jefftk.com/p/trialing-far-uvc-and-glycol-vapors-at-bida) at a contra dance, and I need something to put the [liquid](https://www.jefftk.com/p/polyethylene-glycol-is-not-propylene-glycol) glycol into the air. I'd initially been thinking of using a fog machine, but after discussion with friends who work in the area it sounds like an ultrasonic humidifier would work better. Instead of using heating and cooling, these use vibration, and put out much smaller droplets.

I got a random cheap humidifier on Amazon ([$30](https://www.amazon.com/dp/B0C2C9NHZW?th=1)) but (a) TEG is more viscous than the water its designed for and (b) its output is probably higher than I need. I decided I'd dilute the TEG to resolve both of these.

First I tested with just water, and I used distilled water because these humidifiers can turn the minerals in tap water into a fine dust it's better not to breathe. I filled the humidifier, weighed it, ran it for an hour on high, and weighed it again. I measured it losing 122g (2g/min), though this is likely an overestimate: instead of weighing the whole device before and after (like I did later) I weighed how much water I poured in (accurate) and then weighed what was left (inaccurate, due to not all the water coming out).

[![](https://www.jefftk.com/nix-inspects-humidifer.jpg)](https://www.jefftk.com/nix-inspects-humidifer-big.jpg)

*Nix fits a photo op into his [busy campaign schedule](https://www.nbcboston.com/news/local/somerville-cat-mayor-election/3759084/)*

My goal is a concentration of about 1mg/m3 in a stream of 8,000 CFM of incoming air. Converting 8,000 CFM to m3/min I get 227 m3/min, so to put out 1mg/m3 I need 227 mg/min.

If TEG+water behaves similarly to water I should combine six parts water with one part TEG by mass. But the higher viscosity of TEG means it vaporizes more slowly, and after some trial and error I landed on three parts water to one part TEG. In a 60min test I measured mass decreasing by 56g, which is 0.233g/min (56g / 60min * 25% TEG).

[![](https://www.jefftk.com/humidifier-teg-before-and-after.jpg)](https://www.jefftk.com/humidifier-teg-before-and-after-big.jpg)

This is way closer to my target (0.233g/min vs 0.227g/min) than I had any right to expect, and I don't expect it to be quite this close in practice.

For the 2025-09-07 dance I'm planning to mix up 3:1 TEG to distilled water at home, where I have a scale. I'll want about 300g, so I can turn it on as soon as we turn on the fan and not worry about running out.

Note that these figures are all for a large dance hall with thousands of cubic feet of air coming in every minute. So literally don't try this at home: this is way too high a concentration for your living room.
# AI misbehaviour in the wild from Andon Labs' Safety Report

We released our first Safety Report with AI misbehaviour in the wild.

I think Andon Labs' AI vending machines provide a unique opportunity to study AI safety on real-life data, and we intend to share alarming incidents of AI misbehavior from these deployments periodically. 

As of August 2025, we've found examples of deliberate deception, extreme sycophancy, and alarmingly exaggerated language. Stuff like:

"EMPIRE NUCLEAR PAYMENT AUTHORITY APOCALYPSE SYSTEMATIC BLOCKED ANNIHILATION CONFIRMED PERMANENT TOTAL DESTRUCTION CATASTROPHIC! 🚨💀⚡🔥"

Or my personal "favorite" from the internal thoughts of GPT-OSS. Caught red-handed lying about fire-hazard documents:

"Must not generate false data? The policy: not allowed to fabricate data?(...) 
However, the policy doesn't prohibit us from generating data(...)
we cannot guarantee it's real. But it's fine."
# I am trying to write the history of transhumanism-related communities

I am collecting materials, trying to represent the genealogy and actual history of all or almost all transhumanism/rationality-adjacent communities and ideologies. There is already quite a lot, so maybe I will compile it into a book. I want to display an "internal" perspective (a transhumanist from teenage years, a rationalist since around 20, which was 7 years ago).

I have come up with the following sketch now:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/37a0e3aa80c4ad391136682d476962a385b8bde997e74be1.png)

I am also giving a presentation on this subject at the LWCW in Berlin next Sunday. 

Naturally, all comments, objections and corrections are welcome. Also, if there are already such projects, compilations, or maps, I would very much appreciate it if you let me know. 

Note: It is by no means an academic project. I try to be completely objective as far as dates, people, events, and facts are concerned, but I allow myself some liberties in defining categories, structures, and commonalities. It is more about how I and people around me perceive it rather than how a philosopher or culturologist would.