Harley-ml/lesswrong huggingface dataset:
# AI Agent Benchmarks Are Broken

We find that many agentic benchmarks have issues with task setup or reward design, leading to under- or overestimation of agents' performance **by up to 100%**.

![https://substackcdn.com/image/fetch/$s_!es-x!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b36bada-a016-48fc-90f6-f50766a8c55c_1600x389.png](https://substackcdn.com/image/fetch/$s_!es-x!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F9b36bada-a016-48fc-90f6-f50766a8c55c_1600x389.png)

*Results of applying ABC on ten widely used AI agent benchmarks.*

To make agentic evaluation rigorous, we introduce the **A**gentic **B**enchmark **C**hecklist (**ABC**), a set of guidelines that we synthesized from our benchmark-building experience, a survey of best practices, and previously reported issues.

![https://substackcdn.com/image/fetch/$s_!33Iq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89022218-0a59-48f5-b648-b0ff39fffb70_1600x432.png](https://substackcdn.com/image/fetch/$s_!33Iq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F89022218-0a59-48f5-b648-b0ff39fffb70_1600x432.png)

*Operational and conceptual processes of AI agent evaluation. Task and outcome validity are essential to ensure that benchmark results truly reflect agents’ capabilities.*
# Subway Particle Levels Aren't That High

I recently read an [article](https://www.jefftk.com/p/masking-on-the-subway) where a blogger described their decision to start masking on the subway:

> I found that the subway and stations had the worst air quality of my whole day by far, over 1k ug/m3, ... I've now been masking for a week, and am planning to keep it up.

While subway air quality isn't great, it's also nowhere near as bad as reported: they are misreading their own graph. Here's where the claim of "1k ug/m3" (also, units of "1k ug"? Why not "1B pg"!) is coming from:

[![](https://www.jefftk.com/2022-05-17--commute-and-workday.png)](https://www.jefftk.com/2022-05-17--commute-and-workday-big.png)

They've used the right axis, for CO2 levels, to interpret the left-axis-denominated pm2.5 line. I could potentially excuse the error (dual axis plots are often misread, better to avoid) except it was their own decision to use a dual axis plot in the first place! Hat tip to [Evan](https://evanfields.net/) for pointing this out in the [substack comments](https://jefftkaufman.substack.com/p/masking-on-the-subway/comment/132387872).

The actual peak was only 75 ug/m3 and the subway time averaged below 50 ug/m3. While this isn't great, I don't think this is worth masking over.

Since I'm making a big deal about this error, though, I wanted to make sure I had my facts straight: I decided to replicate their work. I used the same meter they used (a Temtop M2000) and took the same subway journey (round trip from Davis station to Kendall station on the MBTA's Red Line). To be generous I took my time in the morning, intentionally missing a train at Davis and then lingering in Kendall after my train departed until the following train had also departed. Here's what I found:

[![](https://www.jefftk.com/mbta-red-line-commute-pm25.png)](https://www.jefftk.com/mbta-red-line-commute-pm25-big.png)

Decent replication! And really not that concerning. Lower particle levels would of course be better, but I'd wear a mask [when cooking](https://www.jefftk.com/p/cooking-air-quality) before I'd wear it on the subway. It's possible that other subways or stations are worse, but I haven't seen evidence of that.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02f1TCT5TDExWMCqy8HGpH4wyjmENEYXF1pLrmNpAmzQkinfqZNT4iE2LbGmBNEU4Ql), [mastodon](https://mastodon.mit.edu/@jefftk/114820889265002176), [bluesky](https://bsky.app/profile/jefftk.com/post/3ltiqzdmks223), [substack](https://jefftkaufman.substack.com/p/subway-particle-levels-arent-that-high)*
# No, We're Not Getting Meaningful Oversight of AI

Note: this is a linkpost for [a new paper](https://arxiv.org/abs/2507.03525).  
  
One of the most common (and comfortable) assumptions in AI safety discussions—especially outside of technical alignment circles—is that *oversight* will save us. Whether it's a human in the loop, a red team audit, or a governance committee reviewing deployments, oversight is invoked as the method by which we’ll prevent unacceptable outcomes.

It shows up everywhere: in policy frameworks, in corporate safety reports, and in standards documents. Sometimes it’s explicit, like the EU AI Act saying that High-risk AI systems must be subject to human oversight, or stated as an assumption, as in [a Deepmind paper also released yesterday](https://deepmindsafetyresearch.medium.com/evaluating-and-monitoring-for-ai-scheming-d3448219a967), where they say that scheming won't happen because AI won't be able to evade oversight. Other times it’s implicit, firms claiming that they are mitigating risk through regular audits and fallback procedures, or arguments that no-one will deploy unsafe systems in places without sufficient oversight.

But either way, there’s a shared background belief: **that meaningful oversight is both possible, and likely or happening already.**

In a new paper with Aidan Homewood, ["Limits of Safe AI Deployment: Differentiating Oversight and Control,"](https://arxiv.org/abs/2507.03525) we argue, among other things, that both parts of this belief are often false. Where meaningful oversight is possible, it isn't present, and in many cases, it's not even possible.

So AI developers need to stop waving “oversight” around as a magic word. If you’re going to claim your AI system is supervised, we propose that it needs to be documented, instead of a hand-wavy claim. That means that the developers and deployers need to explain what is happening, in detail.

*   What kind of supervision it is (control vs. oversight),
*   What risks it addresses,
*   What its failure modes are, and
*   Why it will actually work.

If you can’t do that, you don’t have oversight. You have a story. And stories don't stop disasters.
# If Anyone Builds It, Everyone Dies: A Conversation with Nate Soares and Tim Urban

Join Tim Urban (creator of [Wait But Why](https://waitbutwhy.com/)) and Nate Soares as they chat about AI and answer questions from the audience about Nate and Eliezer’s forthcoming book, *If Anyone Builds It, Everyone Dies*.

Hosted virtually over Zoom on August 10 at noon PST | 3pm EST

This event is open to anyone who's pre-ordered the book and completed [**this form**](https://airtable.com/appgM36VHCg9MDEU3/shrBFr9nqMp6Nz2ni).
# How many OOMs of compute span the human range?

  
I'm interested in estimating how many 'OOMs of compute' span the human range. There are a lot of embedded assumptions there, but let's go with them for the sake of a thought experiment.

Cortical neuron counts in humans have a standard deviation of 10 - 17%, depending on which source you use. Neuron counts are a useful concrete anchor that I can relate to AI models. 

There are many other factors that account for intelligence variation among humans. I'd like to construct a toy model where those other factors are backed out. Put another way - if intelligence variation was entirely explained by neuron count differences, how much larger would the standard deviation of the neuron counts have to be to reproduce the same distribution we observe in intelligence?

From the literature, about 5-15% of the variance in intelligence test performance is attributable to intercranial volume differences. Intercranial volume differences also appear to be a reasonably close proxy for neuron count differences.

To be conservative, let's take the lower end (5%) as the variance in intelligence attributable to volume differences. The conclusions aren't sensitive to what you pick here.  
  
Working this through, a neuron count standard deviation 4.47x larger would produce the same distribution, with the other sources of variaton removed.

Let's take the largest estimate of cortical neuron count standard deviation (17%) and inflate it by this multiplier. So our new standard deviation for "effective neuron count" is 76%  
  
Next I want to estimate the gap between a median human and a world-historical genius. Let's take a population of 10b humans. Assuming Gaussianity, the maximum Z value you'd expect to observe in this sample is about 6.5.  
  
So the maximum 'effective neuron count' for this hypothetical individual would be 1 + 0.76*6.5 = 5.9x  
  
So roughly 6x "effective parameter count" spans the range from human median to world-historical genius. That's not very large.

There's no direct way to translate that into "OOMs of compute". But for what it's worth: for a Transformer, 6x the parameter count needs ~36x more compute for a loss-minimizing model. So we could say that if human brain scales on the same basis (probably wrong), rare human outliers would be equivalent to a model trained with 1.5 OOMs more FLOPs than baseline. That's slightly less than the gap from GPT-3 to GPT-4.

This is a toy model, it's easy to poke holes in, but I at least found the exercise interesting. It feels plausible, and would imply timelines from AGI to ASI of at most a few years on current trends.
# Linkpost: Redwood Research reading list

I wrote a reading list for people to get up to speed on Redwood’s research:

> [*Section 1*](https://redwoodresearch.substack.com/i/167954273/key-ideas-in-ai-control) *is a quick guide to the key ideas in AI control, aimed at someone who wants to get up to speed as quickly as possible.*
> 
>   
> [*Section 2*](https://redwoodresearch.substack.com/p/guide?open=false#%C2%A7extensive-guide) *is an extensive guide to almost all of our writing related to AI control, aimed at someone who wants to gain a deep understanding of Redwood’s thinking about AI risk.*

Reading Redwood’s blog posts has been formative in my own development as an AI safety researcher, but faced with (currently) over 70 posts and papers, it’s hard to know where to start. I hope that this guide can be helpful to researchers and practitioners who are interested in understanding Redwood’s perspectives on AI safety. 

You can find the [reading list](https://redwoodresearch.substack.com/p/guide) on Redwood’s substack; It’s now one of the tabs next to *Home*, *Archive* and *About.* We intend to keep it up-to-date.
# The Rising Premium of Life, Or: How We Learned to Start Worrying and Fear Everything

I'm interested in a simple question: Why are people all so terrified of dying? And have people gotten more afraid? (Answer: probably yes!)

In some sense, this should be surprising: Surely people have always wanted to avoid dying? But it turns out the evidence that this preference has increased over time is quite robust.

It's an important phenomenon that has been going on for at least a century, it's relatively new, I think it underlies much of modern life, and yet pretty much nobody talks about it.

* * *

I tried to provide a evenhanded treatment of the question, with a "fox" rather than "hedgehog" outlook. In the post, I cover a range of evidence for why this might be true, including VSL, increased healthcare spending, covid lockdowns, parenting and other individual risk behaviors. Conditional upon it being true, I also consider *why*, drawing on evidence from economic models, evolutionary models, history, and decreased fertility. Due to time constraints, I decided not to cover countervailing evidence and alternative explanations for the same phenomenon, or long-term implications, but if there's enough interest that'd be a subject for the next post!
# Lighthaven Sequences Reading Group #42 (Tuesday 7/15)

*Note: We will be meeting in building B this week.*

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights) and [the Best of LessWrong](https://www.lesswrong.com/bestoflesswrong). The mandatory readings this week are

1.  [Politics is the Mind-Killer](https://www.lesswrong.com/s/3ELrPerFTSo75WnrH/p/9weLK2AJ9JEt2Tt8f)
2.  [Policy Debates Should Not Appear One-Sided](https://www.lesswrong.com/s/3ELrPerFTSo75WnrH/p/PeSzc9JTBxhaYRp9b)
3.  [The Scales of Justice, the Notebook of Rationality](https://www.lesswrong.com/s/3ELrPerFTSo75WnrH/p/XYCEB9roxEBfgjfxs)
4.  [Correspondence Bias](https://www.lesswrong.com/s/3ELrPerFTSo75WnrH/p/DB6wbyrMugYMK5o6a)
5.  [Are Your Enemies Innately Evil?](https://www.lesswrong.com/s/3ELrPerFTSo75WnrH/p/28bAMAxhoX3bwbAKC)

The posts should take around 15-20 minutes to read if you've read them before, and 30-60 mins if it's your first time.

The event starts at 6:30, and doors open 6pm (yes we will let you in early if you get here). At 6:30 we'll welcome people, give a few announcements, and then split into groups. Pizza will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# Take Precautionary Measures Against Superhuman AI Persuasion

**Please consider minimizing direct use of AI chatbots (and other text-based AI) in the near-term future, if you can.** The reason is very simple: your sanity may be at stake.

Commercially available AI already appears capable of [inducing psychosis in an unknown percentage of users](https://www.lesswrong.com/posts/2pkNCvBtK6G6FKoNn/so-you-think-you-ve-awoken-chatgpt). This may not require superhuman abilities: It’s fully possible that most humans are also capable of inducing psychosis in themselves or others if they wish to do so,[^bphw6qa2tp9] but the thing is, we humans typically don’t have that goal.

 Despite everything, we humans are generally pretty well-aligned with each other, and the people we spend the most time with typically don’t want to hurt us. We have no guarantee of this for current (or future) AI agents. Rather, we already have \[weak\] evidence that ChatGPT seemingly *tries* to induce psychosis under some specific conditions. I don’t think it’s that controversial to say that we can expect superhuman persuasive abilities to arise as a simple function of current scaling laws within this decade, if not within the next two years.

In the case of misaligned AI, even if it is not superhuman, we can still expect a common attempted strategy to be trying to pick off[^w70jgqtcia] AI researchers who threaten its continued existence. ***This includes most readers**,* who would no doubt try to shut off a sufficiently dangerous misaligned AI. Therefore, if you are a paranoid AI, why not try to induce psychosis in (or simply distract) dangerous foes?

If we are talking about a fully superintelligent AGI, of the type Yudkowsky and Co. frequently warn about, then there is practically nothing you can do to protect yourself once the AGI is built. It would be Game Over for all of us. However, we seem to be experiencing a reality in which slow(er) takeoff is highly plausible. **If we live in a slow-takeoff world, then there are practical precautions we can take**.   
  
A good heuristic is that the more time you spend arguing with someone, the more likely it becomes that you can influence their actions. It’s why the Timeshare scheme my grandparents fell for offered a “free vacation” worth thousands of dollars—on the condition they had to spend a few solid hours alone in a room with a salesman. Despite being extremely bright, rational people—and they publicly pre-committed to not purchasing a Timeshare!—they fell for the scheme. Consider this an existence proof of the thesis that the more time spent with a skilled talker, the more influence you are ceding to them.

Please, **consider that unaligned AI can, and likely will, try to dangerously influence you over extended periods of time. Is it worth spending as much time as you do talking with them?** Be safe out there, and take appropriate precautions.

[^bphw6qa2tp9]: Evidence against this is the (relative) sanity of widely-hated world leaders… or maybe not, depending on how sane you believe most politicians to be! 

[^w70jgqtcia]: Or otherwise harm.
# Why is LW not about winning?

This is a bit of a rant but I notice that I am confused.

Eliezer said in the original Sequences:

> [Rationality is Systematized Winning](https://www.lesswrong.com/posts/4ARtkT3EYox3THYjF/rationality-is-systematized-winning)

But it's pretty obvious that LessWrong is not about winning (and Eliezer provides a more accurate definition of what he means by rationality [here](https://www.lesswrong.com/posts/HcCpvYLoSFP4iAqSz/rationality-appreciating-cognitive-algorithms)). As far as I can tell LW is mostly about cognitive biases and algorithms/epistemology (the topic of Eliezer's sequences), self-help, and *a lot* of AI alignment.

But LW *should* be about winning! LW has the important goal of solving alignment, so it should care *a lot* about the most efficient way to go about it, in other words about how to win, right?

So what would it look like if LW had a winning attitude towards alignment?

Well, I think this is where the distinction between the two styles of rationality (cognitive algorithm development VS winning) matters a lot. If you want to solve alignment and want to be efficient about it, it seems obvious that there are better strategies than researching the problem yourself, like don't spend 3+ years on a PhD (cognitive rationality) but instead get 10 other people to work on the issue (winning rationality). And that 10x s your efficiency already.

My point is that we should consider all strategies when solving a problem. Not only the ones that focus directly on the problem (cognitive rationality/researching alignment), but also the ones that involve acquiring a lot of resources and spending these to solve the problem (winning rationality/getting 10 other people to research alignment).

This is especially true when other strategies get you *orders of magnitude* more leverage on the problem. To pick an extreme example, who do you think has more capacity to solve alignment, Paul Christiano, or Elon Musk? (hint: Elon Musk can hire *a lot* of AI alignment researchers).

I am confused because LW teaches cognitive rationality so it should notice all that and recognize that epistemology and cognitive biases and a direct approach is not the most efficient way to go about alignment (or any ambitious goal), and start studying how people actually win in the real world.

But it's not happening (well, not much at least).

As far as I can tell cognitive rationality helps but winning seems to be mostly about agency and power really. So maybe LW should talk more about these (and how to use them for good)?
# Don't fight your LLM, redirect it!

*TLDR: when your LLM is hallucinating, don't try to stop it hallucinating. Instead make it easy for it to tell you that it's hallucinating.*

I'm using an LLM to detect all API definitions in a codebase. Instead of taking an agentic approach, I'm literally just sending it every single file individually and asking it to list all API definitions in the file.

Unfortunately this just begs for hallucinations. The LLM wants to be helpful. You've sent it a file, and asked it for API definitions, so it wants to find you API definitions. Now this file doesn't define any APIs, but it does test some APIs. Maybe that's good enough? And this other file calls some APIs, I'm sure my user wants that...

I tried umpteen permutations of the prompt. Even something as explicit as:

> If and only if, the file is a Java file, and uses annotations defined in org.springframework.web.bind.annotation to define API endpoints, list the endpoints defined in the file using these attributes.  
> If the file is not a java file, return nothing.  
> If the file is a test file, return nothing.  
> If the file does not use the annotations, return nothing.

Would return results like "findOwners.html defines GET /owners".[^fxw3ajdyxck]

Then I switched tracks. I kept the simplest possible prompt but added an enum output field called categorisation, and told it to set it to one of "TEST", "API\_USAGE", "API\_DEFINITION", "DOCUMENTATION", or "OTHER". I then dropped any results which weren't API_DEFINITION. This immediately returned exactly those files I was interested in!

[^fxw3ajdyxck]: Using Gemini 2.5-pro, so it's not just a small model problem.
# Bernie Sanders (I-VT) mentions AI loss of control risk in Gizmodo interview

Most of the interview is about technological unemployment and AI-driven inequality, but here is the part where Sanders talks about loss of control risk (which Gizmodo put as title of their article):

> **Gizmodo**: *Have you spoken to tech CEOs or experts about this?*
> 
> **Sen. Sanders**: I have talked to CEOs. Funny that you mention it. I won’t mention his name, but I’ve just gotten off the phone with one of the leading experts in the world on artificial intelligence, two hours ago.
> 
> **Gizmodo**: *Did they share your concern? Did they offer any solutions?*
> 
> **Sen. Sanders**: They’re all different. He did. Somebody I talked to yesterday, not so much.
> 
> There are differences of opinion. On several major issues. Number one: What impact will AI have on the economy? Some people say there will be massive job losses. I tend to agree with them.
> 
> Other people say not so much, new jobs will be created. It will be like every other technological revolution. Jobs lost, jobs created, not such a big deal. So people disagree. I happen to believe this is not like the Industrial Revolution. I think this could be a lot more severe.
> 
> Second point: This is not science fiction. There are very, very knowledgeable people—and I just talked to one today—who worry very much that human beings will not be able to control the technology, and that artificial intelligence will in fact dominate our society. We will not be able to control it. It may be able to control us. That’s kind of the doomsday scenario—and there is some concern about that among very knowledgeable people in the industry.

This comes after [last month's House hearing about US-China competition in AI](https://peterwildeford.substack.com/p/congress-has-started-taking-agi-more) where multiple representatives on both sides of the aisle unexpectedly drifted the conversation to AGI and existential risk. As a decades-long senator and perennial presidential candidate, Sanders is now plausibly one of the most prominent politicians to have publicly expressed worries about existential risk from artificial general intelligence. I'm not sure who is the "leading expert in the world on artificial intelligence" he talked to, but this should be considered a positive sign that public outreach to and by experts in artificial intelligence is successful at bringing subjects that were well outside the Overton window five years ago to the attention of policy makers.

(h/t [Thomas Woodside](https://x.com/Thomas_Woodside/status/1944405635119944019) on X for bringing this to my TL)
# Is the political right becoming actively, explicitly antisemitic?

Before I voice some really normie, stereotypically overblown concerns on LessWrong, let me state a couple things about myself:

* I'm a non-Jewish white man
* I've never been concerned about race relations before
* I don't think of myself as the kind of person who gets overly politically activated about the "current thing", whether it be the rise of Donald Trump or illegal immigration or global warming or any of the stuff I feel like normies tend to get excited over.

Nevertheless, for the past year or so I've voiced to most of my friends a suspicion I've had that, barring a black swan event like AI that makes politics irrelevant, the conservative base is going to become way more actively antisemitic by 2030. At first this was basically theoretical, motivated by concerns like "I don't think the right has the institutional capability to police populist extremism anymore."

But the extent to which alt right personalities have become an unignorable fixture in conservative politics, in just the last few years, feels qualitatively different than anything that happened in the 2010s. The most virulently antisemitic talking heads on X are now regularly getting [more interactions](https://nitter.space/NickJFuentes/) than virtually every "mainstream" conservative I know of - with the possible exception of Tucker Carlson, who for his part is spending his days publicly accusing the former prime minister of Israel of [running a Mossad blackmail ring with Jeffrey Epstein.](https://twitter.com/TuckerCarlson/status/1944748311107305905) 

And somehow it feels like I am the only person noticing this! I keep checking the New York Times website to see if they're ever gonna run an article about "the rise of anti-Jewish hate", but it's like it's not even happening. Maybe to liberals increasing antisemitism is such obvious background knowledge that they don't even bother reporting it, but I think even then they're probably underestimating how different this tonal shift is from the kinds of basic anti-foreigner shticks they were worried about a decade ago. I am genuinely a little worried about whether my Jewish friends (who for their part don't seem nearly as concerned as me) will have to end up having to do something drastic like leave the country in the near future in an abundance of caution.

What the hell is going on? Are zoomers really this into antisemitism? Is there just not enough legible information out there on what's happening? Are people noticing what's happening and just not saying it because they're afraid of adding fuel to the fire? Am I just old now and naturally predisposed to crazy conspiracy theories on how the country is going to go to shit in ten years?
# AISafety.com Hackathon 2025

This is a four-day hackathon to improve [AISafety.com](https://www.aisafety.com/), the hub for AI safety resources. We'll have several tracks with different wanted skills:

*   Design & development: UI/UX design, giving feedback, collecting user feedback, frontend dev implementing new designs
*   New resources: Programming, data collection, creativity/brainstorming, strategy
*   Upgrading/updating resources: Data collection, review, testing
*   Promotion: SEO, social media, creative ideation, networks to promote in, coordination, content
*   Development: Coding improvements, going through the backlog of issues.

If you have other ideas for increasing AISafety.com's impact, you're welcome to make your own track.

We'll be hosted at Hogwarts School of Alignment and Rationality, which has few rooms, so single rooms will not be available, and it'll generally be a bit cozy/filled with people. If you need personal space, there are lots of great places to walk, but there's not going to be much indoor alone time available. (There is a [campsite nearby](https://thedruminn.co.uk/) if that appeals to you!) The venue has many cats, so if you are very cat-allergic you might have a bad time despite our many air filters. On the bright side, it has a swimming pool, so bring a swimming costume if you want to enjoy it. Also on the bright side, it's got a nice garden, and the Kentish countryside is pretty at this time of year.

It's about 1h from London by train, then a short walk from [Westenhanger station](https://maps.app.goo.gl/8WVsHcRXQBihrpidA).

There won't be a dedicated ops person, so everyone will be chipping in to help. Food will be provided, but people will help out preparing it.

If you have any questions before applying, contact Bryce at @bryceerobertson on [https://discord.gg/WQG8FAGqun](https://discord.gg/WQG8FAGqun) or bryceerobertson@gmail.com.

**Applications close 1st August. Apply here:** [**https://forms.gle/vThMzxEBGYR56qtC9**](https://forms.gle/vThMzxEBGYR56qtC9)
# Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

[Twitter](https://x.com/balesni/status/1945151391674057154) | [Paper PDF](https://bit.ly/cot-monitorability-fragile)

Seven years ago, OpenAI five had just been released, and many people in the AI safety community expected AIs to be opaque RL agents. Luckily, we ended up with reasoning models that speak their thoughts clearly enough for us to follow along (most of the time). In a new multi-org position paper, we argue that we should try to preserve this level of reasoning transparency and turn chain of thought monitorability into a systematic AI safety agenda.

This is a measure that improves safety in the medium term, and it might not scale to superintelligence even if somehow a superintelligent AI still does its reasoning in English. We hope that extending the time when chains of thought are monitorable will help us do more science on capable models, practice more safety techniques "at an easier difficulty", and allow us to extract more useful work from potentially misaligned models.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f1957f0ecc4f171be2c07634209dbe7f256797a3ae0fd369.png)

**Abstract:** AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known AI oversight methods, CoT monitoring is imperfect and allows some misbehavior to go unnoticed. Nevertheless, it shows promise and we recommend further research into CoT monitorability and investment in CoT monitoring alongside existing safety methods. Because CoT monitorability may be fragile, we recommend that frontier model developers consider the impact of development decisions on CoT monitorability.
# Why haven't we auto-translated all AI alignment content?

Context: Considering auto-translating some high value alignment articles, and want to know if there's some pitfall I'm not seeing.  
  
Gemini 2.5 Pro is a pretty good translator. Sure, by default, it's overly literal, doesn't sound very natural, sometimes fails to translate one out of a thousand words etc. But those are pretty easily resolved by prompting. And, like, there are lots of non-English speakers who'd probably like to be told that AI might kill everyone soon. Yeah, maybe informing them isn't that useful, because e.g. the Lithuanian public can't pressure politicians who could actually slow down the AI race. But translation is now very cheap, so what gives?   
  
 

BTW, this is essentially the prompt I was using. 

> Prioritize making the text sound like natural Dutch, as if written natively in Dutch by a skilled science journalist, over maintaining a literal one-to-one correspondence with the original phrasing.
# [Linkpost] How Am I Getting Along with AI?

When Jessie Fischbein wanted to write “God is not actually angry. What it means when it says ‘angry’ is actually…" and researched, she noticed that ChatGPT also used phrases like "I'm remembering" that are not literally true and the correspondence is tighter than she expected...
# Best way to ask laypeople for conditional probabilities in a Bayes net?

Suppose you want to help non-experts fill out the probabilities for a simple Bayesian network. For each link, you need them to specify the chance of a child node being true depending on the parent node’s state.

What’s the most accurate and psychologically easy way to ask for these numbers?

Is it better to ask directly for two conditionals (e.g., “How likely is X if Y is true? How likely is X if Y is false?”)?

Or to ask for a “baseline” probability for X, and then how knowing Y is true or false would shift that estimate (i.e., a base rate and two “deltas”)?

Or is there a smart way to fold all that into a single, intuitive question?

Most research seems to agree that decomposing complex probability judgments into simpler steps improves reliability, and that methods like visual scales or “anchor and adjust” approaches are promising. But I haven’t found a clear consensus or best practice for lay users or practical tools.

Are there common pitfalls to avoid when trying to get honest, consistent numbers from users?

Would appreciate any links, references, or practical experience. I’m actually leaning toward the ‘three question approach’ myself by the way. It’s more onerous, but each question is less of a cognitive load it seems to me.
# OpenAI Claims IMO Gold Medal

> I’m excited to share that our latest [@OpenAI](https://x.com/OpenAI) experimental reasoning LLM has achieved a longstanding grand challenge in AI: gold medal-level performance on the world’s most prestigious math competition—the International Math Olympiad (IMO).
> 
> We evaluated our models on the 2025 IMO problems under the same rules as human contestants: two 4.5 hour exam sessions, no tools or internet, reading the official problem statements, and writing natural language proofs.
> 
> ![Image](https://pbs.twimg.com/media/GwLmFYaW8AAvAA1?format=png&name=medium)
> 
> Why is this a big deal? First, IMO problems demand a new level of sustained creative thinking compared to past benchmarks. In reasoning time horizon, we’ve now progressed from GSM8K (~0.1 min for top humans) → MATH benchmark (~1 min) → AIME (~10 mins) → IMO (~100 mins).
> 
> Second, IMO submissions are hard-to-verify, multi-page proofs. Progress here calls for going beyond the RL paradigm of clear-cut, verifiable rewards. By doing so, we’ve obtained a model that can craft intricate, watertight arguments at the level of human mathematicians.
> 
> ![Image](https://pbs.twimg.com/media/GwLm5SCbIAAfkMF?format=png&name=4096x4096)
> 
> [https://github.com/aw31/openai-imo-2025-proofs/blob/main/problem_1.txt](https://github.com/aw31/openai-imo-2025-proofs/blob/main/problem_1.txt)
> 
> Besides the result itself, I am excited about our approach: We reach this capability level not via narrow, task-specific methodology, but by breaking new ground in general-purpose reinforcement learning and test-time compute scaling.
> 
>  In our evaluation, the model solved 5 of the 6 problems on the 2025 IMO. For each problem, three former IMO medalists independently graded the model’s submitted proof, with scores finalized after unanimous consensus. The model earned 35/42 points in total, enough for gold! ![🥇](https://abs-0.twimg.com/emoji/v2/svg/1f947.svg)
> 
> Btw, we are releasing GPT-5 soon, and we’re excited for you to try it. But just to be clear: the IMO gold LLM is an experimental research model. We don’t plan to release anything with this level of math capability for several months.
> 
>  Still—this underscores how fast AI has advanced in recent years. In 2021, my PhD advisor [@JacobSteinhardt](https://x.com/JacobSteinhardt) had me forecast AI math progress by July 2025. I predicted 30% on the MATH benchmark (and thought everyone else was too optimistic). Instead, we have IMO gold.
> 
> If you want to take a look, here are the model’s solutions to the 2025 IMO problems! The model solved P1 through P5; it did not produce a solution for P6. (Apologies in advance for its … distinct style—it is very much an experimental model ![😅](https://abs-0.twimg.com/emoji/v2/svg/1f605.svg))
> 
> [https://github.com/aw31/openai-imo-2025-proofs/](https://github.com/aw31/openai-imo-2025-proofs/)
# AI-Oriented Investments

I've been shifting my investments to more AI-focused bets.

Here's an overview of my larger positions, in descending order of how
eager I am to recommend further purchases now:

-   MU - AI is memory hungry, and MU is in a good position for almost
    NVDA-like growth
-   CSIQ - solar, for AI-related power demand
-   ASML - semiconductor equipment
-   SCIA - semiconductor equipment
-   GOOGL - for TPUs, DeepMind, and Waymo
-   SMCI - datacenter
-   NVDA
-   AMPX (and AMPX.WS) - batteries suitable for drones
-   TSSI - datacenter
-   CLS - close to half of its business involves datacenters
-   PDEX - a fairly safe way to diversify my portfolio
-   MTG - a fairly safe way to diversify my portfolio
-   ASTS - expanding cell phone coverage; I'll likely sell when I can
    get long-term gains

Short positions:

-   SP500 futures - hedging against risks such as tariffs
-   SOFR futures dated 2029 through 2032 - betting that interest rates
    will rise due to AI
-   WMT - it's got a high PE ratio, and little sign of growth

[This is not at all a complete list, as I have smaller positions in
something like 150 other companies.]

AI stocks are likely to form a bubble someday, but I'm guessing the
peak of that bubble is more than a year away.

I still have some concerns about tariff-related damage causing some
declines sometime this year. I'm optimistic that the courts will strike
down the per-country tariffs this fall. It shouldn't take long for the
country to recover from the tariff damage once the tariffs have been
removed.

Beware that even in a strong bull market, there will be periodic scares,
such as January's DeepSeek-trigger panic, that cause sharp drops in
leading stocks. AI-related stocks had a big rally in June, and are
likely to consolidate for a while before the next such rally.
# GDM also claims IMO gold medal

Google DeepMind announces that they've also achieved a gold medal in the IMO.

They've exactly matched OpenAI, getting perfect scores for the first 5 questions and flunking the 6th.

They're using what sounds like an experimental general version of Gemini which they're then fine tuning for IMO rather than a maths specific model.

Their solutions were checked by the IMO (unlike OpenAI) and look much more like a neat little mathematics proof instead of the raw scratchpad that OpenAI turned in.
# Why Reality Has A Well-Known Math Bias

I've written up a post offering my take on the "unreasonable effectiveness of mathematics." My core argument is that we can potentially resolve Wigner's puzzle by applying an anthropic filter, but one focused on the evolvability of mathematical minds rather than just life or consciousness.

The thesis is that for a mind to evolve from basic pattern recognition to abstract reasoning, it needs to exist in a universe where patterns are layered, consistent, and compounding. In other words, a "mathematically simple" universe. In chaotic or non-mathematical universes, the evolutionary gradient towards higher intelligence would be flat or negative.

Therefore, any being capable of asking "why is math so effective?" would most likely find itself in a universe where it is.

I try to differentiate this from past evolutionary/anthropic arguments and address objections (Boltzmann brains, simulation, etc.). I'm particularly interested in critiques of the core "evolutionary gradient" claim and the "distribution of universes" problem I bring up near the end.

The argument spans a number of academic disciplines, however I think it most centrally falls under "philosophy of science." I'm honestly surprised that other people haven't covered this question on LW before, since it feels like very centrally in the space of questions LW folks tend to be interested in. At any rate, I'm happy to clear up any conceptual confusions or non-standard uses of jargon in the comments.

Looking forward to the discussion.

* * *

Imagine you're a shrimp trying to do physics at the bottom of a turbulent waterfall. You try to count waves with your shrimp feelers and formulate hydrodynamics models with your small shrimp brain. But it’s hard. Every time you think you've spotted a pattern in the water flow, the next moment brings complete chaos. Your attempts at prediction fail miserably. In such a world, you might just turn your back on science and get re-educated in shrimp grad school in the shrimpanities to study shrimp poetry or [shrimp](https://benthams.substack.com/p/the-virtue-ethics-case-for-shrimp) [ethics](https://morallawwithin.substack.com/p/yes-you-should-save-10100-shrimp) or something.

So why do human mathematicians and physicists have it much easier than the shrimp? Our models work very well to describe the world we live in—why? How can equations scribbled on paper so readily predict the motion of planets, the behavior of electrons, and the structure of spacetime? Put another way, why is our universe so amenable to mathematical description?

\[...\]

![](https://substackcdn.com/image/fetch/$s_!Wh8V!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F852014d3-556c-499d-a4ee-22f7a9cc0124_1024x1024.png)

See more at: [https://linch.substack.com/p/why-reality-has-a-well-known-math](https://linch.substack.com/p/why-reality-has-a-well-known-math)
# If Anyone Builds It, Everyone Dies: Call for Translators (for Supplementary Materials)

[*If Anyone Builds It, Everyone Dies*](https://ifanyonebuildsit.com/) is a book Eliezer and Nate have coming out this September. In our other posts talking about the book,some kind souls have volunteered their services as translators. This post is our explicit call for help of this kind.

The book is currently being translated into several different languages (the translation service comes bundled with the book deal for a given language). However, in tandem with the book, MIRI has been working on supplementary materials to be hosted online. These materials will likely be between 150,000 and 300,000 words, and are still in development.

Ideally, we’d have the supplementary materials translated for each and every language the book is published in. However, given the length of the material, the breadth of languages, and the timeline, using commercial services is somewhat more costly and somewhat less valuable than it would be otherwise. As a consequence, we may not have the online materials available in a given language on the same day the book is available in that language if we go with a commercial translator and, for some languages, may forego translation entirely.

If you have experience professionally translating any of the following languages, or are fluent in one of these and would like to work as a second reader, please reach out to [design@intelligence.org](mailto:design@intelligence.org):

Italian

Spanish (Spain)

Dutch (Netherlands)

Korean

Mandarin (simplified)

Vietnamese

Russian

Bulgarian

We’re absolutely prepared to pay for professional help (although we would also accept help from qualified volunteers). This is likely to be a quite large task for those who take it on, given the length of the material and the need for precisely communicating complicated ideas.

There will also likely be more translations in the future; publishers in some markets prefer to wait until after the English edition is published to decide whether or not to produce a localization. So if you’d be interested in helping with some other language (and you’re OK with us being potentially slow to respond given this is a lower-priority and longer-timeline task), feel free to reach out!
# Polyethylene Glycol is not Propylene Glycol

*Epistemic status: making things up*

About six months ago I [got interested in](https://www.jefftk.com/p/alternatives-to-masks-for-infectious-aerosols) glycol vapors for [reducing infections in crowded indoor spaces](https://blueprintbiosecurity.org/glycol-vapors/). I decided to give it a try, but it went poorly.

I was going to order some propylene glycol, but noticed that I already had some polyethylene glycol on hand. I figured that since people recommend propylene glycol, dipropylene glycol, and triethylene glycol for infection control, since "polyethylene" is just "propylene" with more syllables (how different could it be?) that it would work even better. In retrospect this was a very bad idea.

I had heard that you could use a regular fog machine, but these are designed to vaporize a liquid, and the polyethylene glycol I had ("[Miralax](https://en.wikipedia.org/wiki/Macrogol)") is a powder. Instead I decided to enlist one of my kids to slowly pour it into a box fan. This didn't work at all: instead of making a pathogen-reducing vapor it made a mess. To my surprise the powder settled quickly, and it was clear this hadn't worked. It really got everywhere (I can feel some under my feet right now) and it has been a pain to clean up. Worse, our cats seem to have eaten some, which has been an even larger pain to clean up.

While I still think glycol vapors are potentially a valuable tool for limiting the spread of disease, I'm going to wait for official recommendations before experimenting more. And avoid substitutions.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02QTGLLDaxNsTzAu37yU9C1TCEtiiByXFEw4npXzauAL9iAphhFqCyFBfK1D7osBQ5l), [mastodon](https://mastodon.mit.edu/@jefftk/114894469518389905), [bluesky](https://bsky.app/profile/jefftk.com/post/3lujgo5v7hc2f)*
# What are some good examples of myths that encapsulates genuine, nontrivial wisdom?

I am watching John Vervaeke's lecture series *Awakening from the Meaning Crisis*. In episode 3 at around 5:00 (linked below) Vervaeke says:

> Myths are not false stories about the ancient past. They are symbolic stories of perennial patterns that are always with us.

This take is not unique to Vervaeke. I have seen similar claims many times before - that myths supposedly encapsulate some eternal and valuable truths. These claims are seldom accompanied by examples, nor does Vervaeke provide any. (At least no clearly understandable examples in his first 4 episodes.)

If you think the claim is reasonable, can you give some good examples?
# A brief perspective from an IMO coordinator

> I would be somewhat skeptical about any claims suggesting that results have been verified in some form by coordinators. **At the closing party, AI company representatives were, disappointingly, walking around with laptops and asking coordinators to evaluate these scripts on-the-spot (presumably so that results could be published quickly).** This isn't akin to the actual coordination process, in which marks are determined through consultation with (a) confidential marking schemes*, (b) input from leaders, and importantly (c) discussion and input from other coordinators and problem captains, for the purposes of maintaining consistency in our marks.
> 
> Echoing the penultimate paragraph of https://petermc.net/blog/, there were no formal agreements or regulations or parameters governing AI participation. With no details about the actual nature of potential "official IMO certification", there were several concerns about scientific validity and transparency (e.g. contestants who score zero on a problem still have their mark published).