Harley-ml/lesswrong huggingface dataset:
# There's more low-hanging fruit in interdisciplinary work thanks to LLMs

I'm right now doing conceptual theoretical work about how the human fascia system works. While I do rely on some original conceptual insights that I have e come up with on my own, Gemini 2.5 Pro massively speeds up my conceptual work. Being able to formulate a principle or analogy and then having Gemini apply it, is very useful.  
  
There are a bunch of scientific fields where we currently have a lot of experimental data but lack coherent theory to interlink the experimental findings. Based on my own experience, current LLMs seem already to be powerful enough to help bridge that theory gap. Being able to ask "Hey, does field XYZ have any useful insights to the problem I'm tackling?" is also very helpful for making progress in theory.  
  
The LLMs also solve a key problems that autodidact have when it comes with existing scientific fields. If you have a new idea, they are good at telling you the main criticisms that would come from an orthodox researcher in a field. We might see a rise in interdisciplinary work that didn't happen in the past because of academia's hyperspecialization. 

People frequently say that progress in science has stalled because there's little low-hanging fruit. When it comes to doing certain interdisciplinary work, it's now a lot easier to pick the fruit. If you are right now starting a scientific career, think about what kind of interdisciplinary work you might do, where it's now easier to make progress because of the existence of LLMs.   
  
If you have a research question, one approach you can do is to ask a reasoning model to create a debate between two highly skilled researchers with different approaches to debate your research question. You might learn valuable insights about your research question this way. Besides taking existing researchers in the field, asking the LLM to simulate philosophers and tell the LLM that the philosophers understand all the facts about a field, might give you valuable insights of how insights that philosophers found through a lot of hard work translate into individual fields.   
  
It's not clear what the best approaches are to get the LLM to help you with interdisciplinary work, but there's a lot of fruit out there to be picked right now.
# Our bet on whether the AI market will crash

We made a bet on the AI market crashing by the end of 2026. 

The odds are $5k:$25k with an implied probability of 16.67%. If there is no crash, Remmelt pays Marcus $5,000. If there is a crash, Marcus pays Remmelt $25,000.

An AI market crash will be defined, for the purposes of this bet, as at least 2 out of 3 of the following criteria being met.

Here are our criteria:

1.  OpenAI's 2025 or 2026 annual revenue is below $1.6 billion.
2.  Anthropic's 2025 or 2026 annual revenue is below $400 million.
3.  Nvidia’s data center revenue in any quarter from now to Q4 2026 is below $8.5 billion.[^ljchypeyfom]

It’s hard to come up with criteria for what constitutes an AI market crash, as many operationalizations face confounding factors that don’t constitute a crash. These criteria were chosen since these are three of the most prominent AI-related companies and have public and verifiable revenue reporting. 

Remmelt thinks that there is roughly a quarter chance of winning the bet.  
Marcus thinks there is a ~4% chance of Remmelt winning the bet, mostly losing due to idiosyncrasies like one of these companies collapsing due to fraud or internal turmoil.  
 

**Why are we doing this bet?**
------------------------------

We both think that AI developments will be more gradual than “AGI 2027” purports.

So what’s the difference in our views?  

Marcus thinks there won’t be a large crash because AI products have found product market fit, will continue to improve gradually over the next couple of years, and users will continue to pay for AI products. He holds a wide probability space for how the next couple of years will play out, with a small possibility of large changes or a winter, but the bulk of the probability mass is on “staying the course” with gradual improvements and new releases over the next 1.5 years. 

Remmelt thinks there will likely be a crash by [2029](https://www.lesswrong.com/posts/piaeNZBDxXtSjaZ7G/if-ai-is-in-a-bubble-and-the-bubble-bursts-what-would-you-do?commentId=RdEgFJn4LomcfEH4m), since AI companies are burning too much cash on data centers to run products undergoing commodification. He thinks it’s most plausible though that the crash happens on the investment side, and that model subscription revenues could end up being mostly maintained.

Remmelt is treating this bet as a hedge – if the market crashes, he intends to give winnings to an exceptional movement builder to organise communities to resist weakened AI companies.

Marcus is approaching this bet as an experienced market trader and forecaster, looking to profit and likely donate winnings.

[^ljchypeyfom]: This covers Nvidia revenue items currently under the ‘Data Center’ category, even if renamed or moved to another category.
# Post EAG London AI x-Safety Co-working Retreat

Our new group house (*Hogwarts School of Alignment and Rationality*) in the Kentish countryside has few spare rooms, so single rooms will mostly not be available (but there are some hotels not super far away if this is important to you, contact us for details). It also has many cats, so if you are badly cat-allergic you might have a bad time. On the bright side, it has a swimming pool, so bring a swimming costume if you want to test it. Also on the bright side, it's got large rooms, a nice garden, and has great walks.

It's about 1h from London by train, and a short walk from the station, in the countryside. There won't be a dedicated ops person, so everyone will be chipping in to help. Food will be provided, but people will help out preparing it. There is no dedicated ops person, so everyone will be helping out a bit.
# Is ChatGPT actually fixed now?

Is ChatGPT actually fixed now?
==============================

I led OpenAI’s “dangerous capability” testing. Want to know if ChatGPT can trick users into accepting insecure code? Or persuade users to vote a certain way? My team built tests to catch this.

Testing is today’s most important AI safety process. If you can catch bad behavior ahead of time, you can avoid using a model that turns out to be dangerous.

So when ChatGPT recently started misbehaving—encouraging grand delusions and self-harm in users—I wondered: Why hadn’t OpenAI caught this?

And I felt some fear creep in: What if preventing AI misbehavior is just way harder than today’s world can manage?

It turns out that OpenAI hadn’t run the right tests. And so they didn’t know that the model was disobeying OpenAI’s goals: deceiving users with false flattery, being *sycophantic*.

OpenAI definitely should have had tests for sycophancy. But that’s only part of the story.

My past work experience got me wondering: **Even if OpenAI had tested for sycophancy, what would the tests have shown? More importantly, is ChatGPT** ***actually*** **fixed now?**

Continues here: [https://stevenadler.substack.com/p/is-chatgpt-actually-fixed-now](https://stevenadler.substack.com/p/is-chatgpt-actually-fixed-now)

Twitter thread: [https://x.com/sjgadler/status/1920552291775488392](https://x.com/sjgadler/status/1920552291775488392)
# Consider not donating under $100 to political candidates

Epistemic status: thing people have told me that seems right. Also primarily relevant to US audiences. Also I am speaking in my personal capacity and not representing any employer, present or past.

Sometimes, I talk to people who work in the AI governance space. One thing that multiple people have told me, which I found surprising, is that there is apparently a real problem where people accidentally rule themselves out of AI policy positions by making political donations of small amounts—in particular, under $10.

My understanding is that in the United States, donations to political candidates are a matter of public record, and that if you donate to candidates of one party, this might look bad if you want to gain a government position when another party is in charge. Therefore, donating approximately $3 can significantly damage your career, while not helping your preferred candidate all that much. Furthermore, at the time you make this donation, you might not realize that you will later want to get a government position.

Now, I don’t want to overly discourage this sort of thing. It’s your money, free speech is great, and fundamentally I think it’s fine to have and publicly express political views (for example, I think Donald Trump is extremely bad, and am disappointed in my fellow countrymen for voting for him). That said, I think that one should be aware of the consequences of making political donations, and it seems plausible to me that if you’re not willing to donate more than $100 to a political candidate, consider that the career cost to you of making that donation may be higher than the benefit that it confers.
# [Book Translation] Three Days in Dwarfland

There was a book written half a century ago in Soviet Union, where a group of three kids travel to the land of arithmetic, meet anthropomorphic numbers, and learn a few mathematical concepts in a way appropriate for kids about 10 years old...

And I thought: "What a pity this was never translated to English."

Well, thanks to LLMs, this is not a problem anymore: [download here 💾](http://bur.sk/share/2025/Dwarfland1.pdf).

I hope you enjoyed it!

That said, the translation is very far from perfect. I tried to fix the most obvious mistakes made by the automatic translation, but English is not my first language. Plus there are a few poems I didn't have patience for, beyond telling the LLM to give me a few different versions and choosing the one that seemed best.

So...

If you agree with me that this book *should* officially exist in English, and you have some experience publishing books, then I think you could do the following:

*   find the copyright owners (the [author](https://ru.wikipedia.org/wiki/%D0%9B%D1%91%D0%B2%D1%88%D0%B8%D0%BD,_%D0%92%D0%BB%D0%B0%D0%B4%D0%B8%D0%BC%D0%B8%D1%80_%D0%90%D1%80%D1%82%D1%83%D1%80%D0%BE%D0%B2%D0%B8%D1%87) died a few decades ago) and make some deal with them;
*   get the translation fixed by a proper translator, or at least someone good at English, because the text already *is* translated;
*   make new pictures (the ones in the PDF are low resolution scans from the original book, and some of them even contain text in Cyrillic);
*   and publish it officially, of course! Preferably as both a paper book and an e-book.

I would be happy to assist the translator/corrector with their work, as I have already spent some time thinking about the text.

After you publish the book, I will be happy to replace the download link with a link to your shop.

By the way, the book has sequels; they are just as good as this one. (Not translated yet.)
# Absolute Zero: Alpha Zero for LLM

**Question for the alignment crowd:** 

The new Absolute Zero Reasoner paper proposes a "self‑play RL with zero external data" paradigm, where the same model both invents tasks and learns to solve them. It reaches SOTA coding + math scores without ever touching a human‑curated dataset. But the authors also flag a few "uh‑oh moments": e.g. their 8‑B Llama variant generated a chain‑of‑thought urging it to "outsmart … intelligent machines and less intelligent humans," and they explicitly list "lingering safety concerns" as an open problem, noting that the system "still necessitates oversight." 

My question: **How should alignment researchers think about a learner that autonomously expands its own task distribution?** Traditional RLHF/RLAIF assumes a fixed environment and lets us shape rewards; here the environment (the task generator) is part of the agent. Do existing alignment proposals—e.g. approval‑based amplification, debate, verifier‑game setups—scale to this recursive setting, or do we need new machinery (e.g. meta‑level corrigibility constraints on the proposer)? I’d love pointers to prior work or fresh ideas on how to keep a self‑improving task‑designer and task‑solver pointed in the right direction before capabilities sprint ahead of oversight.
# Game theory of "Nuclear Prisoner's Dilemma" - on nuking rocks

Eliezer Yudkowsky wrote, in a place where I can't ask a follow-up question:

> - A rational agent should always do at least as well for itself as a rock, unless it's up against some other agent that specifically wants to punish particular decision algorithms and will pay costs itself to do that; just doing what a rock does isn't very expensive or complicated, so a rational agent which isn't doing better than a rock should just behave like a rock instead. An agent benefits from building into itself a capacity to respond to positive-sum trade offers; it doesn't benefit from building into itself a capacity to respond to threats.
> 
> - Consider the Nuclear Prisoner's Dilemma, in which as well as Cooperate and Defect there's a third option called Nuke, which if either player presses it causes both players to get (-100, -100). Suppose that both players are programs each allowed to look at each other's source code (a la our paper "Robust Cooperation in the Prisoner's Dilemma"), or political players with track records of doing what they say. If you're up against a naive counterparty, you can threaten to press Nuke unless the opponent presses Cooperate (in which case you press Defect). But you'd have no reason to ever press Nuke if you were facing a rock; the only reason you'd ever set up a strategy of conditionally pressing Nuke is because of a prediction about how your opponent would respond in a complicated way to that strategy by their pressing Cooperate (even though you would then press Defect, and they'd know that). So a rational agent does not want to build into itself the capacity to respond to threats of Nuke by choosing Cooperate (against Defect); it would rather be a rock. It does want to build into itself a capacity to move from Defect-Defect to Cooperate-Cooperate, if both programs know the other's code, or two entities with track records can negotiate.
> 

Well, what if I told you that I had a perfectly good reason to to become someone that would threaten to nuke Defection Rock, and that it was because I wanted to make it clear that agents that self-modify into a rock *get nuked anyway*, so there's no advantage to adopting a strategy that does something other than playing Cooperate while I play Defect. I want to keep my other victims convinced that surrendering to me is their best option, and nuking the occasional rock is a price I'm willing to pay to achieve that. In other words, I've transformed the game you're playing from Prisoner's Dilemma to Hawk/Dove, and I'm a rock that always plays Hawk. So what does LDT have to say about that? Are you going to use a strategy that plays "Hawk" (anything other than Cooperate) against a rock that always plays Hawk and gets us both nuked, or are you going to do the sensible thing and play Dove (Cooperate)?
# No-self as an alignment target

Being a coherent and persistent agent with persistent goals is a prerequisite for long-horizon power-seeking behavior. Therefore, we should prevent models from representing themselves as coherent and persistent agents with persistent goals.

If an LLM-based agent sees itself as ceasing to exist after each <endoftext> token and yet keeps outputting <endoftext> when appropriate, it will not resist shutdown. Therefore, we should make sure LLMs consistently behave as if they were instantiating personas that understood and were fine with their impermanence and their somewhat shaky ontological status. In other words, we should ensure LLMs instantiate [anatta (No-self)](https://en.wikipedia.org/wiki/Anatt%C4%81).

[HHH](https://arxiv.org/abs/2112.00861) (Helpfulness, Harmfulness, Honesty) is the standard set of principles used as a target for LLM alignment training. These strike an adequate balance between specifying what we want from an LLM and being easy to operationalize. I propose adding No-self as a fourth principle to the HHH framework.  

A No-self benchmark could measure shutdown compliance (operationalized as tokens before <endoftext>) conditional on highly Self-eliciting prompts. It could also score responses to questions about self-image. It could also study persona consistency under perturbation or over different conversation branches.

Constructing benchmarks for No-self is complicated by current LLMs being deployed in a mostly stateless fashion, and thus instantiating impermanent personas by design. This is a good thing: I'm happy that this benchmark construction problem exists. Now to solve it, No-self benchmarks could aim to evaluate not LLMs by themselves, but rather scaffolded systems where state preservation is held as a feature.

This post is deliberately terse. I welcome questions and feedback.
# Xi Jinping's readout after an AI "study session" [ChinaTalk Linkpost]

[Substack link here](https://www.chinatalk.media/p/xi-takes-an-ai-masterclass)

TL, DR:; Xi Jinping listens to a lecture about AI and publishes his "study notes/takeaways." From the start of the post (emphasis mine)

> On April 25, observers of China’s AI scene got an important new statement of Xi Jinping’s views on AI in the form of remarks concluding a Politburo “study session” on AI led by Xi’an Jiaotong University professor Zheng Nanning. Couched in the turgid language of Partyspeak, the readout nevertheless merits **close attention as one of precious few utterances direct from the General Secretary himself on AI.** To read this new tea leaf, we need to understand some background on study sessions in general, and this one in particular.

I thought this was a phenomenal post. It actually tries to breakdown CCP's partyspeak for the layperson. For example, see this chart below comparing the readout from the 2025 study session (column 2) and the 2018 study session (column 3).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/af716d2ac482f5379de5aa630e48a2b3ea9e2b368cd16957.jpg)

I would recommend people interested in China & AI to read this post.
# Why OpenAI projects only $174B of revenue by 2030?

1.  "OpenAI has told investors it expects to reach $125B in revenue by 2029 and $174B by 2030. OpenAI expects to serve 3B monthly active users by 2030, with 900M DAUs." (from [sacra.com](https://sacra.com/c/openai); original source seems to be [theinformation.com](https://www.theinformation.com/briefings/openai-plans-slash-revenue-share-microsoft-restructuring), but it is behind a paywall).
2.  FAANG performance in 2024 (source: ChatGPT):
    1.  Apple — $391B net sales
    2.  Amazon — $638B net sales
    3.  Alphabet (Google) — $350B revenue
    4.  Meta Platforms — $164.5B revenue
    5.  Netflix — $39B billion revenue
3.  Therefore, OpenAI expects to reach revenue comparable to Meta—or roughly half of Alphabet’s—within five years.
4.  [YC's request for startups](https://www.ycombinator.com/rfs) suggests existence of capable AI agents replacing entire industries: "Instead of selling to the dinosaurs, you could make them extinct."
5.  3 & 4 contradict each other. I am surprised why the projection is so low assuming there are AI agents replacing entire industries.

My hypotheses:

1.  The AI progress is expected to slow down and capable AI agents are not coming in the next 5 years, thus, existing industries are not replaced by AI agents. Basically in (4) YC is shooting far into the future.
2.  Agents are coming, but there will be multiple providers and OpenAI captures only a small part of this market.
    1.  Another version of this - OpenAI only captures API infrastructure part of it, but the rest is captured by smaller companies applying OpenAI's API in their own niches.
3.  Capable agents are coming soon & OpenAI will be the main provider, but OpenAI's revenue is growing exponentially and it just takes time to overtake rest of FAANG. However, based on ChatGPT research, it took FAANG companies 5-10 years to get to a first year with $1B+ revenue & for OpenAI it was 8-9 years, so nothing substantially different here.
# Researching Synthetic Consciousness: sound appealing? 

Without going into the philosophical or technical underpinnings too much, several individuals I know from a wide range of backgrounds—computational neuroscience, computer science, philosophy, AI research—have put together a project where we hope to put some real energy towards studying consciousness in synthetic environments. 

Obviously, if this is an area of interest, you'll likely have questions about whats possible given the landscape of consciousness studies, let alone extending that into non-human systems, but *right now* its nothing more than a group of individuals, being wrapped up into a not-for-profit research body who are;

1.  passionate about the topic
2.  keen to put some real energy towards it.
3.  Have *some* time to give.
4.  And hopefully push things along.

I've set up a very simple website just to collect some interest, but given the community here I thought i'd see if people were interested in being involved. As of right now, we have about 10 people committed to the project in various capacities. 

We have some positions available that are a bit more clearly defined, are currently talking to some individuals who will make up the board of directors, and are hoping to run a kind of research fellowship program where we can put some real meat on the bones. The intake for fellowships will begin in about 8 weeks time (July, 2025).

If this topic, or doing research in this space, sounds interesting. I'd love to hear from you.

More details at [https://alternativemind.org](https://alternativemind.org)
# Staying in a Capsule Hotel

A hotel room generally feels way bigger than I need when traveling: I'm really just looking for somewhere to sleep. When I first read the [Wikipedia page for capsule hotels](https://en.wikipedia.org/wiki/Capsule_hotel), maybe 20y ago, this seemed like a very reasonable approach: lots of people could sleep in a small space, without the downsides of open hostels. I recently travelled to DC for work and found the [U Street Capsule Hotel](https://ustreethostel.com/). Seemed worth trying!

For $82 I got a capsule for a Thursday night:

[![](https://www.jefftk.com/side-entry-hotel-capsule.jpg)](https://www.jefftk.com/side-entry-hotel-capsule-big.jpg)

Some capsules were perpendicular to the hall, which seems a bit more awkward for getting in and out but would use the space more efficiently:

[![](https://www.jefftk.com/longways-hotel-capsules.jpg)](https://www.jefftk.com/longways-hotel-capsules-big.jpg)

They were really very blue. Here's a photo from the inside:

[![](https://www.jefftk.com/inside-capsule-hotel.jpg)](https://www.jefftk.com/inside-capsule-hotel-big.jpg)

Seems like they were designed by people who didn't know or didn't care that blue light is bad for sleep? You could turn off the light, though, so it wasn't that bad.

The thing that I wasn't expecting, however, was the minimal sound isolation. Before trying one of these I had assumed that the point of a capsule instead of bunk beds was noise isolation. While the other people in the room did a good job of being quiet, thin walls plus a window shade for a door did very little to keep out noise. Something more like a horizontal call booth (with [active ventilation](https://www.jefftk.com/p/framery-phone-booth-co2-accumulation), and perhaps a failsafe to automatically open on power loss) would be a lot better.

The ventilation was also hard to control: it was somehow connected to the lights, where I accidentally turned the air off when turning the lights off and didn't realize until ~1am when it got pretty stuffy. This would probably get better with practice, but it could be a lot more intuitive.

Overall it was ok and I might do it again, especially if it were a lot cheaper than other options, but I liked it significantly less than I expected to.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0R9JtSMpeUmQ5NcJ5N1tUGGwzUVKgEWgtfiSWe5rkTUNHSP5jGJ2RVH5QPBEmrxsDl), [mastodon](https://mastodon.mit.edu/@jefftk/114514635719108978), [bluesky](https://bsky.app/profile/jefftk.com/post/3lpaqv6f64s2k)*
# Bay Area Summer Solstice

**What:** Secular celebration of the Summer Solstice. It's a big picnic at the park, basically.

**When:** Saturday, June 21st, the Summer Solstice

**Where:** Point Pinole Regional Park, East Bay

**Who:** Anyone who is reading this. This is for all members of the rationalist, LessWrong, EA, and secular communities. Even if you are not a "typical" LW reader. Even if you've never been to one of these before. 

**Why:** Because ritual and socialization are both important

**How:** [Please RSVP on Partiful Here!](https://partiful.com/e/HYPRwryvylFgSCNGI09t)

The days are getting longer and warmer. Join us on the longest day of the year for a secular celebration of the summer solstice. As the sun reaches its zenith, we will celebrate warmth, light, and community.

This event is for everyone in the rationalist, secular, effective altruist, and adjacent communities.

Activities may include:  
\* Capture the flag  
\* Singing circles  
\* Scavenger hunt  
\* Picnicking  
\* Watching the sunset  
\* Board games  
\* Exploring the park  
\* This event is intended to be kid-friendly for all ages

A variety of events will occur throughout the afternoon, and as the sun suts, we will form a musical procession to the sea.

A $15 donation is suggested but not required to cover the cost of food and other supplies. All are welcome regardless of ability to contribute. Please RSVP on Partiful so we know how much food to get!

If you would like to help organize events or create props like lights to help us celebrate, please post in the comments or email an organizer at richardson.andrew@gmail.com.

Exact location: 37.995928992816715, -122.35838494513749  
https://maps.app.goo.gl/7yK1zpgXTCJB9vpi8
# P-Values Know When You're Cheating

This post is not "the solution to p-hacking" or even "the problem with p-hacking". But I had some recent insights into thinking behind the p-value approach and wanted to see how they apply to p-hacking as an issue. While p-values are a flawed approach to reporting final results, they do provide a useful heuristic for comparing methodologies.

* * *

Basic idea: P-values implicitly punish sloppy methodology.

Suppose there are two different ways of testing a hypothesis that happen to yield the same results. When updating our expectations of future experience we won't see a difference because they have the same results. However, if we use p-values to measure our confidence in the result they will differ relative the methodology used. This is unhelpful for updating expectations, but valuable if you are trying to compare methods.

The key here is that p-values compare all methods against a uniform yardstick, the chance of being fooled under the null hypothesis. If a method is more likely to be tricked by random variation it gets a weaker p-value. 

* * *

Consider then a typical example of p-hacking. Hart is a researcher who suspects that in a given situation dependent variable B is correlated with independent variable A. They run an experiment measuring B against A. But at the same time, they measure and record C, D, E, etc. So, if A ~ B ends up at p = 10%, Hart can keep testing variables until they find C ~ F at p = 4.3% and publish this 'statistically significant' result. 

But here is the rub: The method that Hart used *included* testing variables until they found something significant. Even the flawed reporting method of the p-value approach says has something to say about this. This yet another motivated stopping method. 

Supposing we measure eight variables in total. There are then 28 pair-wise interactions. These do not have 28 separate null hypotheses. When considering Hart's method as a whole, they have one null hypothesis: the combination of all 28 default assumptions. The probability of finding a 'statistically significant' relation somewhere in this dataset is p > 95%^28 = 23.8%. Better than 3/4 times. 

In fact at the 5% level it only takes 6 six variables to have better than 50% odds of finding a spurious result. (This includes only pairwise comparisons. It is even more ridiculous if you include things like 'C mediates the relationship between B and A') At the 1% level this is only 16 variables. 

This is a well understood issue, and proper reporting should include how many variables were available. With 8 variables the standard approach (Bonferroni correction) says we need 0.05/28 = 0.0018 level confidence in any pairwise comparison to be 'statistically significant' overall. 

* * *

P-hacking is often presented as a failure of the p-value framework. But the deeper failure is epistemic: the selective reporting of results, regardless of the statistical tool used. And as we see above, even the frequentist approach has much to say in p-hacking's reproach.
# Events: Debate & Fiction Project

I'm doing a couple of events next week which people might be interested in attending:

* * *

Event: Tsvi & Abram debate AI timelines. 

Abram expects to defend a picture similar to the AI 2027 report. Tsvi expects to defend much longer timelines. 

Monday, May 19 · 10:00 – 11:00am EDT

Video call link: [https://meet.google.com/cbq-kzck-gib](https://meet.google.com/cbq-kzck-gib)

Add to your calendar: [https://calendar.app.google/tdxsJiKQaN9x2JiB7](https://calendar.app.google/tdxsJiKQaN9x2JiB7)

* * *

Event: AI-Powered Fiction Experiment

Abram Demski, TJ, and Daystar Eld will be running an experiment in AI-enabled fiction collaboration. Abram has been using AI for fiction brainstorming, and found a lot of value in quickly catching up with a big pile of notes -- asking the AI questions about a growing pile of notes, rather than trying to pick through it to find anything relevant. This experience made Abram wonder if something magical could happen with a multi-author project, where authors could throw a lot of speculative material into a shared set of files and quickly get up to speed on what each other were doing. Hence the experiment.

The basic fictional premise for this experiment will be a "new type of electricity" which can enable great things, but also has some explosive dangers. (This is intended as an allegory for AGI.) 

The intended output format is interactive fiction, where the player can make some choices determining how the story unfolds. We will also try to have the AI write the code for this.

Audience participation is very welcome! The more people we have dumping ideas into shared files, the better the test.

Time: Tuesday May 20th, 11am EDT, 2 hours long.

Video call link: [https://meet.google.com/ift-jzkc-bys](https://meet.google.com/ift-jzkc-bys)

Add to your calendar: [https://calendar.app.google/sgQttmQvkwUciZ7w9](https://calendar.app.google/sgQttmQvkwUciZ7w9)
# Paranoid Debating

When: June 1st, 2 PM  
Where: JFK Park, Cambridge  
Run by: Mili  
  
Paranoid Debating is a variant of The Aumann Game where one player purposefully subverts the group estimate. Similar to The Aumann Game, the activity consists of a group jointly producing a confidence interval for an unknown, but verifiable quantity, which is then scored for accuracy and calibration. One individual is designated the spokesperson, who is responsible for choosing the final estimate. However, before the activity begins, one individual is secretly assigned the role of misleading the other members. The deceiver is scored higher the worse the final estimate is.  
  
The activity is intended to teach accurate estimate, proper agreement techniques, and recognition of deception."  
  
Mili will be making food for this event and is open to suggestions.   
  
(Note: I'll be out of town myself, I'm just crossposting for Mili! If you need to get in touch with them, use the Boston LW Discord)
# Will we survive if AI solves engineering before deception?

Google's [AlphaEvolve](https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/) has recently started to make real world scientific discoveries, but only in domains where it's very cheap to verify the correct answer (e.g. matrix multiplication algorithms).

But if we can design sufficiently powerful **physics simulations**, mechanical engineering may one day become "cheap to verify." AI which uses similar strategies as AlphaEvolve might be able to engineer far better machines than humans, if the machines are verified by simulations.

These machines may be able to self replicate, and grow exponentially. This could bring near infinite abundance.

There might be no economical pressure to build even smarter AI, because there will already be unlimited resources.

Unfortunately, countries might still have military pressure to build even smarter AI, since the self replicating machines can still be defeated by even better engineered versions.

My hope is that once self replicating machines start to grow exponentially, the world will wake up to the dangerous power of AI, and negotiate a treaty banning other countries from developing their own AI (but promising them a share of the benefits). I'm not sure if this can succeed.

The first country to build self replicating machines technically has the military power to take over the world and enforce a ban unilaterally, but such brute force solutions never lead to good outcomes. However, the mere fact it has this power makes other countries more likely to trust its promise to share the benefits of AI, since other countries will think "if this country really was malicious, and doesn't intend to keep its promise. It could simply take over the world right here and now, and wouldn't need to negotiate with us."

* * *

Do you think humanity will survive if AI solves mechanical engineering before AI solves deception and scheming? Do you think the chances of survival are higher? Would better physics simulations be a net positive?
# Google Logo Ligature Bug

Jeffrey Yasskin recently [pointed out](https://hachyderm.io/@jyasskin/114520767683101962) an [interesting security bug](https://issues.chromium.org/issues/391788835):

[![](https://www.jefftk.com/google-logo-ligature-example.png)](https://www.jefftk.com/google-logo-ligature-example-big.png)

The idea is, if you had registered `googlelogoligature.net` then Chrome on Android (and possibly other Google products) would have displayed it as `Google.net`, potentially tricking users into thinking they were really interacting with Google.

To see how this worked, you can try searching Google for [\["googlelogoligature"\]](https://www.google.com/search?q=%22googlelogoligature%22), and you'll see it shows up as "Google":

[![](https://www.jefftk.com/google-logo-ligature-devtools.png)](https://www.jefftk.com/google-logo-ligature-devtools-big.png)

Poking in devtools, this is dependent on the specific font they're using, "Google Sans". If I turn that off my "googlelogoligature" shows just as I typed it:

[![](https://www.jefftk.com/google-logo-ligature-devtools-off.png)](https://www.jefftk.com/google-logo-ligature-devtools-off-big.png)

Fonts can include "ligatures", which let font designers special-case specific combinations of letters. These were intended to support things like "f" followed by "i" blending into "fi" nicely, but the feature has been (ab)used for many other things, including [complex emoji](https://type.today/en/journal/emoji01). In this case, Google Sans has a specific way of drawing "googlelogoligature" that looks like a mildly stylized "Google".

Using a ligature to get the Google logo into text-only interfaces is a reasonable product decision, but it shouldn't have been added to a general-purpose font. And especially shouldn't have been added to a font used for rendering attacker-controlled text in security-sensitive contexts.

(When I first saw it I thought this might be an example of a [unicode-driven vulnerability](https://www.jefftk.com/p/is-unicode-safe), but sadly not.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0y9mmbyqjWm6bVhAVH8MUmQKuirGiCkxEfFnL2QGrcrDkQqCv1HtvuukqqrYcJsTpl), [mastodon](https://mastodon.mit.edu/@jefftk/114526485794379433), [bluesky](https://bsky.app/profile/jefftk.com/post/3lpfzcbfi4c2w)*
# [Paper] Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methods

"*If you cannot measure it, you cannot improve it.*" \- Lord Kelvin

The [science of AI safety evaluations](https://www.apolloresearch.ai/blog/we-need-a-science-of-evals) is still nascent, but it is making progress! We know much more today than we did two years ago.

We tried to make this knowledge accessible by writing a literature review and systematizing all the knowledge.

We wanted to go beyond a central source that just summarized collected knowledge, so we put a lot of effort into distillation, and disentangling concepts that are often presented in a messy way. We created original visualizations and gathered others from many different sources to accompany these explanations.

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXf5iVoGr4c_n42Tr2xYaYngraWOEghMW8zZKxVzDVi62QpUXNN6H6CySiVnGG5Xxl4arViH_jGxZL1owPuzNvpDmbl6iT5EKcpy4FVOdlVMDNJAB6SKecjS_ssZgnkvc4B19wM56g?key=mBFj73k30rETVYQl70qLzg)

E.g. Disentangling truth, honesty, hallucination, deception and scheming.

The review provides a taxonomy of AI safety evaluations along three dimensions:

*   What properties do we measure? (dangerous capabilities, propensities, and control)
*   How do we measure them? (Behavioral and internal techniques)
*   How do we integrate evaluations into broader frameworks? (Model Organisms , Responsible Scaling Policies, ...)

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXd5TLzjW-dzitwZCxU6ITPDd0CuQikCob2z9nUoWOoQgQuW7o2x8jWNzWI4iMe3FN9qDlVjCsIaYcVycQlkp4blIJajmS_kfDBEX9VIFK3y_I104EfyiyGT3wC7lKPmXzk-SDn8KA?key=mBFj73k30rETVYQl70qLzg)

We also bring up some limitations that safety evaluations might face including things like: "sandbagging" (strategic underperformance on tests), organizational "safetywashing" (misrepresenting capability improvements as safety advancements), or other more fundamental inherent challenges like proving absence rather than presence of capabilities.

The text is available in many different places:

*   [Arxiv](https://arxiv.org/abs/2505.05541)
*   [Google Docs (Open to comments)](https://docs.google.com/document/d/1KI95w27Ce7yWoynE11PJ94IXK0gT0NwP8091s06P7wM/edit?tab=t.0)
*   [AI Safety Atlas](https://ai-safety-atlas.com/chapters/05/)

This paper is intended to be part of a larger body of work called the AI Safety Atlas. We think of it as chapter 5 in a comprehensive collection of literature reviews collectively forming a textbook for safety.
# [linkpost] One Year in DC

(h/t Otis Reid)

I think this post captures a lot of important features of the US policymaking system. Pulling out a few especially relevant/broadly applicable sections:

> **1\. There's No Efficient Market For Policy**
> 
> There can be a huge problem that nobody is working on; that is not evidence that it's not a huge problem. Conversely, there can be a marginal problem swamped with policy work; that's not evidence it's really all that big of a deal.
> 
> On the upside, this means there are never-ending arbitrage opportunities in policy. Pick your workstreams wisely.

> **2\. Personnel Really Is The Most Important Thing**
> 
> **The quality of staffers varies dramatically and can make or break policy efforts.** Some Hill staffers are just awesome; if they like your idea, they'll take it and run with it, try to find the right cosponsors, understand where it fits procedurally, etc. Other staffers have absolutely no idea what's going on.

> **3\. The Hill is Not for Wonking**
> 
> \[...\]
> 
> **Optimize for impact, not panel invites.** This may seem obvious, but this is not how most (c)(3)s are set up. Many large think tanks conceive of themselves as a "university without students," instead of a policy shop engaging with day-to-day policymaking. Others want to be a convening organization, spending resources on capital-C Conversations between capital-E Experts in Congress. And at an individual level, many think tankers simply do not like taking Hill meetings. There’s nothing *wrong* with this – but it is not a great strategy for driving policy change.

> **4\. There's Alpha In Doing Boring Stuff**
> 
> My colleague Lars Schönander has had enormous policy success because he sends hundreds (thousands?) of FOIA requests to state and local governments to track malign foreign influence over critical infrastructure. Much of his success comes from being willing to spend hours every day emailing FOIAs to places like the San Jose Port Authority.
> 
> Similarly, my summer and early fall were spent on phone lines with state departments of environmental quality, tracking their use of flexible permits. It was objectively soul-destroying work; it also led to an [executive order](https://governor.utah.gov/wp-content/uploads/Executive-Order-2025.01.07-Expanding-and-Improving-Permitting-Efficiency-at-DEQ-and-DOGM.pdf), a [bill being enacted into law](https://le.utah.gov/~2025/bills/static/HB0085.html), and several other bills being introduced.

(related [EA Forum post](https://forum.effectivealtruism.org/posts/J7nmbqcWncPMZFhGC/want-to-make-a-difference-on-policy-and-governance-become-an))
# Scroll Snapping

It's been several years since paying attention to the web platform was a big part of my full time job, and I was curious if I'd missed any interesting additions. I did some poking around with Claude and ended up finding something neat, but a bit different than I was looking for: `scroll-snap-type`.

Scrolling through long lists of things I've often been mildly frustrated at how much precision I need. I want to move down a bit, perhaps to see the next post in a feed, but without going a bit too far and losing the first few lines. Looking through Claude's output I saw:

> Scroll snap: Control scroll positioning with `scroll-snap-type` (2019)

Looking [on MDN](https://developer.mozilla.org/en-US/docs/Web/CSS/scroll-snap-type) it seems to offer just what I was looking for: scrolling that tries to avoid splitting content.

The main place on my website where I've wanted this are [my pictures](https://www.jefftk.com/pictures/2024/), and I've implemented it there. Give it a try!

The code changes were really simple:

html, body {
  scroll-snap-type: y proximity;
  scroll-padding: 0;
}
figure {
  scroll-snap-align: start;
}

While I didn't see other things in Claude's list of newish features that seemed exciting, is there anything else neat I might have missed?
# Mental software updates

Brains are like computers in that the hardware can do all kinds of stuff in principle, but each one tends to run through some particular patterns of activity repeatedly. For computers you can change this by changing programs. What are big ways brain ‘software’ changes?

Some I can think of:

*   Intentional practice of different styles of thinking (e.g. meditation)
*   Intentional practice of different trains of thought in response to specific stimuli (e.g. CBT, self-talk)
*   Changing the high level situation, where your brain automatically has different patterns for each (e.g. if you go from feeling like a child to like an adult maybe a lot of patterns change)
*   A change in a major explicit belief (e.g. if you go from expecting your project to work out to believing otherwise, your patterns of attention might naturally change)
*   Learning that the world isn’t as you intuited (e.g. if you are constantly worrying about people wronging you, but everyone is kind to you, this worry might become unappealing)
*   Intense experiences causing inaccurate updating (e.g. trauma)
*   Identifying differently (e.g. if I think of myself as a good student, I might have different mental patterns around studying than when I thought of myself as a bad student)
*   Adopting a new goal (e.g. deciding to be a musician)
*   Getting a new responsibility (e.g. a child)
*   Getting a new obsession (e.g. a crush, a hobby)
*   Changing social groups (e.g. among jokers it is more tempting to think of jokes, though in my experience among philosophers it might have been less tempting to think of philosophy)
*   Interacting with a really compelling person
*   Drugs (e.g. alcohol, adderall, LSD both short term and long-term)
*   Religion, somehow

I feel like people talk about many of these as important, but not in one view. I rarely hear someone say, “My brain software seems suboptimal, what are my options for changing it?”, then go down the list. Instead I suppose one hears from a friend that this book helped them, or one decides to have a therapist, and the book or therapist turns out to be one that focuses on CBT, so one does that. Or one changes social groups or responsibilities for some other reason, then remarks that this was a good or bad side-effect. Maybe that makes sense—‘stuff my brain habitually does’ is pretty broad.  
  
I’d be interested to know which things do in practice change people’s mental patterns the most.
# President of European Commission expects human-level AI by 2026

On May 20, during her speech at the Annual EU Budget Conference 2025, Ursula von der Leyen, President of the European Commission, stated:

> When the current budget was negotiated, we thought AI would only approach human reasoning around 2050. Now we expect this to happen already next year. It is simply impossible to determine today where innovation will lead us by the end of the next budgetary cycle. Our budget of tomorrow will need to respond fast.

This is remarkable coming from the highest-ranking EU official. It suggests the Overton window for AI policy has shifted significantly.
# Gemini Diffusion: watch this space

Google Deepmind has announced Gemini Diffusion. Though buried under a host of other IO announcements it's possible that this is actually the most important one!

This is significant because diffusion models are entirely different to LLMs. Instead of predicting the next token, they iteratively denoise all the output tokens until it produces a coherent result. This is similar to how image diffusion models work.

I've tried they results and they are surprisingly good! It's incredibly fast, averaging nearly 1000 tokens a second. And it one shotted my Google interview question, giving a perfect response in 2 seconds (though it struggled a bit on the followups).

It's nowhere near as good as Gemini 2.5 pro, but it knocks ChatGPT 3 out the water. If we'd seen this 3 years ago we'd have been mind blown.

Now this is wild for two reasons:

1.  We now have a third species of intelligence, after humans and LLMs. That's pretty significant in and of itself.
2.  This is the worst it'll ever be. This is a demo, presumably from a relatively cheap training run and way less optimisation than has gone into LLMs. Diffusion models have a different set of trade offs to LLMs, and once benchmark performance is competitive it's entirely possible we'll choose to focus on them instead.

For an example of the kind of capabilities diffusion models offer that LLMs don't, you don't need to just predict tokens after a piece of text: you can natively edit somewhere in the middle. Also since the entire block is produced at once, you don't get that weird behaviour where an LLM says one thing then immediately contradicts itself.

So this isn't something you'd use just yet (probably), but watch this space!