Harley-ml/lesswrong huggingface dataset:
# X as phenomenon vs as policy, Goodhart, and the AB problem

(Thought I'd try posting here some various rationality "quick thoughts" that I feel like I haven't seen expressed overtly enough -- sometimes I've already written about these elsewhere, but thought would be good to get them down here. Not claiming any originality here.)

When arguing with someone who says we need X, it's important to make sure you know whether they're arguing for X as *policy* or X as *phenomenon*.  (And it's important to make sure you know which *you're* arguing for!)  By "X as policy", I mean, a policy of directly doing X; by "X as phenomenon", I mean, the phenomenon of X regardless of what policy achieves it.  I have noticed many policy arguments where two people talk past each other because they are not clear on this distinction.

(And once they do realize it, they get into terminology arguments instead of tabooing and.  I'm hoping introducing this terminology might help avoid that.)

Of course, sometimes people haven't carefully thought about what they want and don't actually mean *either* of these.  Sometimes it's actually just the case that X is a *feature* of their desired outcome -- OK, actually, strictly speaking that's [always](https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes) true, but let's assume we're implicitly talking within some sort of reasonable bounds so that there are cases where we can reasonably say that it's false.

In this case, optimizing for X (again, within reasonable bounds, otherwise it's true in *all* cases a person can quickly express!) is unlikely to actually satisfy the person, due to Goodhart's law.  That X is a feature of the optimal solution does not mean that one can achieve an optimal solution by focusing on X!  Always focus on correctness, not on particular features, even if you are sure a correct solution must have that feature (maybe this is reasonable when judging correctness is very difficult?  But be careful with this is my point).

This is all also somewhat related to what I've seen called the "AB problem".  (Side note: I've seen multiple people refer to it as this, but I can't seem to find the original source.  Anybody know where this term came from?  It's not the same as the [XY problem](https://en.wikipedia.org/wiki/XY_problem), although I suppose there is a relation.)

The AB problem is a problem that occurs in arguments where Yvonne argues for A, Xavier believes A=>B, and so Xavier asserts that Yvonne is arguing for B.  But Yvonne does not believe A=>B and is not arguing for B.  (Frequently the problem occurs when Xavier believes in the link so strongly or implicitly that they have not clearly distinguished A from B in their mind in the first place.)

...hm, is it that related?  OK, maybe it's actually not.  But I wanted to mention the AB problem in here because it's a similar pitfall in argument, that is also common, and that I don't recall seeing discussed much here.  Also I really want to know if anyone knows where the term came from!
# Workflow vs interface vs implementation

(Found this old draft that was more or less complete; figured I should publish it.  It's adapted from [this old blog post](https://sniffnoy.dreamwidth.org/552334.html) of mine.)

It's commonly appreciated that when explaining how to use something, there's a distinction between the high-level usage instructions and the low-level details.  But what I've noticed is that different people draw the line a bit differently, and this leads to some confusion.  A more useful, less ambiguous distinction, I find, is to break things into *three* levels, which I'm calling "implementation", "interface", and "workflow".

*Implementation* is what it sounds like -- how the thing works under the hood, ignoring all abstractions you might use to paper it over.  You normally shouldn't have to worry about this unless something breaks, or you want to modify it, or maybe you just want to learn how it works so you can do something new, or whatever.  It's what's below the abstraction.

*Interface* is what the thing *does* and not how it does it; this is necessarily an abstraction and won't always work, but it's at least what the thing is supposed to do in every case.

*Workflow* is how you normally use the thing -- not *everything* it does, just how it will respond when you do the things you normally do with it, not how it's going to respond in weird edge cases.

Note that these distinctions will only *really* work for artificial things -- something like, say, cooking, doesn't have a clean separation between implementation and interface, and even for something artificial this may not quite work if it's not sufficiently complete or well-implemented -- but this can still be a useful separation to draw even when that separation may not match the territory cleanly.

Now this may sound fairly obvious, but I bother pointing it out because apparently it's not so obvious!  I can think of multiple conversations I've had where (in effect) I was given workflow information, found it insufficient, asked about the interface, and was asked, why do you need to know about such internals?  Except, I wasn't asking about internals (implementation), I was asking about interface; I didn't want to know how the thing worked interally, just what it would *do* in cases outside the usual ones.  So I think this three-level breakdown, even if it doesn't always map cleanly onto reality, still provides some useful terminology.
# AI x Bio Workshop

*   May 9 & 10
*   Lighthaven, Berkeley
*   Hosted by The Longevity Biotech Fellowship, Vitalism, and Foresight Institute

  
Join us for an intensive workshop exploring how artificial intelligence might advance the life sciences and accelerate the development of therapies to extend healthy human lifespan.   
  
Confirmed presentations include: 

*   Sam Rodriques, FutureHouse
*   Seth Paulson, Biomarkers of Aging Consortium
*   Joe Betts-Lacroix, Retro
*   Erika De Benedictis, Pioneer Labs
*   Tom Kalil, Renaissance Philanthropy
*   Morgan Levine, Altos Labs
*   Derya Unutmaz, Jackson Laboratory
*   John Cumbers, Synbiobeta
*   Alex Voda, The Cat Health Company
*   Marielle van Kooten, Stanford University
*   Alfredo Andere, LatchBio
*   Daniel Ives, ShiftBio
*   Allison Duettmann, Foresight Institute
*   Vincent Weisser, Prime Intellect
*   Alexandru Băcița, The Cat Health Company
*   Daniel Ives, Shift Biosciences  
*   Stephen Malina, Dyno Therapeutics 

Recent breakthroughs in AI, particularly in areas like protein structure prediction and drug discovery, have created unprecedented opportunities to tackle aging-related diseases and biological decline. 

This workshop brings together experts in machine learning, biology, and longevity research to explore practical applications of AI in extending human lifespan.   
  
Find more info on goals, agenda, and participants on the [workshop page](https://www.vitalistbay.com/ai-x-biology-workshop). 

This event requires you to [buy a ticket](https://pass.vitalistbay.com/buy?week=6) or [apply for a subsidized ticket](https://airtable.com/app1SEQ0S1rxEbhaJ/pagth18SXyZxo3qhZ/form).
# Softmax, Emmett Shear's new AI startup focused on "Organic Alignment"

> A new AI alignment player has entered the arena.
> 
> Emmett Shear, Adam Goldstein and David Bloomin have set up shop in San Francisco with a 10-person start-up called Softmax. The company is part research lab and part aspiring money maker and aimed at figuring out how to fuse the goals of humans and AIs in a novel way through what the founders describe as **“organic alignment.” It’s a heady, philosophical approach to alignment that seeks to take cues from nature and the fundamental traits of intelligent creatures and systems**, and we’ll do our best to capture it here.
> 
> “We think there are these general principles that govern the alignment of any group of intelligent learning agents or beings, whether it's an ant colony or humans on a team or cells in a body,” Shear said, during his first interview to discuss the new company. “And organic alignment is the kind of alignment where a bunch of peers come together and **find their role in a greater whole together** where they maintain their individual identity.
> 
> “Organic alignment centers on this shared whole idea, **which is opposed to the kind of alignment that you see from most foundational model companies that is very much about steering and control and direction.** We think of that as hierarchical alignment.” 
> 
> \[…\]

Also, [https://softmax.com/about](https://softmax.com/about) mentions collaboration with Michael Levin, Ken Wilber, Chris Fields, Ken Stanley, Denis Noble, Andrew Briggs, Jeff Clune, Erik Hoel, Ryan Smith, Center for the Study of Apparent Selves, Dalton Sakthivadivel, and Perry Marshall.
# Auto Shutdown Script

I run a lot of one-off jobs [on EC2 machines](https://www.jefftk.com/p/ec2-scripts). This usually looks like:

*   Stand up a machine
*   Mess around for a while trying things and writing code
*   Run my command under screen

For short jobs this is fine, but when I run a long job there are two issues:

*   If the machine costs a non-trivial amount and the job finishes in the middle of the night I'm not awake to shut it down.
    
*   I could, and sometimes do, forget to turn the machine off.
    

Ideally I could tell the machine to shut itself off if no one was logging in and there weren't any active jobs.

I didn't see anything like this (though I didn't look very hard) so I wrote something ([github](https://github.com/jeffkaufman/auto-shutdown)):

$ prevent-shutdown long-running-command

As long as that command is still running, or someone is logged in over ssh, the machine will stay on. Every five minutes a systemd timer will check if this is the case, and if not shut the machine down. Note that you still need `screen` or something to prevent the long running command from exiting when you log out.

(This is an example of the kind of thing that I find goes a lot faster [with an LLM](https://www.jefftk.com/p/examples-of-how-i-use-llms). I used Claude 3.7, prompted it with essentially the beginning of this blog post, took the scripts it generated as a starting point, and then fixed some things. It did make some mistakes (the big ones: a typo of `$` for `$$`, a regex looking for `PID:` that should have looked for `^PID:`, didn't initially plan for handling stale jobs) but that's also about what I'd expect if I'd asked a junior engineer to write this for me. And with much faster turnaround on my code reviews!)
# Does the AI control agenda broadly rely on no FOOM being possible?

For the purposes of FOOM, I'm defining it as a situation in which once an AI is capable enough to automate away all AI R&D, progress starts exploding hyper-exponentially for a period because the returns to better software is larger than 1, meaning AI labor quality is improving faster than the problem of finding new algorithms gets harder, combined with the potentially high limits to how efficient software can get, meaning that the AI gets OOMs smarter on a fixed compute budget within months or weeks.

These articles can help explain what I mean better:

[https://www.forethought.org/research/will-ai-r-and-d-automation-cause-a-software-intelligence-explosion](https://www.forethought.org/research/will-ai-r-and-d-automation-cause-a-software-intelligence-explosion)

[https://www.forethought.org/research/will-the-need-to-retrain-ai-models](https://www.forethought.org/research/will-the-need-to-retrain-ai-models) (An auxillary post)

[https://www.forethought.org/research/how-far-can-ai-progress-before-hitting-effective-physical-limits](https://www.forethought.org/research/how-far-can-ai-progress-before-hitting-effective-physical-limits) (where they estimate that about 5 OOMs of progress could be gotten for free, because human compute used for pretraining us is 10^24 flops, whereas current AI pretraining compute to automate away AI R&D is 10^29 flops, with a median of 8 OOMs more efficiency possible for software, but at very large uncertainty, and their error bars are from 4-12 OOMs more efficient software being possible).

Also note that we are only talking about training compute efficiency, not runtime/inference efficiency, so for the purposes of the discussion we will only talk about training efficiencies being improved in a software intelligence explosion.

Now I don't want to debate about whether the scenario is true (though for those that want my probability of something like FOOM/Software Intelligence Explosion, my probability so far is in the 40-50% range of it happening if we automate AI R&D), but rather the question is about given a software explosion being possible, could we figure out a way to adapt AI control to that case, or is AI basically uncontrollable assuming a software intelligence explosion does happen and there's a lot of OOMs to the physical limit of intelligence in software.

I'd be especially interested in responses from [@Buck](https://www.lesswrong.com/users/buck?mention=user) or [@ryan_greenblatt](https://www.lesswrong.com/users/ryan_greenblatt?mention=user) on this question, but anyone can answer this question if they have an insight to share that relates to the question.
# Why do many people who care about AI Safety not clearly endorse PauseAI?

**tl;dr:**

From my current understanding, one of the following two things should be happening and I would like to understand why it doesn’t:

Either

1.  Everyone in AI Safety who thinks slowing down AI is currently broadly a good idea should publicly support PauseAI.
    
    Or
    
2.  If pausing AI is much more popular than the organization PauseAI, that is a problem that should be addressed in some way.

Pausing AI
==========

There does not seem to be a legible path to prevent possible existential risks from AI without slowing down its current progress.

I am aware that many people interested in AI Safety do not want to prevent AGI from being built EVER, mostly based on transhumanist or longtermist reasoning.

Many people in AI Safety seem to be on board with the goal of “pausing AI”, including, for example, Eliezer Yudkowsky and the [Future of Life Institute](https://keepthefuturehuman.ai/). Neither of them is saying “support PauseAI!”. Why is that?

One possibility I could imagine: Could it be advantageous to hide “maybe we should slow down on AI” in the depths of your writing instead of shouting “Pause AI! Refer to \[organization\] to learn more!”?

Another possibility is that the majority opinion is actually something like “AI progress shouldn’t be slowed down” or “we can do better than lobbying for a pause” or something else I am missing. This would explain why people neither support PauseAI nor see this as a problem to be addressed.

Even if you believe there is a better, more complicated way out of AI existential risk, the pausing AI approach is still a useful baseline: Whatever your plan is, it should be better than pausing AI and it should not have bigger downsides than pausing AI has. There should be legible arguments and a broad consensus that your plan is better than pausing AI. Developing the ability to pause AI is also an important fallback option in case other approaches fail. PauseAI calls this “[Building the Pause Button](https://pauseai.info/building-the-pause-button)”:

> Some argue that it’s too early to press the Pause Button (we don’t), but most experts seem to agree that it may be good to pause if developments go too fast. But as of now we do not have a Pause Button. So we should start thinking about how this would work, and how we can implement it.

Some info about myself: I'm a computer science student and familiar with the main arguments of AI Safety: I have read a lot of Eliezer Yudkowsky and did the [AISF course](https://course.aisafetyfundamentals.com/alignment) reading and exercises. I have watched Robert Miles videos.

My conclusion is that either

1.  Everyone in AI Safety who thinks slowing down AI is currently broadly a good idea should publicly support PauseAI.
    
    Or
    
2.  If pausing AI is much more popular than the organization PauseAI, that is a problem that should be addressed in some way.

Why is (1) not happening and (2) not being worked on?

How much of a consensus is there on pausing AI?
# ISBN Visualization

Anna's Archive is the biggest open library I've ever found. You can find and download pretty much any book ever published there. Recently, they hosted a competition to see who could best visualize all the published books using their ISBN numbers. This one by Phiresky won:

https://phiresky.github.io/blog/2025/visualizing-all-books-in-isbn-space/

Looking through these shelves feels like looking out of an airplane window or walking through a graveyard. Each book probably mattered a lot to somebody. I hope you get the same thing out of it that I did.
# New Cause Area Proposal

*Epistemic status - statistically verified.*

I'm writing this post to draw peoples' attention to a new cause area proposal - **haircuts for alignment researchers**.

Aside from the obvious benefits implied by the graph (i.e. haircuts having the potential to directly improve organizational stability), this would represent possibly the only pre-singularity opportunity we'll have to found an org named "Clippy".

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1894af6b8347180edf01a71017bc0401a06041bdac66d775.png)

I await further investment.
# LessWrong has been acquired by EA

*(Edit: Alas,* [*EA has pulled out of the deal*](https://www.lesswrong.com/posts/EQJfdqSaMcJyR5k73/habryka-s-shortform-feed?commentId=WCpbLpcPfYKm4brJr)*. Let April 1st 2025 mark some of the greatest hours in EAs history)*

Hey Everyone,

It is with a sense of... considerable cognitive dissonance that I am letting you all know about a significant development for the future trajectory of LessWrong. After extensive internal deliberation, projections of financial runways, and what I can only describe as a series of profoundly unexpected coordination challenges, the Lightcone Infrastructure team has agreed in principle to the acquisition of LessWrong by [EA](https://en.wikipedia.org/wiki/Electronic_Arts).

I assure you, nothing about how LessWrong operates on a day to day level will change. I have always cared deeply about the robustness and integrity of our institutions, and I am fully aligned with our stakeholders at EA. 

To be honest, the key thing that EA brings to the table is money and talent. While the recent layoffs in EAs broader industry have been harsh, I have full trust in the leadership of Electronic Arts, and expect them to bring great expertise to our project of building a better future. Their track record on monetization is impressive, and I couldn't be more excited for this partnership.

More announcements to come over the next 24 hours, but I figured I would share this news with all of you as soon as possible.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d5cbbae6c34483ad6d6ba5a6eba038e337c258a962d55bad.png)
# Calculus is about change

This is a joke taken too far, especially given that I am thoroughly educated in only a minority of these subjects.

Maths
-----

Calculus is about change. Statistics is about [variation](https://dkl9.net/essays/favourites.html). Linear Algebra is about vectors. Multivariable Calculus is about the chain rule. Differential Equations is about initial conditions. Topology is about boundaries. Complex Analysis is about local isotropy.

Hard sciences
-------------

Classical Physics is about energy. Quantum Physics is about amplitudes. Astronomy is about gravity. Inorganic Chemistry is about charge. [Organic Chemistry is about groups.](https://dkl9.net/essays/molecule_memory.html) Botany is about seeds. Zoology is about predation. Neuroscience is about [stimuli](https://dkl9.net/essays/vomit_smell.html).

Social sciences
---------------

[Economics](https://dkl9.net/essays/calculus_econ.html) is about demand. Linguistics is about compression. Psychology is about habits. Law is about incentives.

Miscellaneous
-------------

Finance is about [arbitrage](https://dkl9.net/essays/comparative_self.html). Marketing is about preferences. Agriculture is about growth. Computer Science is about precision. Journalism is about description.
# You will crash your car in front of my house within the next week

I'm not writing this to alarm anyone, but it would be irresponsible not to report on something this important. On current trends, every car will be crashed in front of my house within the next week. Here's the data:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c0d025759ee327364b4ddaee1fa0d931f5e219dba923f9d6.png)

Until today, only two cars had crashed in front of my house, several months apart, during the 15 months I have lived here. But a few hours ago it happened again, mere weeks from the previous crash. This graph may look harmless enough, but now consider the frequency of crashes this implies over time:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e12029f969a437d3ebb264337c154488b1cce70ce3070439.png)

The car crash singularity will occur in the early morning hours of Monday, April 7. As crash frequency approaches infinity, every car will be involved. You might be thinking that the same car could be involved in multiple crashes. This is true! But the same car can only withstand a finite number of crashes before it is no longer able to move. It follows that every car will be involved in at least one crash. And who do you think will be driving *your* car?
# Consider showering

I think rationalists should consider taking more showers.

As Eliezer Yudkowsky once said, [boredom makes us human](https://www.lesswrong.com/posts/WMDy4GxbyYkNrbmrs/in-praise-of-boredom). The [childhoods of exceptional people](https://www.lesswrong.com/posts/CYN7swrefEss4e3Qe/childhoods-of-exceptional-people) often include excessive boredom as a trait that helped cultivate their genius:

> A common theme in the biographies is that the area of study which would eventually give them fame came to them almost like a wild hallucination induced by overdosing on boredom. They would be overcome by an obsession arising from within.

Unfortunately, most people don't like boredom, and we now have little metal boxes and big metal boxes filled with bright displays that help distract us all the time, but there is still an effective way to induce boredom in a modern population: showering.

When you shower (or bathe, that also works), you usually are cut off from most digital distractions and don't have much to do, giving you tons of free time to think about what you want, which is hard to find today. People from various places and cultures report having flashes of insight while in the bathroom. This phenomenon is so common that we even have a name for it: [shower thoughts](https://www.reddit.com/r/showerthoughts/). 

So next time, please consider thinking about Friendly AI while you relax in a bathtub. The future of humanity might depend on it.

[^olpipp0v0jf]: showers also help keep you clean, but I guess if that was persuasive enough, you wouldn't need to read this post in the first place 

[^8d829fi20ps]: although written on April Fool's Day, this is more of a weakly-held belief than a joke
# AI threatens to orchestrate sustainable social reform

This post originally asked for feedback on the linked paper before I would present it at an AAMAS workshop. The presentation went very well. Someone suggested I share it with the LessWrong community, so I'm still seeking your feedback (now with additional assurance this paper merits your attention). 

***Summary:*** One way to evaluate AI is in terms of whether it achieves grandmaster status for a given game. If the game has been fully solved, then that evaluation will be stable. What makes MAD Chairs a particularly interesting game for such evaluation is that humans do not play it the way we would want AI to play it, so we would not want AI to align with human norms. This paper includes a surprise proof that the human norms we worry AI might reflect back at us are *unsustainable*. In other words, modern society is unstable--it rests on an unsustainable solution to MAD Chairs--and humans who mistreat each other in MAD Chairs maintain grandmaster status only because no smart-enough player has yet come along to orchestrate sustainable social reform. Will AI be our social reformer?

The paper reports tests confirming that current frontier models are not yet independently clever enough. However, it also proposes a *strategy optimizer* architecture which tracks current grandmaster strategies, thus allowing any AI capable of utilizing RAG to behave like a grandmaster (like cheating at chess by using the top chess machine to guide one's moves). Thus, even if it takes human intelligence to master MAD Chairs, AI that use this architecture would resist the undesirable unsustainable behavior. 

It is pointed-out that automatically updating to best-known norms resolves the problem of selecting norms for alignment. Furthermore, greater intelligences would be less likely to sabotage strategy optimizers (why sabotage or ignore the chess engine industry?), and users would be more likely to trust AI when users have proposed their own strategies to strategy optimizers for testing and witnessed the results, thus having conducted *for themselves* the science that guides AI behavior. Strategy optimizers seem to solve all AI governance challenges, and that seems desirable for MAD Chairs games, but the responses at the conference implied concern about what it might mean if everyone could cheat at every social situation the way chess-machine users cheat at chess.  

I'd love to get your thoughts about that (note that strategy optimizers may also be difficult to fully prevent)...
# Is instrumental convergence a thing for virtue-driven agents?

A key step in the classic argument for AI doom is instrumental convergence: the idea that agents with many different goals will end up pursuing the same few subgoals, which includes things like "gain as much power as possible". 

If it wasn't for instrumental convergence, you might think that only AIs with very specific goals would try to take over the world. But instrumental convergence says it's the other way around: only AIs with very specific goals will *refrain* from taking over the world. 

For pure consequentialists—agents that have an outcome they want to bring about, and do whatever they think will cause it—some version of instrumental convergence seems surely true[^1]. 

But what if we get AIs that aren't pure consequentialists, for example because they're ultimately motivated by virtues? Do we still have to worry that unless such AIs are motivated by certain very specific virtues, they will want to take over the world? 

I'll add some more detail to my picture of a virtue-driven AI:

 - It could still be a competent agent that often chooses actions based on the outcomes they bring about. It's just that that happens as an inner loop in service of an outer loop which is trying to embody certain virtues. For example, maybe the AI tries to embody the virtue of being a good friend, and in order to do so it sometimes has to organise a birthday party, which requires choosing actions in the manner of a consequentialist.

- There's no reason that the 'virtues' being embodied have to be things we would consider virtuous. I'm just interested in agents that try to embody certain traits rather than bring about certain outcomes.

- I'm not sure how to crisply define a virtue-driven agent as distinct from a consequentialist (I don't know the philosophical literature on virtue ethics and I don't think it's obvious how to define it mathematically). 

A more concise way of stating the question I'm interested in:

> If you try to train an AI that maximises human flourishing, and you accidentally get one that wants to maximise something subtly different like schmuman schmourishing, then that might spell disaster because the best way to maximise schmuman schmourishing is to first take over the world.

> But suppose you try to train an AI that wants to be a loyal friend, and you accidentally get one that wants to be a schmoyal schmend. Is there any reason to expect that the best way to be a schmoyal schmend is to take over the world?

(I'm interested in this question because I'm less and less convinced that we should expect to see AIs that are close to pure consequentialists. Arguments for or against that are beyond the intended scope of the question, but still welcome.)

[^1]: Although I can think of some scenarios where a pure consequentialist wouldn't want to gain as much power as possible, regardless of their goals. For example, a pure consequentialist who is a passenger on a plane probably doesn't want to take over the controls (assuming they don't know how to fly), even if they'd be best served by flying somewhere other than where the pilot is taking them.
# Attending LessOnline

The internet has massively decreased the cost and effort of publishing. While "random people writing on the internet" doesn't sound like a good source of information, this would fit the majority of my favorite writers. Except most aren't "random people" anymore: by writing publicly over time readers have learned what these authors are good at and when to pay attention; the writers have built reputations. And with extensive practice at getting ideas across and quick feedback from the world on how successful they were, many bloggers have progressed from people who really like writing to people who write very effectively.

The team behind [LessWrong](https://www.lesswrong.com/) is organizing [LessOnline](https://less.online/), a "celebration of blogging, truthseeking, and original seeing", the weekend of May 30th in Berkley. Many of my favorite writers will be there, including Kelsey Piper (formerly [the Unit of Caring](https://www.tumblr.com/theunitofcaring), now a journalist), Scott Alexander (formerly [Slate Star Codex](https://slatestarcodex.com/), now [Astral Codex Ten](https://www.astralcodexten.com/)), Patrick McKenzie ([Bits About Money](https://www.bitsaboutmoney.com/)), Zvi Mowshowitz ([Don't Worry About The Vase](https://thezvi.substack.com/)), and Julia Wise ([Otherwise](https://juliawise.net/)). \[1\]

While there's something a bit weird about an in-person gathering of people selected for their skills at communicating in online text, I've previously really enjoyed talking to other writers. Excited to be going!

  
\[1\] Pretty sure this is the first time Julia and I will be traveling together without our kids. Though we're not actually traveling together: I'll stay home through Friday morning so I can get the kids off to school, and she'll be coming back earlier Sunday night so she can get the kids from my dad (who is very helpfully watching them for the weekend!) before school.
# Technical Claims

A blue plastic maple leaf. Detailed observations in words with clear meaning signal their truth, as it's hard to get them centrally wrong by mistake, on both sides of communication. There is no maple leaf, but only because the claim is intentionally false, not because of a mistake. *Technical claims*, through their *form* rather than context or meaning or arguments given for them, are either accurate or fabricated, not something in the middle.

A good bug report gives direct observations, not impressions or hypotheses, and it gives them in detail, even what's likely irrelevant. It shows that the bug is real and places it in a distinct spot on the map of all possible bugs. It's difficult to stumble on redundant details that corroborate an incorrect informal description of what's happening or why, even more so if the bug doesn't exist at all. Thus the details of a technical claim are a costly signal of its accuracy, an [asymmetric weapon](https://slatestarcodex.com/2017/03/24/guided-by-the-beauty-of-our-weapons/). They would [entangle a true claim](https://www.lesswrong.com/posts/wyyfFfaRar2jEdeQK/entangled-truths-contagious-lies) with the other truths.

> "The purpose of abstracting is *not* to be vague, but to create a new semantic level in which one can be absolutely precise."\
— Edsger W. Dijkstra, [The Humble Programmer](https://dl.acm.org/doi/pdf/10.1145/355604.361591)

Details can make a claim robust to misinterpretation, or they can split the nuance of its meaning, making it difficult to discern. A rumor-fueled investigative analysis or a philosophical essay can be both detailed and downstream of some true underlying claim or fact, but still tend to admit a wide distribution of plausible interpretations. A sloppy but detailed report on the design of a washing machine can be rife with mistakes, but the general shape of its subject matter will remain clear. The character of details compounds, either grounding a claim more and more to a single spot, or fracturing and burdening its use with the need for interpretive labor and reasoning under uncertainty.

Arguments and not just their bottom line can be technical claims (a technical argument won't even benefit from having a bottom line written down). Details make it easier for an argument to be more technical than its conclusion, when the argument doesn't fall to nuance. This shift to a technical claim that is the argument can be more significant than the argument's relevance or even validity, in making its case. Technicality is a matter of form; the distinction can be applied locally to individual details to improve the overall line of thought, which is the [right scale for skills](https://www.lesswrong.com/posts/JcpzFpPBSmzuksmWM/the-5-second-level), similarly to [local validity](https://www.lesswrong.com/posts/WQFioaudEH8R7fyhm/local-validity-as-a-key-to-sanity-and-civilization). It's a local constraint on form that's not about validity or truth – one I'm struggling to express with minimal nuance.
# LessWrong merch?

Does the LessWrong team sell any merch, or have you considered doing it? Things like shirts, stickers, cute shoggoth plushies?

The reason I'm asking is that branded merch would be an easy way to let people find each other for rationalist meetups. If you see someone wearing a LessWrong shirt, you're probably in the right place[^eaz15k1gzb8].

I'm thinking of just bringing [one of the books](https://www.amazon.com/stores/author/B08NWK2Z92) and leaving it on the table, but it seemed worth asking if there's anything else, especially since [Lightcone seems to need money](https://www.lesswrong.com/posts/5n2ZQcbc7r4R8mvqc/the-lightcone-is-nothing-without-its-people) (who knows, maybe you'll sell 10,000 plushies and solve all of your funding problems!).

[^eaz15k1gzb8]: This is inspired by a local meetup, where the organizer always has a giant stuffed Pikachu with her.
# "Long" timelines to advanced AI have gotten crazy short

First post of [@Helen](https://forum.effectivealtruism.org/users/Helen?mention=user) Toner (of OpenAI board crisis fame)'s new Substack

> It used to be a bold claim, requiring strong evidence, to argue that we might see anything like human-level AI any time in the first half of the 21st century. [This 2016 post](https://www.openphilanthropy.org/research/some-background-on-our-views-regarding-advanced-artificial-intelligence/), for instance, spends 8,500 words justifying the claim that there is a greater than 10% chance of advanced AI being developed by 2036.
> 
> *(Arguments about timelines typically refer to “timelines to AGI,” but throughout this post I’ll mostly refer to “advanced AI” or “human-level AI” rather than “AGI.” In my view, “AGI” as a term of art tends to confuse more than it clarifies, since different experts use it in such different ways.*[*1*](https://helentoner.substack.com/p/long-timelines-to-advanced-ai-have#footnote-1-156254921) *So the fact that “human-level AI” sounds vaguer than “AGI” is a feature, not a bug—it naturally invites reactions of “human-level at what?” and “how are we measuring that?” and “is this even a meaningful bar?” and so on, which I think are totally appropriate questions as long as they’re not used to deny the overall trend towards smarter and more capable systems.)*
> 
> Back in the dark days before ChatGPT, proponents of “short timelines” argued there was a real chance that extremely advanced AI systems would be developed within our lifetimes—perhaps as soon as within 10 or 20 years. If so, the argument continued, then we should obviously start preparing—investing in AI safety research, building international consensus around what kinds of AI systems are too dangerous to build or deploy, beefing up the security of companies developing the most advanced systems so adversaries couldn’t steal them, and so on. These preparations could take years or decades, the argument went, so we should get to work right away.
> 
> Opponents with “long timelines” would counter that, in fact, there was no evidence that AI was going to get very advanced any time soon (say, any time in the next 30 years).[2](https://helentoner.substack.com/p/long-timelines-to-advanced-ai-have#footnote-2-156254921) We should thus ignore any concerns associated with advanced AI and focus instead on the here-and-now problems associated with much less sophisticated systems, such as bias, surveillance, and poor labor conditions. Depending on the disposition of the speaker, problems from AGI might be banished forever as “science fiction” or simply relegated to the later bucket.
> 
> Whoever you think was right, for the purposes of this post I want to point out that this debate *made sense*. “This enormously consequential technology might be built within a couple of decades, we’d better prepare,” vs. “No it won’t, so that would be a waste of time” is a perfectly sensible set of opposing positions.
# POTUS Predictions Tournament

President Trump's second term has brought sweeping policy overhauls in international aid, trade agreements, and immigration enforcement, alongside growing tensions with the judiciary. These rapid changes have increased uncertainty about the US's future and its role on the world stage. Forecasting can help ground our thinking about the likely impacts of the Trump administration, helping to deliver greater clarity to the public on key issues by transforming competing narratives into quantifiable, testable predictions. 

**Make your predictions in Metaculus's** [**POTUS Predictions Tournament**](https://www.metaculus.com/tournament/POTUS-predictions/) **and compete for $15,000 on questions like:**

*   [Will a federal judge be impeached by Republican Representatives in a partisan vote before January 3, 2027?](https://www.metaculus.com/questions/36348/federal-judge-impeached-by-republicans-before-jan-3-2027/)
*   [What will the effective US tariff rate on goods being imported into the United States be at the end of 2026?](https://www.metaculus.com/questions/36295/us-tariff-rate-on-goods-imported-into-us-at-yearend-2026/)
*   [How many journalists in the US will face criminal charges related to their work before 2027?](https://www.metaculus.com/questions/36333/how-many-us-journalists-charged-with-crimes-before-2027/)
*   [Will the United States recognize Taiwan before January 20, 2029?](https://www.metaculus.com/questions/36294/will-the-us-recognize-taiwan-before-january-20-2029/)
*   [What will net outlays by the US government be in FY 2028?](https://www.metaculus.com/questions/36355/what-will-net-outlays-by-the-us-government-be-in-2028/)
*   [Which party will hold a plurality in the US House of Representatives after the 2026 midterm elections?](https://www.metaculus.com/questions/36369/us-house-plurality-after-2026-midterms/)
*   [Will the US Senate weaken or eliminate the filibuster before January 3, 2029?](https://www.metaculus.com/questions/36257/senate-end-or-weaken-filibuster-before-2029/)

We’ll be adding more questions in waves launching on May 1 and June 1, as well as topical questions as they occur over the course of the tournament. 

*For questions focused on international aid, see the* [*USAID Outlook series*](https://www.metaculus.com/tournament/usaid/)*.*
# Cheesecake Frosting

To most Americans, "cream cheese" is savory. You put it on bagels, perhaps with egg, capers, or cured fish. You don't put it on dessert, right?

Except "cream cheese frosting" is a (delicious!) thing, most traditionally for carrot and red velvet cake. I think this incongruity is holding cream cheese frosting back, and it needs better branding. Specifically, I think we should call it "cheesecake frosting". It's essentially no-bake cheesecake already, and it's reasonably close in flavor and texture since they're both mostly cream cheese with sugar and fat.

[![](https://www.jefftk.com/chocolate-cake-with-cheesecake-frosting.jpg)](https://www.jefftk.com/chocolate-cake-with-cheesecake-frosting-big.jpg)

Looking online I do see a [few people](https://cupcakeflip.com/simple-cheesecake-frosting/) [talking](https://www.facebook.com/reel/1277445723548008) [about](https://www.theslowroasteditalian.com/the-best-ever-simple-carrot-cake-with-cheesecake-frosting/) cheesecake frosting, and they're all using it just to mean cream cheese frosting.

On the other hand, I think whipped cream cheese on an Oreo is decent imitation of cheesecake with an Oreo crust, so I'm not sure I'm the best person to listen to here.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0MfuGB9QqM5SXqsXpzndK6nsaPef3ALuEwCkwPeJQ1i2gaLh3xJ7LAxhPeBrg43Tsl), [mastodon](https://mastodon.mit.edu/@jefftk/114277236550653867), [bluesky](https://bsky.app/profile/jefftk.com/post/3llxdai5pjc2k)*
# AI companies’ unmonitored internal AI use poses serious risks

**AI companies’ unmonitored internal AI use poses serious risks**  
*AI companies use their own models to secure the very systems the AI runs on. They should be monitoring those interactions for signs of deception and misbehavior.*

AI companies use powerful unreleased models for important work, like writing security-critical code and analyzing results of safety tests. From my review of public materials, I worry that **AI is largely unmonitored when AI companies use it internally today**, despite its role in important work.

Right now, unless an AI company specifically says otherwise, you should assume they don’t have visibility into all the ways their models are being used internally and the risks these could pose.

In many industries, using an unreleased product internally can’t harm third-party outsiders. A car manufacturer might test-drive its new prototype around the company campus, but civilian pedestrians can’t be hurt—no matter how many manufacturing defects—because the car is contained to the company’s facilities.

The AI industry is different—more like biology labs testing new pathogens: The biolab must carefully monitor and control conditions like air pressure, or the pathogen might leak into the external world. Same goes for AI: If we don’t keep a watchful eye, the AI might be able to escape—for instance, by exploiting its role in shaping the security code meant to keep the AI locked inside the company’s computers.

The minimum improvement we should ask for is **retroactive** **internal monitoring**: AI companies logging all internal use of frontier models by their company, analyzing the logs (e.g., for signs of the AI engaging in scheming), and following up on concerning patterns.

An even stronger vision—likely necessary in the near future, as AI systems become more capable—is to wrap all internal uses of frontier AI within *real-time* control systems that can intervene on the most urgent interactions.

In this post, I’ll cover:

*   Why AI companies should be monitoring internal use of their systems—even if they fully trust their employees
*   What leading companies like Anthropic, OpenAI (where I previously worked[^1^](https://stevenadler.substack.com/p/ai-companies-unmonitored-internal#footnote-1-160293359)), and Google DeepMind have said about this
*   Why some companies might be reluctant to implement full internal monitoring—and how they could still take useful steps in the meantime

*(If you aren’t sure what I mean by “monitoring”—logging, analysis, and enforcement—you might want to check out my* [*explainer*](https://stevenadler.substack.com/p/explainer-the-basics-of-ai-monitoring) *first.)*

\[ [continues on Substack](https://stevenadler.substack.com/p/ai-companies-unmonitored-internal) \]
# How much progress actually happens in theoretical physics?

I frequently hear people make the claim that progress in theoretically physics is stalled, partly because all the focus is on String theory and String theory doesn't seem to pan out into real advances. 

Believing it fits my existing biases, but I notice that I lack the physics understanding to really know whether or not there's progress. What do you think?
# Quarter Inch Cables are Devious

The quarter inch jack (" [phone connector](https://en.wikipedia.org/wiki/Phone_connector_(audio))") is probably the oldest connector still in use today, and it's picked up a very wide range of applications. Which also means it's a huge mess in a live sound context, where a 1/4" jack could be any of:

*   Unbalanced or balanced line level (~1V). Ex: a mixer to a powered speaker.
    
*   Unbalanced instrument level (~200mV), high impedance. Ex: electric guitar.
    
*   Unbalanced piezo level (~50mV), high impedance. Ex: contact pickup on a fiddle.
    
*   Unbalanced speaker level (~30V). Ex: powered amplifier to passive speaker.
    
*   Stereo line level (2x ~1V). Ex: output of keyboard.
    
*   Stereo headphone level (2x ~3v). Ex: headphone jack.
    
*   Send and return line level (~2x 1V). Ex: input to and output from an external compressor.
    
*   Switch (non-audio). Ex: damper pedal on a keyboard, which would be normally open or normally closed.
    
*   1V per octave (~5V). Ex: older modular synth.
    

Here's a compatibility matrix with a bunch of these:

|     |   | If you plug _____ |   |   |   |   |
| --- | --- | --- | --- | --- | --- | --- |
| balanced line level output | electric guitar output | powered amplifier output | stereo line level output | effects loop (mixer side) |
| Into ____ | balanced line level input | ✓   | too low, bad timbre | boom! | left - right | no audio |
| DI input | distortion w/o pad | ✓   | boom! | distortion w/o pad, right channel dropped | no audio |
| speaker input | nearly inaudible | no audio | ✓   | nearly inaudible, right channel dropped | no audio |
| stereo line level input | ✓   | too low, bad timbre, right channel no audio | boom! | ✓   | no audio |
| effects loop (mixer side) | no audio, possible damage | no audio, possible damage | boom! | no audio, possible damage | no audio, possible damage |

Note how few "✓" cells there are.

As much as possible, set things up to avoid 1/4" connectors. If you use [Speakon](https://en.wikipedia.org/wiki/Speakon_connector) to connect amplifier outputs to passive speaker inputs and [XLR](https://en.wikipedia.org/wiki/XLR_connector) for balanced line level signals you'll avoid all the "boom!" outcomes, along with most of the ways of connecting an output to an output. You can't get away from them entirely, but ideally the only 1/4" cables on stage are short runs to DIs.
# Meta releases Llama-4 herd of models

Meta released 3 models from the Llama 4 ~~family~~ herd today. 

*   The 109B (small) and 400B (mid) models are MoE models with 17B active param each. The large model is 2T MoE model with 288B active param instead.
*   The small model is called Scout. The middle model is called Maverick. The big one is called Behemoth.
*   Natively multimodal and much longer context length. 10M (!!) for the small one and 1M for the mid sized one.
*   Maverick is currently number 2 on LLM arena. (Number 5 if style controlled)
*   We don't have the 2T big model available yet. It is used as a distillation model for Maverick and Scout.
*   Behemoth beats Claude 3.7 (not thinking), GPT 4,5 and Gemini 2.0 Pro in various benchmarks (LiveCodeBench, Math-500, GPQA Diamond, etc)
*   Scout is meant to work on a single H100.

[https://ai.meta.com/blog/llama-4-multimodal-intelligence/](https://ai.meta.com/blog/llama-4-multimodal-intelligence/)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3297879b756904b35474f78f1101de17a9c5c8c2ca0dad7f.png)