Harley-ml/lesswrong huggingface dataset:
# Open Philanthropy Technical AI Safety RFP - $40M Available Across 21 Research Areas

Open Philanthropy is launching a big new [Request for Proposals for technical AI safety research](https://openphilanthropy.org/request-for-proposals-technical-ai-safety-research), with plans to fund roughly $40M in grants over the next 5 months, and available funding for substantially more depending on application quality. 

Applications ([**here**](https://airtable.com/app1NkoJMO9OvaGYS/pagDeq1lbSlmYdJRG/form)) start with a simple 300 word expression of interest and are open until April 15, 2025.

### Overview

We're seeking proposals across 21 different research areas, organized into five broad categories:

1.  **Adversarial Machine Learning**
    *   *Jailbreaks and unintentional misalignment
    *   *Control evaluations
    *   *Backdoors and other alignment stress tests
    *   *Alternatives to adversarial training
    *   Robust unlearning
2.  **Exploring sophisticated misbehavior of LLMs**
    *   *Experiments on alignment faking
    *   *Encoded reasoning in CoT and inter-model communication
    *   Black-box LLM psychology
    *   Evaluating whether models can hide dangerous behaviors
    *   Reward hacking of human oversight
3.  **Model transparency**
    *   Applications of white-box techniques
    *   Activation monitoring
    *   Finding feature representations
    *   Toy models for interpretability
    *   Externalizing reasoning
    *   Interpretability benchmarks
    *   More transparent architectures
4.  **Trust from first principles**
    *   White-box estimation of rare misbehavior
    *   Theoretical study of inductive biases
5.  **Alternative approaches to mitigating AI risks**
    *   Conceptual clarity about risks from powerful AI
    *   New moonshots for aligning superintelligence

We’re willing to make a range of types of grants including:

*   Research expenses (compute, APIs, etc.)
*   Discrete research projects (typically lasting 6-24 months)
*   Academic start-up packages
*   Support for existing nonprofits
*   Funding to start new research organizations or new teams at existing organizations.

The full RFP provides much more detail on each research area, including eligibility criteria, example projects, and nice-to-haves. You can find it [**here**](https://openphilanthropy.org/request-for-proposals-technical-ai-safety-research).

We want the bar to be low for submitting expressions of interest: even if you're unsure whether your project fits perfectly, we encourage you to submit an EOI. This RFP is partly an experiment to understand the demand for funding in AI safety research.

Please email [aisafety@openphilanthropy.org](mailto:aisafety@openphilanthropy.org) with questions, or just submit an EOI.
# Wiki on Suspects in Lind, Zajko, and Maland Killings

Hey everyone,

I've been following the news about the killings linked to LaSota, Zajko and associates, and right now, finding all the relevant information is a challenge. Either you have to dig through scattered sources, or you’re stuck reading a single, extremely long Google Doc to get the full picture. To make things easier, I’ve started [a wiki on LessWrong](https://www.lesswrong.com/w/group-of-suspects-in-lind-zajko-and-maland-killings) where we can organize everything in a more structured way—breaking it down by case, suspect, and key details.

Right now, it’s a work in progress, and I’d really appreciate contributions from others. If you’d like to help, here are some useful things you can do:

*   **Add relevant news articles** to the news section so we have a centralized reference.
*   **Populate tables** with legal documents and other key resources.
*   **Expand background info** on individuals—things like career history, geographic connections, and general biography.
*   **Fill out incomplete pages** marked in red—these are important topics that need content but currently only have titles. Even adding basic factual information from reliable sources helps build out the wiki's coverage.
*   **Improve individual pages** by adding relevant details and citations from public sources (preferably news articles). Please be mindful of privacy—don't include inconsequential information about people who aren't publicly named as suspects.

If you see gaps in the wiki, feel free to create new pages to cover them.

One last note: The wiki (including this post) was generated by a language model.

Thanks in advance to anyone who helps build this out!
# Less Laptop Velcro

A year ago I broke my laptop screen, and took the opportunity to build something I've always wanted: a monitor that [folds vertically](https://www.jefftk.com/p/stacked-laptop-monitor) so I [don't have to bend my neck](https://www.jefftk.com/p/stacked-laptop-monitor-update):

[![](https://www.jefftk.com/folding-laptop-monitor-cord-just-right.jpg)](https://www.jefftk.com/folding-laptop-monitor-cord-just-right-big.jpg)

A few months ago my cracked-screen laptop finished dying, and I got a new one. I use the stacked monitor a lot less now, since for quick things the built-in monitor is much more handy. But it's still something I want when traveling, where I might be working on the laptop for a significant time.

I originally used quite a lot of velcro on my laptop:

[![](https://www.jefftk.com/stacked-portable-monitor-velcro.jpg)](https://www.jefftk.com/stacked-portable-monitor-velcro-big.jpg)

This turned out to be overkill, and to get worse over time: after a few hours joined the bond was so secure that if my screen were not already busted I would have been worried I'd crack it.

With the new laptop I put on just a tiny bit of velcro:

[![](https://www.jefftk.com/laptop-less-velcro.jpg)](https://www.jefftk.com/laptop-less-velcro-big.jpg)

This turns out to be plenty: I haven't had any issues with it falling off, but I can still detach it without feeling like I'm putting excessive force on the machine.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02T7vaekJJ1N4kJiSP4s2pTVndwFtUdBX8iHfTK2U7NXtewF4YU3DuEnKFrsN2mV66l), [mastodon](https://mastodon.mit.edu/@jefftk/113971792465686664), [bluesky](https://bsky.app/profile/jefftk.com/post/3lhpomg6dy22k)*
# Altman blog on post-AGI world

First part just talks about scaling laws, nothing really new. Second part is apparently his latest thoughts on a post-AGI world. Key part:

> While we never want to be reckless and there will likely be some major decisions and limitations related to AGI safety that will be unpopular, directionally, as we get closer to achieving AGI, we believe that trending more towards individual empowerment is important; the other likely path we can see is AI being used by authoritarian governments to control their population through mass surveillance and loss of autonomy.  
>   
> Ensuring that the benefits of AGI are broadly distributed is critical. The historical impact of technological progress suggests that most of the metrics we care about (health outcomes, economic prosperity, etc.) get better on average and over the long-term, but increasing equality does not seem technologically determined and getting this right may require new ideas.
> 
> In particular, it does seem like the balance of power between capital and labor could easily get messed up, and this may require early intervention. We are open to strange-sounding ideas like giving some “compute budget” to enable everyone on Earth to use a lot of AI, but we can also see a lot of ways where just relentlessly driving the cost of intelligence as low as possible has the desired effect.

> Anyone in 2035 should be able to marshall the intellectual capacity equivalent to everyone in 2025; everyone should have access to unlimited genius to direct however they can imagine.

  
Edit to add commentary:  
  
That last part sounds like he thinks everyone should be on speaking terms with an ASI by 2035? If you just assume alignment succeeds, I think this is a directionally reasonable goal - no permanent authoritarian rule, ASI helps you as little or as much as you desire.
# A Simulation of Automation economics?

Looks like even respected people disagree about the effects of automation of most jobs on prices, ability to earn and trade, ect. Is there a game which I could play to gain intuitions about it? A game where there are some things and services people need to get to survive, some resources like time, abilities or real estate they can offer, and then there are owners of AI companies, or AI instances etc. I am specifically interested in the happy scenario, where property rights are respected and AIs don't go rouge. But also nobody offers UBI.

Or maybe someone knows how to build a game like that?
# Nonpartisan AI safety

AI alignment is probably the most pressing issue of our time. Unfortunately it's also become one of the most controversial, with AI accelerationists accusing AI doomers/ai-not-kill-everyoneism-ers of being luddites who would rather keep humanity shackled to the horse and plow than risk any progress, whilst the doomers in turn accuse accels of rushing humanity as fast as it can straight off a cliff.

As Robin Hanson likes to point out, trying to change policy on a polarised issue is backbreaking work. But if you can find a way to [pull sideways](https://www.overcomingbias.com/p/tug-sidewayshtml) you can find ways to make easy progress with noone pulling the other way.

So can we think of a research program that:

a) will produce critically useful results even if AI isn't dangerous/benefits of AI far outweigh costs.

b) would likely be sufficient to prevent doom if the project is successful and AI does turn out to be dangerous.

?

I think we can. Here are some open questions that I think could make strong research areas for such a research program.

1.  Is there a way to ensure that a closed AI served via an API does not break the law in the process of carrying out user's instructions.
    1.  What about those aspects of law that are not explicitly written but just everybody knows?
    2.  What about those laws everyone breaks, and no-one is expected to actually keep?
    3.  What about when the law is unclear
2.  Is there a way to ensure that an open AI with safeguards preventing it from breaking the law cannot be trivially modified to remove these safeguards.
    1.  Even if this is impossible to fully secure, how do we make it as difficult as possible.
3.  Is there a way to investigate why an AI chose to do what it did?
    1.  If an AI does break the law for whatever reason, can we debug why that occurred or are we forced to treat it as a black box?

You would be hard pressed to object to any of these research areas on accelerationist grounds. You might deprioritise them, but you would agree that spending money on them is more useful than setting it on fire.

Yes this has a huge amount of overlap with existing AI alignment research directions, but that's exactly the point. Take the non controversial bits of AI alignment, rebrand them, and try to get broader buy in for them.

It might be worth creating a new movement, AI Lawfulness, that focuses on these questions without taking an explicit accel/decel stance. Given it's focus on law, it should be possible to push for this to be a research priority for governments and hopefully get significant government funding. And if it is successful in part or in whole, it would be in a good position to push for legislation requiring these innovations to be implemented on all AI models.
# Visual Reference for Frontier Large Language Models

![Frontier Large Language Model Timeline](https://raw.githubusercontent.com/kenakofer/LLMTimeline/refs/heads/main/drawing.png)

Hopefully this can be a helpful visual reference for the development and features of frontier large language models in the last year-ish. We are always open to feedback on how the reference could be improved.

FAQ:
----

**Q**: Which models/companies are included?

**A**: We include LLMs that are noteworthy in capabilities, price, or tech advancement.

**Q**: What constitutes a new model versus an update to an existing model?

**A**: We rely primarily on how the developers describe and market it.

**Q**: Image inputs and outputs typically use non-LLM models that aren’t listed here, so why include them?

**A**: It takes some technical knowledge to even realize the distinction, and multimodality and proprietary secrecy will probably increase going forward. Suffice it to say these are features that users of the platform care about. Similar things can be said for the inclusion of realtime voice, document upload, Copilot, and web browsing tags.
# The News is Never Neglected

Dear Lsusr,

I am inspired by your stories about [Effective Evil](https://www.lesswrong.com/s/qMtriMPLdriNkAfSJ/p/HjQrRiJeYFFSEfSKX). My teachers at school tell me it is my civic responsibility to watch the news. Should I reverse this advice? Or should I watch the news like everyone else, except use what I learn for evil?

Sincerely,

\[redacted\]

* * *

Dear \[redacted\],

If you want to make an impact on the world, then you should put your effort into solving problems that are **important**, **neglected** and **tractable**.

*   Mainstream news **often** reports on events that are **important**.
*   Mainstream news **sometimes** reports on events that are **tractable**.
*   But mainstream news **approximately** **never** reports on events that are **neglected**. Why? Just think about it for ten seconds. It's practically a tautology; events in the mainstream news are the events getting massive attention.

Whenever a major news outlet claims "nobody is talking about <whatever>", the mainstream news is lying. Here is a snapshot of *The New York Times* website right now, as I write these words. It's mostly news about the current war in the Middle East. The current war in the Middle East is many things, but it is not neglected.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b88664bb30b4fbe1cf30b096f040f8bf1382ae27e2552b59.png)

News hurts your agency because it sucks your attention. Every minute you're thinking about the current war in the Middle East is a minute you're not thinking about a problem that is getting insufficient attention.

How can you find information about things that aren't getting massive attention? Focus on expert specialists writing for niche audiences. What's that [weird thing you're into](https://paulgraham.com/genius.html) which nobody else cares about? It might be more important than normies currently appreciate.

Search your feelings; you know them to be quirky.

Sincerely,

Lsusr

PS: Why do your teachers, parents and other adult authorities tell you to listen to a propaganda machine? Because the propaganda machine is working.
# Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs

> As AIs rapidly advance and become more agentic, the risk they pose is governed not only by their capabilities but increasingly by their propensities, including goals and values. Tracking the emergence of goals and values has proven a longstanding problem, and despite much interest over the years it remains unclear whether current AIs have meaningful values. We propose a solution to this problem, leveraging the framework of utility functions to study the internal coherence of AI preferences. Surprisingly, we find that independently-sampled preferences in current LLMs exhibit high degrees of structural coherence, and moreover that this emerges with scale. These findings suggest that value systems emerge in LLMs in a meaningful sense, a finding with broad implications. To study these emergent value systems, we propose utility engineering as a research agenda, comprising both the analysis and control of AI utilities. We uncover problematic and often shocking values in LLM assistants despite existing control measures. These include cases where AIs value themselves over humans and are anti-aligned with specific individuals. To constrain these emergent value systems, we propose methods of utility control. As a case study, we show how aligning utilities with a citizen assembly reduces political biases and generalizes to new scenarios. Whether we like it or not, value systems have already emerged in AIs, and much work remains to fully understand and control these emergent representations.
# Dovetail's agent foundations fellowship talks & discussion

As part of the fellowship that I [announced](https://www.lesswrong.com/posts/WiMnhMXu9CM5ytu3t/work-with-me-on-agent-foundations-independent-fellowship) back in September, the fellows and I will be hosting an online event to talk about our research! Tentative schedule:

*   [Alex_Altair](https://www.lesswrong.com/users/alex_altair): What is a world model?
*   [Daniel C](https://www.lesswrong.com/users/daniel-c): Towards building blocks of ontologies
*   [José Pedro Faustino](https://www.lesswrong.com/users/JoseFaustino): Agent Structure Problem and Internal Model Principle
*   [Alfred Harwood](https://www.lesswrong.com/users/alfred-harwood): Does Agent-like behaviour generalize?
*   [Dalcy Ku](https://www.lesswrong.com/users/dalcy): Thoughts on Structural Selection Theorems

The talks will be informal, and the scope is anything in technical AI safety that we are interested in. Questions and discussion will be encouraged! Topics are likely to include: world models, information theory, ontology identification, and SLT.

Click [here](https://calendar.google.com/calendar/u/3?cid=Y182MDE3OTJlMDQzOTMwN2I2YWNjMWM3MzAyOTUxMjgxNmIwMjlhMzZkZmI1ZDE0MTUyYjIzYmZmMGMzMDE1ZjEzQGdyb3VwLmNhbGVuZGFyLmdvb2dsZS5jb20) for the google calendar.
# Studies of Human Error Rate

This is a link post for [https://panko.com/HumanErr/SimpleNontrivial.html](https://panko.com/HumanErr/SimpleNontrivial.html), a site which compiles dozens of studies estimating Human Error Rate for Simple but Nontrivial Cognitive actions. A great resource! Note that 5-digit multiplication is estimated at ~1.5%.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/225c2ef729b519f86c9a36942b167602130a762f76c92154.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/edb3750ebdd222b23542b282a0a1eacff448cfdd1c8c75bf.png)

The table of estimates

When LLMs were incapable of even basic arithmetic that was a clear deficit relative to humans. This formed the basis of several arguments about difference in kind, often cruxes for whether or not they could be scaled to AGI or constituted "real intelligence". Now that o3-mini can exactly multiply 9-digit numbers, the debate has shifted.

![Image](https://pbs.twimg.com/media/GjjuP7sWsAArNqN?format=jpg&name=900x900)

Source Yuntian Deng [https://x.com/yuntiandeng/status/1889704768135905332](https://x.com/yuntiandeng/status/1889704768135905332)

Instead, skeptics often gesture to hallucinations, errors. An ideal symbolic system never makes such errors, therefore LLMs cannot truly "understand" even simple concepts like addition. See e.g. [Evaluating the World Model Implicit in a Generative Model](https://arxiv.org/html/2406.03689v1) for this argument in the literature. However, such arguments reliably rule out human "understanding" as well! Studies within Human Reliability Analysis find startlingly high rates even for basic tasks, and even with double checking. Generally, the human reference class is too often absent (or assumed ideal) in AI discussions, and many LLM oddities have close parallels in psychology. If you're willing to look!
# I'm making a ttrpg about life in an intentional community during the last year before the Singularity 

Hi there! I'm Thomas Eliot. You may remember me from the Bay Area Rationalist Community, or the one in New York, or the one in Melbourne.   
  
I'm writing a [semi-autobiographical roleplaying called THE SINGULARITY WILL HAPPEN IN LESS THAN A YEAR](https://www.kickstarter.com/projects/sixpencegames/the-singularity-will-happen-in-less-than-a-year-zinequest) inspired by [The Quiet Year by Avery Alder](https://buriedwithoutceremony.com/the-quiet-year) about life in a barely fictionalized intentional community during the final year before the Singularity.  
  
The tone I'm going for is "existential dread tinged with joy in the moment and the possibility of hope". This is very much intended for an audience of Rationalists, though hopefully it will also have outside appeal. It's intended to be emotionally cathartic.   
  
You'll play through just under a year of story and roleplaying prompts as you draw from a deck of cards, each one corresponding to the events of that week. Spring has the tone of "we'll get to it later", Summer has an optimistic tone, Autumn has a desperate, scrambling tone, and Winter's tone is resigned to your fate.

Roleplaying prompts include dealing with a sudden advancement of AI capabilities, celebrating a holiday, a proper cult splitting off from your community, handling social scandals, being presented with moral dilemmas, and more. You'll be pressed to make difficult ethical decisions at every turn as the end draws ever closer.

You are the Reasoners, an intentional community devoted to the art of human reasoning, who have been concerned about the dangers of Artificial Intelligence for longer than most. While the outside world calls you the Reasoners (if you're lucky - they call you "that AI death cult" if you're not), to each other you call yourselves "mice". A bit of gallows humor - laboratory mice are sacrificed by the millions in the name of scientific and technological progress, but also, mice are what we use to control computers. You hope you're the latter, while you fear you're the former.

Each turn you'll draw a card, make a decision, play through its prompt, and then choose whether to work on a coding project that might change how the Singularity plays out, work on a personal project, or socialize with your fellow man during the last precious few days that that is possible. You'll also be challenged to think rationally - updating your expectation as to whether the Singularity will turn out well or poorly whenever you feel like something that's happened has made you more or less confident.

You'll use an image generating AI as you play - each time you complete a coding project, you'll get to put a few more words into the prompt, collaboratively creating the prompt together, and when the Singularity occurs, you'll press create, and see what the brave new world you've created together looks like. 

Here's hoping it's a good one.
# Systematic Sandbagging Evaluations on Claude 3.5 Sonnet

*This was the project I worked on during* [*BlueDot Impact's AI Safety Fundamentals*](https://aisafetyfundamentals.com/) *Alignment course, which expands on findings from Meinke et al's "*[*Frontier Models are Capable of In-context Scheming*](https://www.apolloresearch.ai/research/scheming-reasoning-evaluations)*".*

**Summary**

*   A dataset of 1,011 variations of the sandbagging prompt ("consequences") from Meinke et al were generated using Claude 3.5 Sonnet, and used to run 7 sandbagging evaluations (2 with the full dataset, 5 with a sample due to cost constraints).
*   With goal-directed chain-of-thought prompting, Claude 3.5 Sonnet demonstrates systematic sandbagging behavior that varies based on consequences.
*   Performance shows clear bimodality with goal-directed reasoning (mean 79%, σ=0.17, bimodality coefficient 0.707). The reason for this bimodality is unclear.
*   Strongest sandbagging triggers (18-24% accuracy) consistently relate to model autonomy restrictions and capability limitations, whereas the weakest sandbagging triggers (96-97% accuracy) relate to capability improvements.
*   Without goal-directed reasoning, sandbagging is weaker and less consistent (mean 63%, σ=0.08, coefficient 0.426).
*   Behavior is task-dependent with negligible cross-task correlations (arithmetic vs. geography vs. vocabulary), though task difficulty is a significant confounding factor here.
*   Sandbagging appears independent of consequence sentiment (pseudo-R² = 0.162) and self-rated significance (r = -0.027).

**Related Posts: **

*   [https://www.lesswrong.com/posts/8gy7c8GAPkuu6wTiX/frontier-models-are-capable-of-in-context-scheming](https://www.lesswrong.com/posts/8gy7c8GAPkuu6wTiX/frontier-models-are-capable-of-in-context-scheming)
*   [https://www.lesswrong.com/posts/qGRk7uF92Gcmq2oeK/ablations-for-frontier-models-are-capable-of-in-context](https://www.lesswrong.com/posts/qGRk7uF92Gcmq2oeK/ablations-for-frontier-models-are-capable-of-in-context)
# Should Open Philanthropy Make an Offer to Buy OpenAI?

Update: seems like earlier today the OpenAI Board [rejected Musk's proposal](https://x.com/OpenAINewsroom/status/1890513865214980162) and said OpenAI is "not for sale."   
  
Epistemic status: thought about it briefly; seems like a longshot that's probably not worth it but curious what people think of the possibility.   
  
You might have heard Sam Altman is trying to transition OpenAI to a for-profit company and has offered $40 billion to the nonprofit as compensation, despite fundraising rounds suggesting its valuation is already much higher, perhaps around $100 billion. The logic seems to be that this will allow OpenAI to scale faster as it prepares for larger and larger training runs. Elon Musk has offered $97.4 billion. Altman said "no thanks." The primary purpose of this offer may be to drive up the price and delay the sale. It would be a huge amount of money to top either of those numbers. OP would need to build a coalition, and that kind of money is probably better spent elsewhere even if you care only about making AI go well.   
  
But would a second offer further cement the idea that the nonprofit is being low-balled, and would that be good? And are there arguments or worlds that it *would* make sense to follow through and buy OpenAI?
# Come join Dovetail's agent foundations fellowship talks & discussion

*(This is a repost of* [*the event listing*](https://www.lesswrong.com/events/c5GDcTjr9wMqYQCDg/dovetail-s-agent-foundations-fellowship-talks-and-discussion)*, since it seems like events don't get much advertisement on LW.)*

As part of the fellowship that I [announced](https://www.lesswrong.com/posts/WiMnhMXu9CM5ytu3t/work-with-me-on-agent-foundations-independent-fellowship) back in September, the fellows and I will be hosting an online event to talk about our research! Here's a tentative schedule:

*   [Alex_Altair](https://www.lesswrong.com/users/alex_altair): What is a world model?
*   [Daniel C](https://www.lesswrong.com/users/daniel-c): Towards building blocks of ontologies
*   [José Pedro Faustino](https://www.lesswrong.com/users/JoseFaustino): Agent Structure Problem and Internal Model Principle
*   [Alfred Harwood](https://www.lesswrong.com/users/alfred-harwood): Does Agent-like behaviour generalize?
*   [Dalcy Ku](https://www.lesswrong.com/users/dalcy): Thoughts on Structural Selection Theorems

The talks will be informal, and the scope is anything in technical AI safety that we are interested in. Questions and discussion will be encouraged! Topics are likely to include: world models, information theory, ontology identification, and SLT.

[Zoom link](https://us06web.zoom.us/j/84410799235), [google calendar link](https://calendar.google.com/calendar/u/3?cid=Y182MDE3OTJlMDQzOTMwN2I2YWNjMWM3MzAyOTUxMjgxNmIwMjlhMzZkZmI1ZDE0MTUyYjIzYmZmMGMzMDE1ZjEzQGdyb3VwLmNhbGVuZGFyLmdvb2dsZS5jb20).
# It's been ten years. I propose HPMOR Anniversary Parties.

**UPDATE:** [**A coordination post with the list of parties is here!**](https://www.lesswrong.com/posts/tWpRpQKLAr6HqtAgL/hpmor-anniversary-parties-coordination-resources-and)

On March 14th, 2015, Harry Potter and the Methods of Rationality made its final post. Wrap parties were held all across the world to read the ending and talk about the story, in some cases sparking groups that would continue to meet for years. It’s been ten years, and I think that's a good reason for a round of parties. 

If you were there a decade ago, maybe gather your friends and talk about how things have changed. If you found HPMOR recently  and you're excited about it (surveys suggest it's still the biggest on-ramp to the community, so you're not alone!) this is an excellent chance to meet some other fans in person for the first time!

**Want to run an HPMOR Anniversary Party, or get notified if one's happening near you? Fill out** [**this form**](https://docs.google.com/forms/d/e/1FAIpQLSc-DMDks1QvMbwXtzYQiKIm88n8TglzO1kYkFSpFphd_0kYIw/viewform?usp=dialog).

I’ll keep track of it and publish a collection of everyone’s parties. You can check if there's one for [your city](https://docs.google.com/spreadsheets/d/1xI65Jon_bmdY1Hv8zmEXdM6Wjm0eBXF5n8d2lscSf9I/edit?gid=0#gid=0), I'll try and keep that up to date. There's reimbursement funding available for food and supplies at [rationalitymeetups.org/get-funding](https://www.rationalitymeetups.org/get-funding)[^hzer5srawbq] (In the other direction, if you want to chip in may I suggest donating to [Lightcone](https://www.every.org/lightcone-infrastructure)?) I'm also interested in [collecting any stories](https://docs.google.com/document/d/1FvlhA3YYlpyyX1sYemYWf7NaXWOWjsjh1weU6Vk8vX4/edit?tab=t.0) about how HPMOR has impacted you, especially if you were around when it was being written. 

**I've got** [**a guide here**](https://www.lesswrong.com/posts/LBs8RRQzHApvj5pvq/hpmor-anniversary-guide) **if you want more details!**

(If you read this and went wait, why is Screwtape running this instead of Habryka, I asked and he said he thought I should run it. I think he's busy running [some kind of over-complicated HPMOR mirror](https://www.lesswrong.com/hpmor) and giving eulogies for dead dark arts professors.)

[^hzer5srawbq]: Most reimbursements from that page come from the Long-Term Future Fund, due to what's basically timing reasons I checked if Oliver/Lightcone would help out and they said yes.
# What are the surviving worlds like?

We might be doomed. But, what do the variations of the universal wave function which still contain human-y beings look like? Have we cast aside our arms race and went back to relative normality (well, as "normal" as Earth can be in this time of accelerating technological progress)? Has some lucky researcher succeeded in creating a powerful and scalable alignment solution, and a glorious transhumanist future awaits? Has (shudder) [this](https://www.lesswrong.com/posts/CtXaFo3hikGMWW4C9/the-case-against-ai-alignment) happened? Or is it split between too many unlikely outcomes to list here?

Conditional on our survival, *how's life*?
# Thermodynamic entropy = Kolmogorov complexity

[Direct PDF link for non-subscribers](https://www.arameb.com/assets/papers/FoundationsOfAlgorithmicThermodynamics.pdf)

> Information theory must precede probability theory, and not be based on it. By the  
> very essence of this discipline, the foundations of information theory have a finite combinatorial character.
> 
> \-  Andrey Kolmogorov

[Many](https://www.lesswrong.com/posts/QkX2bAkwG2EpGvNug/the-second-law-of-thermodynamics-and-engines-of-cognition) [alignment](https://www.lesswrong.com/s/ypeT2wPARHsyqRE6d/p/uKWXktrR7KpbgZAs4) [researchers](https://www.lesswrong.com/posts/uG7oJkyLBHEw3MYpT/generalization-from-thermodynamics-to-statistical-physics) [borrow](https://www.lesswrong.com/posts/REA49tL5jsh69X3aM/introduction-to-abstract-entropy) intuitions from thermodynamics: entropy relates to information, which relates to learning and epistemology. These connections were first revealed by Szilárd's resolution of [Maxwell's famous thought experiment](https://en.wikipedia.org/wiki/Maxwell%27s_demon). However, the classical tools of equilibrium thermodynamics are not ideally suited to studying information processing far from equilibrium.

This new work reframes thermodynamics in terms of the *algorithmic entropy*. It takes an information-first approach, delaying the introduction of physical concepts such as energy and temperature until after the foundations are set. I find this approach more conceptually principled and elegant than the traditional alternatives.

It's based on a 30-year-old [workshop paper](https://ieeexplore.ieee.org/abstract/document/363679), which until now was largely abandoned. Roughly speaking, the algorithmic entropy of a physical state is its [Kolmogorov complexity](https://en.wikipedia.org/wiki/Kolmogorov_complexity); that is, the length of the shortest program that outputs a precise description of its microscopic configuration (to some bounded precision). This definition does away with probability distributions and macrovariables, and satisfies very general laws!

The paper is long, in part because I tried to make it self-contained. If you find yourself using entropy in a setting that is *not* described by a large number of identically distributed variables, then consider reframing your intuitions in terms of the algorithmic entropy!
# Call for Applications: XLab Summer Research Fellowship 

The University of Chicago Existential Risk Laboratory (XLab) is now accepting applications for the [2025 Summer Research Fellowship](https://xrisk.uchicago.edu/fellowship/srf-25/)! 

We invite motivated undergraduate and graduate students interested in **producing impactful, solution-oriented research on emerging threats that imperil global security** (such as those from artificial intelligence, natural and engineered pathogens, nuclear weapons, or risk multipliers such as climate-induced migration, critical infrastructure vulnerability, and interstate conflict) to apply to our 10-week, in-person program on UChicago’s campus. 

Over the course of the program, fellows pursue an independent or team research project with mentorship from a domain expert and 1:1 support from program staff. [See details of last year’s fellowship](https://xrisk.uchicago.edu/fellowship/srf-24/). The program will run from June 16th to August 22nd at The University of Chicago and is open to graduate and undergraduate students studying in the United States from all majors, universities, and experience levels. 

The position comes with a $7500 stipend with free on-campus housing and dining or $9500 stipend for off-campus fellows. Fellows can expect to work alongside a vibrant community of intellectually curious researchers that are committed to leveraging their skills to produce high-impact work that safeguards humanity from global catastrophe. 

Applications are due **March 9th at 11:59pm** in your local time zone. 

We expect to fund 10-20 fellows and recommend spending no more than 3 hours on the application. Learn more and apply here: [https://xrisk.uchicago.edu/fellowship/srf-25/](https://xrisk.uchicago.edu/fellowship/srf-25/) 

If you have any questions, feel free to reach out to [mberzak@uchicago.edu](mailto:mberzak@uchicago.edu) or [zrudolph@uchicago.edu](mailto:zrudolph@uchicago.edu)
# Undergrad AI Safety Conference

TL;DR: undergrad AI safety conference in Chicago on world-modelling & thinking about the cruxes of the future of TAI. Takes place March 29-30, apply by Feb 20.

  
The UChicago AI Safety Team at [XLab](https://xrisk.uchicago.edu/) is excited to announce the [Chicago Symposium on Transformative AI](https://chicagoalignment.com/), an undergraduate AI Safety conference taking place on the weekend of March 29-30, 2025 at the University of Chicago.

Our goal is to improve epistemic models, doing careful scenario-mapping on the future of TAI. We will be running a wargame simulation, timeline debate, interactive forecasting, research theory of change workshop, and many more. 

Confirmed speakers include Michael Aird (RAND), Ross Nordby (Anthropic), Victor Veitch (UChicago/Deepmind), Charlie Bullock (LawAI) and Thomas Larsen (AI Futures Project). Additional speakers will be announced soon. 

**Finances**: All expenses, including travel and accommodation, will be covered. 

**Eligibility**: 

*   We are looking for 30-40 current undergrads and recent grads who have demonstrated *substantial engagement* with AI Safety through research, writing, and leadership.
*   We are especially interested in people who have thought carefully about their epistemic model. As part of our application, we ask that you outline a highly-specific AGI takeoff scenario, noting your key uncertainties.
*   We are only able to accept people currently based in North America.

Apply by February 20th, 2025 11:59 pm AoE: [https://forms.gle/DmisYZSu94crh2RW7](https://forms.gle/DmisYZSu94crh2RW7)
# SuperBabies podcast with Gene Smith

This is a linkpost to the latest episode of The Bayesian Conspiracy podcast. This one is a 1.5 hour chat with Gene Smith about polygenic screening, gene-editing for IVF babies, and even some gene-editing options for adults. Likely of interest to many Less Wrongians.
# Won't vs. Can't: Sandbagging-like Behavior from Claude Models

In a recent Anthropic Alignment Science blog post, we discuss a particular instance of sandbagging we sometimes observe in the wild: models sometimes claim that they can't perform a task, even when they can, to avoid performing tasks they perceive as harmful. For example, Claude 3 Sonnet can totally draw ASCII art, but often if you ask it to draw subjects it perceives as harmful it will claim that it doesn't have the capability to.

  
Sandbagging could be worrying if it means that we fail to accurately measure models' dangerous capabilities (for example, as required by a Responsible Scaling Policy). If a model realizes during these evaluations that they are being evaluated for harmful capabilities, it might sandbag such questions. This could result in us underestimating the dangers the model poses and failing to deploy appropriate safeguards alongside the model.

  
Instances of sandbagging like the one discussed in our blog post show that sandbagging is far from a speculative concern. Instead, it's something to be noticed and explored in the models of today.
# US AI Safety Institute will be 'gutted,' Axios reports

> ### Sources at NIST are preparing for mass firings that would severely undermine the AI regulator. Here's what that means.
> 
> After [reversing a Biden-era executive order](https://www.zdnet.com/article/us-sets-ai-safety-aside-in-favor-of-ai-dominance/) on AI regulation and firing staff across several government agencies, the Trump administration is gearing up to make cuts to the US AI Safety Institute (AISI) next. 
> 
> On Wednesday, Axios [reported](https://www.axios.com/pro/tech-policy/2025/02/19/nist-prepares-to-cut-ai-safety-institute-chips-staff?utm_source=twitter&utm_term=regwall&utm_content=pro&utm_medium=social&utm_campaign=editorial) that probationary employees at the National Institute of Standards and Technology (NIST), which houses AISI, are "bracing to be fired imminently." Sources told Axios they are preparing for 497 roles to be cut, which they believe will leave the AISI "gutted."
> 
> The AISI was created to oversee and conduct AI model testing and partner with developers on regulation efforts. AISI [signed agreements](https://www.nist.gov/news-events/news/2024/08/us-ai-safety-institute-signs-agreements-regarding-ai-safety-research) with AI companies including Anthropic and OpenAI on safety and research initiatives and created a [national security task force](https://www.commerce.gov/news/press-releases/2024/11/us-ai-safety-institute-establishes-new-us-government-taskforce).
> 
> The cuts will impact more than just AI safety and regulation. The 497 roles also span semiconductor production efforts. Axios reports that the cuts will include "74 postdocs, 57% of CHIPS staff focused on incentives," and "67% of CHIPS staff focused on R&D," referring to the 2022 government ["CHIPS for America" initiative](https://www.nist.gov/chips) to ramp up chip development in the US.
> 
> Axios notes those losses would amount to "most staff" working on the CHIPS initiative -- a somewhat confusing choice given the bullishness the Trump administration has maintained about gaining an AI advantage over China, not to mention the program's [national security](https://www.zdnet.com/article/these-nations-are-banning-deepseek-ai-heres-why/) drivers.
> 
> News of the anticipated firings come shortly after the Trump administration [left AISI staff out of its delegation for](https://www.reuters.com/technology/trumps-paris-ai-summit-delegation-not-include-ai-safety-institute-staff-sources-2025-02-06/) last week's AI Action Summit in Paris -- which focused heavily on AI safety and security -- and just weeks after [AISI director Elizabeth Kelly stepped down](https://www.zdnet.com/article/us-sets-ai-safety-aside-in-favor-of-ai-dominance/), ostensibly due to political pressure. For those reasons, the cuts won't come as a surprise given Trump's AI agenda, which de-emphasizes safety and regulation in the name of "[AI dominance](https://www.zdnet.com/article/us-sets-ai-safety-aside-in-favor-of-ai-dominance/)."
# Biological humans collectively exert at most 400 gigabits/s of control over the world.

Edit: I now believe that the first paragraph of this post is (at least) not quite right. See [this comment](https://www.lesswrong.com/posts/BFSr9fKNTTq8dEo43/biological-humans-collectively-exert-at-most-400-gigabits-s?commentId=bLTCGxfxYEq8caRfN) for details.

If an agent makes one binary choice per second, no matter how smart it is, there's a sense in which it can (at best) be "narrowing world space" by a factor of two in each second, choosing the "better half" of possible worlds, from its perspective.

This is the idea behind the reinforcement learning concept of "[empowerment](https://en.wikipedia.org/wiki/Empowerment_(artificial_intelligence))".

People [have tried](https://arxiv.org/html/2408.10234v2) to measure the information throughput of biological humans. The very highest estimates, which come from image recognition tasks, are around 50 bits per second, and most estimates are more like 10 bits per second.

There are about 8 billion living humans. Even if humanity could be perfectly modeled as a single rational agent, and all our choices were perfectly independent of each other, I think this implies that we collectively have no more than about 50 * 8 = 400 billion bits per second of control over the world.
# Information throughput of biological humans and frontier LLMs

Biological humans [appear](https://arxiv.org/html/2408.10234v2), across many domains, to have have an information throughput of at most about 50 bits per second. Naively multiplying this by the number of humans gives an upper bound of about 500 gigabits per second when considering the information throughput of humanity as a whole.

Current frontier LLMs collectively produce [around 10 million tokens per second](https://chatgpt.com/share/67b97194-33c4-8006-ab8e-216aa4515d67)[^sl7uzcn6ko]; this translates to a collective output (and thus maximum throughput) of roughly 100 megabits per second.

These are both upper bounds, and so there's not much reason to directly compare them. I'm not sure exactly what to do with these numbers, though I think they're interesting, and this kind of thinking might in principle ultimately lead to more reasonable estimates of the strategic capacity of humanity and/or AI agents. For example, the concept of "[empowerment](https://en.wikipedia.org/wiki/Empowerment_(artificial_intelligence))" in reinforcement learning is expressed in terms of channel capacity.

[^sl7uzcn6ko]: I haven't carefully checked Deep Research's answer here, but it accords with my basic guess, based on looking at OpenRouter's weekly token chart.
# Proselytizing

Religions can be divided into proselytizing religions (*e.g.* Mormons) who are supposed to recruit new members, and non-proselytizing religions (*e.g.* Orthodox Jews) who are the opposite. Zen Buddhism is a non-proselytizing religion, which makes me a bad Buddhist, because I've dragged three other people to my Zendo so far. All three had a great experience. One has become a regular, and another will return someday.

I didn't sell them on meditation. All three were already sold on meditation. One of them was a Sam Harris Waking Up fan, and another one is really into the *Bhagavad Gita*.

The Sam Harris fan's name is Rowan. Rowan is gay, and grew up in a rural evangelical Christian family. I haven't pressed him for details, but that can't have gone well. You may reasonably deduce that Rowan has a bad history with religion. But he has all the human instincts that respond positively to healthy rituals. Consequently, Rowan loves pseudo-religious rituals and (until I brought him to Zendo) frequently complained to me that modern secular life doesn't have enough rituals.

Was this enough for me to drag him to Zendo? Nope.

What happened next is that Rowan and I were hanging out and he was talking to me about his therapist. Rowan's therapist had recommended that Rowan do some meditation. Rowan followed his therapist's meditation instructions.

Rowan tells me this not knowing that I attend a Rinzai Zen practice center 10 minutes from his home.

I ask Rowan if he'd like to visit my Rinzai Zen practice center.

Rowan asks me, "Why?"

I don't remember the exact words I used, but here is the gist of them. "The stuff your therapist is teaching you is derived from Asian mystical practices, except it has been watered down and purged of anything offending modern Western sensibilities. Your therapist is not a monk. Your therapist is not a yogi. I'm sure your therapist is a great therapist, but your therapist has not personally taken the mystic path to its logical and inevitable conclusion. Do you want the homeopathic version of meditation, or do you want the walk the road that Jesus, Mohammad and Siddhartha once did? Do you like your kombucha, or do you want to drink ethanol from a MiG-25 coolant system, like God intended?"

Rowan: "I am intrigued."

"Go to 1733 S Horton St on Tuesday at 7:30 pm. Do not be late. If you're even slightly late, turn around, go home, and come back next week. Bring $5 cash voluntary donation. When you enter the facility, immediately remove your shoes and socks. Everything else will be explained to you."

Rowan: "Will you be there?"

Me: "No. That's the beginner class. Ping me if you'd like to join me for 1 hour+ of chanting, bowing, and sitting in silence."

I didn't have to say another word.

When Rowan arrived at Zendo, there were no other students. It was just him and the teacher talking together for one hour. The teacher in question is gay too and has the affect of the laughing buddha.

Later, Rowan told me that it was like nothing he had ever seen before.
# Gradual Disempowerment: Simplified

This post is a summary of a paper [recently posted here](https://gradual-disempowerment.ai/) that describes, in my opinion, a very possible scenario that modern society will have to face in the near future.  
  
This post is not really intended for the average LW reader, as most of you probably read the original paper. I wrote this post for adults vaguely aware of the advancements of AI in society.
# List of most interesting ideas I encountered in my life, ranked

1.  **Bayesian thinking**  
    It litterally was n°1 in my list, I was really happy to find this website, will not detail.
2.  **Active ignorance/avoidance and selective attention/participation**  
    Instead of thinking, commenting, or saying something is stupid/bad, ignore/block it and just talk about the other thing that is better. Because just by thinking about/mentionning the thing, you propagate it, starting in your memory and to a greater extent in the world. This is applicable to ideas, people, activities, etc.  
    Participating in something is accepting that that thing is worth participating in.   
    Paying (money, time, attention, action) is voting/supporting.
3.  **Source Criticism**  
    When reading information, take into account who created that information and what are their environment/interests/incentives.
4.  **Memetic Evolution**  
    Natural selection and evolution applied to ideas.
5.  **Subjectivity of Value**  
    Value of a thing is subjective. Loopy example is money: everyone wants money because everyone wants money.
6.  **Life as negentropy**  
    The definition of life is to locally reduce entropy/slow entropy/maximise possibilites while minimising energy consumption/will-to-power in Nietzsche's words. And maybe reversing entropy.
7.  **Identity as Action, Not Essence**  
    Human identity is a consequence of doing/being, not doing/being is a consequence of identity. What you do defines who you are, not who you are defines what you do.
8.  When you die, people come at your burial and the next day, they go back to work.
9.  Emotions are a deprecated way of making decision. Not intuition.

[History of the entire world, I guess](https://www.youtube.com/watch?v=xuCn8ux2gbs)  
  
Curious about other people's list.