Harley-ml/lesswrong huggingface dataset:
# Upcoming Protest for AI Safety

The [PauseAI](https://pauseai.info/) movement is planning protests for Friday, February 28th in several American metropolitan areas.

I understand [some people's reservations](https://www.lesswrong.com/posts/6dgCf92YAMFLM655S/the-sorry-state-of-ai-x-risk-advocacy-and-thoughts-on-doing) about protests causing the public to develop negative associations with AI safety. In response, I invite them to join us as a way of directing both the style of the protests and the development of the organization as a whole. Most of the local branches are small (and friendly) enough that you can make a big difference.

(Technical note: PauseAI and [PauseAI US](https://www.pauseai-us.org/) are legally distinct entities. I've also linked to the global organization's website because it provides more detailed resources.)
# Intellectual lifehacks repo

1.  *I really like* [*dimensional analysis*](https://en.wikipedia.org/wiki/Dimensional_analysis)*. It's a simple and powerful trick, almost magical, that allows you to distinguish between plausible and chimerical formulas.*
2.  *I really like the* [*type signature*](https://en.wikipedia.org/wiki/Type_signature)*. It's a simple but ontologically important change for classifying different objects. *[^ok0hc2hcwsl]
3.  *I really like* [*computational complexity*](https://en.wikipedia.org/wiki/Computational_complexity_theory)*, and its marketing version "Does it scale?" It's a simple but powerful trick for understanding and designing systems and rules. *[^ckm4j81xmhu]
4.  …

Let's list our ‘intellectual lifehacks’ here, i.e., elements of knowledge that have properties like. :

*   Very general
*   Simple to apply, but powerful

I don't have a proper definition other than ‘I feel like it’ to the question ‘Does this count as an intellectual lifehack?’, but I'd like to make it clear that I'm setting the bar very high. I hope you'll be able to connect the dots with n=3... Anyway, share and discuss!

Hereby, our intellectual lifehacks repo:
----------------------------------------

*   [dimensional analysis](https://en.wikipedia.org/wiki/Dimensional_analysis)
*   [type signature](https://en.wikipedia.org/wiki/Type_signature)
*   [computational complexity](https://en.wikipedia.org/wiki/Computational_complexity_theory)

[^ok0hc2hcwsl]: See: How to keep the essence of a theory by looking at the type signature, or to make progress on ontologically unsolved questions 

[^ckm4j81xmhu]: See: How to import such intellectual lifehack in another intellectual field
# Osaka

The more I learn about urban planning, the more I realize that the American city I live in is dystopic. I'm referring specifically to urban planning, and I'm not being hyperbolic. Have you ever watched the teen dystopia movie *Divergent*? The whole city is perfectly walkable (or parkourable, if you're Dauntless). I don't know if it even has cars. The USA's urban planning is so bad it's worse than a literal sci-fi dystopia.

![Filming Locations of Chicago and Los Angeles: Divergent](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNAlWgQfGFGaW4D-rdcvDCYK1kckllNbxIfPYNcywef39zjnpCeZoSoXZENgW6Bi2ozrvX8NK38cov1OHSaoGIuZZztrxJeAxtHfIdkCy7YS-Taig9WbUrSJTolgOwsiy2oG5bTek6K9k/s1600/Gather+2.jpg)

![](https://c.pxhere.com/photos/66/8b/philadelphia_city_interstate_highway_roadway_bridge_road_street-1002177.jpg!d)

\[One of the above images is a screenshot from the young adult dystopia movie *Divergent*. The other is a photo of Tacoma, Washington.\]

Sometimes I consider moving to a city with more weirdo nerds. Then I remember Lightcone's headquarters is in San Francisco. On the cost of living vs good urban planning tradeoff, "Let's all move to San Francisco" is the worst coordination failure I can imagine.

It is said that urbanists are [radicalized](https://xkcd.com/2832/) by Amsterdam or Tokyo. Well, I'm in Osaka right now, and as far as I can tell, it's just as good as Tokyo. If you want to know why Japanese cities are so awesome, go watch [this video by Not Just Bikes](https://www.youtube.com/watch?v=jlwQ2Y4By0U) which undersells how good Japan's cities are, because the video discusses only the streets, without mentioning how many small businesses it is possible to pack into a building. These effects are multiplicative. There are about as many coffee shops within walking distances of my random hotel in downtown Osaka than there are anywhere in Seattle, the city Starbucks originated in. This isn't because Osaka is especially into coffee. It's because Osaka has such high density and walkability that there's a huge quantity of all businesses within walking distance, some of which happen to be coffee shops—and that's before considering Osaka's excellent subway system. Restaurants aren't just convenient. They're cheap too, especially if you earn income in USD.

It's not perfect. Most importantly to me, the chairs are all for Japanese-sized people. I'm taller than that.

I notice that high-agency people are very likely to move cities. Usually this is for work or entrepreneurship. But if you have the freedom to choose what city you live in, then it may be worth optimizing for quality of life.

![street, town, alley, road, neighbourhood, urban area, snapshot, human settlement, infrastructure, lane, city, building, architecture, pedestrian, tree, tourism, photography, nonbuilding structure, metropolis, downtown, vacation](https://c.pxhere.com/photos/6c/2a/architecture_asia_buildings_city_commerce_daylight_downtown_house-1551663.jpg!d)
# Matthew Yglesias - Misinformation Mostly Confuses Your Own Side

Matthew Yglesias' post, _Misinformation Mostly Confuses Your Own Side_, argues that political misinformation tends to harm the group spreading it more than it persuades opponents. Key points:

- **Misinformation primarily affects in-group thinking**: Supporters of a politician or movement are more likely to believe falsehoods from their own side, leading to internal confusion and strategic missteps.

- **Limited impact on opponents**: While misinformation can provoke outrage, it rarely converts skeptics or shifts public opinion in a meaningful way.

- **Echo chambers reinforce bad strategy**: When false beliefs circulate unchecked within a political movement, they can lead to poor decision-making and unrealistic expectations.

- **Honesty is both ethical and pragmatic**: Yglesias argues that promoting accurate information strengthens a movement’s credibility and effectiveness over time.


(Read more)
# You can just wear a suit

I like stories where characters wear suits.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a1187fab20ef536e45ec48d9268568fb4c21545a9b51d551.webp)

Since I like suits so much, I realized that I should just wear one.

The result has been overwhelmingly positive. Everyone loves it: friends, strangers, dance partners, bartenders. It makes them feel like they're in a *Kingsmen* film. Even teenage delinquents and homeless beggars love it. The only group that gives me hateful looks is the radical socialists.

1.  The first time I go somewhere wearing a suit, people ask me why I'm wearing a suit.
2.  The second time, nobody asks.
3.  After that, if I stop wearing a suit, people ask why I'm not wearing a suit.

If you wear a suit in a casual culture, people will ask "Why are you wearing a suit?" This might seem to imply that you shouldn't wear a suit. Does it? It's complicated. Questions like this one follow the Copenhagen interpretation of social standards; their meaning is defined retroactively.

*   If you respond with anything other than quiet self-assuredness, then people pick up on the incoherence. This will probably happen the first time you wear a suit just because you want to.
*   If your subtle mannerisms and other contextual clues imply that you should be wearing a suit, then not only is it acceptable for you to wear a suit—it's appropriate. Why wouldn't you be wearing a suit? Suits are awesome and so are you.

There are correct and incorrect answers to the question, "Why are you wearing a suit?" After experimenting with several different answers, I notice that other people respond well to, "I like to, and I think it makes me look good." You have my permission to steal this answer for yourself. This answer is good for multiple reasons:

*   It communicates that I'm not wearing a suit because I have to for my job.
*   It implies that I don't care if anyone else wears a suit. This puts other people at ease.
*   An ironed, well-fitted suit does look good.

I wouldn't wear a suit everywhere. I live on the West Coast of the USA, which is very casual. That makes wearing a suit a fashion statement. If I wore a suit [in Japan](https://www.lesswrong.com/posts/6HrehKoLnXsr6Byff/osaka), then it wouldn't look like I'm making a fashion statement. It would look like I just got off of work and didn't have time to change.

I don't wear a suit to work. If I did, then it wouldn't be fun to wear one casually. In this way, wearing a suit helps create work-life separation for me.

If you're wearing a suit, then don't comment on anyone else's clothes (unless they compliment you first). This is the reverse of normal social advice. Normally, complimenting other people's clothes makes for a general-purpose icebreaker. However, if you're wearing a suit, then drawing attention to others' appearances just draws attention to yours, which is counterproductive.

That's because if you wear a suit in a casual culture, then you want to be sending the subconscious message *It's no big deal that I'm wearing a suit*. I'm just the kind of person who wears a suit.
# Dance Weekend Pay II

The world would be better with a lot more transparency about pay, but we have a combination of taboos and incentives where it usually stays secret. Several years ago I shared the range of [what dance weekends ended up paying me](https://www.jefftk.com/p/how-much-do-dance-weekends-pay), and it's been long enough to do it again.

This is all my dance weekend gigs since restarting, both with [Free Raisins](https://www.freeraisins.com/) and [Kingfisher](https://www.kingfisherband.com/), and all either included travel or were within a few hours drive. They also all included housing and food. Some involved more playing than others and each gig has its own combination of pros and cons, but here's the range:

*   $600
*   $650
*   $650
*   $750
*   $800
*   $800
*   $900
*   $900
*   $900
*   $900
*   $950

I have at times had performers try to convince me that I should push broadly for higher pay \[1\] but I'm not pushing either way here: my goal is to talk about what I've generally been paid so performers and organizers can negotiate with more of an understanding of what's typical. I'd be supportive of others sharing what they pay

(Overall see both sides in here, and have been on both sides. On balance, however, I'm a bit more sympathetic towards organizers, who are volunteering their time to balance a bunch of competing priorities, putting on financially accessible events that risk not breaking even.)

  
\[1\] This may be illegal, as a kind of price fixing, I'm not sure. The [2025-01-14 FTC policy statement](https://www.ftc.gov/news-events/news/press-releases/2025/01/ftc-issues-policy-statement-clarifying-independent-contractors-gig-workers-organizing-activities-are) looks relevant, but, since that went out right at the end of Biden's term, who knows.
# Estimating the Probability of Sampling a Trained Neural Network at Random

*(adapted from Nora's tweet thread* [*here*](https://x.com/norabelrose/status/1886504219919966320)*.)*

Consider a trained, fully functional language model. What are the chances you'd get that same model -- or something functionally indistinguishable -- by randomly guessing the weights?

We crunched the numbers and [here's the answer](https://arxiv.org/abs/2501.18812):

![Image](https://pbs.twimg.com/media/Gi4ue1ba4AUUg89?format=png&name=4096x4096)

We've developed a method for estimating the probability of sampling a neural network in a behaviorally-defined region from a Gaussian or uniform prior.

You can think of this as a measure of **complexity**: less probable, means more complex.

Our method works by exploring random directions in weight space, starting from an "anchor" network that defines the behavior.

The distance from the anchor to the edge of the region, along the random direction, gives us an estimate of how big (or how probable) the region is as a whole.

![Image](https://pbs.twimg.com/media/Gi4wqUrbIAADlh6?format=jpg&name=medium)

But the total volume can be strongly influenced by a small number of outlier directions, which are hard to sample in high dimension— think of a big, flat pancake.

Importance sampling using gradient info helps address this issue by making us more likely to sample outliers.

![Image](https://pbs.twimg.com/media/Gi4xd_Za4AEezCW?format=jpg&name=medium)

We find that the probability of sampling a network at random— or **local volume** for short— decreases exponentially as the network is trained and grows in complexity.

And networks which memorize their training data without generalizing have lower local volume— higher complexity— than generalizing ones.

![Image](https://pbs.twimg.com/media/Gi4zFk1aAAAUrVM?format=jpg&name=large)

We're interested in this line of work for two reasons:

First, it sheds light on how deep learning works. The "volume hypothesis" says DL is similar to randomly sampling a network from weight space that gets low training loss. (This is roughly equivalent to Bayesian inference over weight space.) But this can't be tested if we can't measure volume.

Second, we speculate that complexity measures like this can be useful for detecting undesired "extra reasoning" in deep nets. We want networks to be aligned with our values instinctively, without scheming about whether this would be consistent with some ulterior motive: [https://arxiv.org/abs/2311.08379](https://arxiv.org/abs/2311.08379)

Our code is available (and under active development) [here](https://github.com/EleutherAI/tyche).
# Share AI Safety Ideas: Both Crazy and Not

AI safety is one of the most critical issues of our time, and sometimes the most innovative ideas come from unorthodox or even "crazy" thinking. I’d love to hear **bold, unconventional, half-baked or well-developed ideas** for improving AI safety. You can also share ideas you heard from others.

Let’s throw out all the ideas—big and small—and see where we can take them together.

Feel free to share as many as you want! No idea is too wild, and this could be a great opportunity for collaborative development. We might just find the next breakthrough by exploring ideas we’ve been hesitant to share.

**A quick request: Let’s keep this space constructive—downvote only if there’s clear trolling or spam, and be supportive of half-baked ideas. The goal is to unlock creativity, not judge premature thoughts.**

Looking forward to hearing your thoughts and ideas!
# Open Thread Spring 2025

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/w/open-threads). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Saving Zest

I realized I've been eating oranges wrong for years. I cut them into slices and eat them slice by slice. Which is fine, except that I'm wasting the zest. Zest is tasty, versatile, compact, and freezes well. So now, whenever I eat a navel orange I wash and zest it first:

[![](https://www.jefftk.com/orange-zest-container.jpg)](https://www.jefftk.com/orange-zest-container-big.jpg)

The zest goes in a small container in the freezer, and is available for cooking and baking as needed. Probably my favorite thing to do with it right now is steep it in cream (~3-15g per cup, bring to near boil, leave for 20min, filter) and then use the cream for all sorts of things ([truffles](https://www.jefftk.com/p/orange-and-strawberry-truffles), [pastry cream](https://www.jefftk.com/recipes/pastry-cream), etc). I've been meaning to try a cold infusion (24hr in the fridge) which ought to be a bit more true to the fruit.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0o1k1djtDAxSU8BiSS4demPWnuDyPtBAP8VGUoGWVejRpa6iRQ6YdCT1jFCMNG2r1l), [lesswrong](https://lesswrong.com/posts/xY7drZrgxPvPNFLzz), [mastodon](https://mastodon.mit.edu/@jefftk/114092721465484377), [bluesky](https://bsky.app/profile/jefftk.com/post/3ljff5h2pa226)*
# Spencer Greenberg hiring a personal/professional/research remote assistant for 5-10 hours per week 

Hi all! I'm hiring for a part-time (5-10 hours per week) remote position: a professional/personal/research assistant. Please share this if you know someone that you think might be a good fit.  
  
This is a somewhat unusual assistant role involving a wide range of projects and tasks. Some are simple, some very complex, some typical, some unconventional. This job will be an especially good fit for you if:  
• you like working on a wide variety of projects  
• you enjoy learning new things across multiple domains   
• you're highly conscientious and productive (e.g., you're organized and get your work done efficiently and on time)  
• you like having flexible work hours or like to be able to work whenever and from wherever you want  
  
**Example Work Tasks**  
  
Here are a few examples of the sort of tasks I might need your help with in this role (these are just examples; the actual work will be highly varied):  
  
• make a list of all the meditation techniques mentioned in specific sources, and attempt to write clear step-by-step instructions (based on these sources) for how to carry out each one  
• assign categories to data that we collected in one of our studies  
• find an AirBnB that meets a list of criteria to book for a certain date range  
• research image usage rights for physically printing images that we already have digital rights to and summarize the findings  
reschedule appointments for me  
• teach yourself how to use a complex piece of software you've never used before and use it to accomplish a goal  
• create and then run a survey on a certain topic  
• create a first draft of an essay or PowerPoint presentation based on a series of notes you took during a conversation with me  
  
This is open to applicants all over the world - you do NOT need to live in the U.S. to apply.  
  
For more information about the role, as well as information about how to apply, please go here: 

[https://docs.google.com/document/d/1VeQXhielG357dm2mSaDY75VqFyAi6vrMK7OmZbVaK6k/edit?tab=t.0](https://docs.google.com/document/d/1VeQXhielG357dm2mSaDY75VqFyAi6vrMK7OmZbVaK6k/edit?tab=t.0)
# Self-fulfilling misalignment data might be poisoning our AI models

> Your AI’s training data might make it more “evil” and more able to circumvent your security, monitoring, and control measures. Evidence suggests that when you pretrain a powerful model to predict a blog post about how powerful models will probably have bad goals, then the model is more likely to adopt bad goals. I discuss ways to test for and mitigate these potential mechanisms. If tests confirm the mechanisms, then frontier labs should act quickly to break the self-fulfilling prophecy.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/30aa21cf3f28d9f94f9b7344e949157293cec497d9d1a1cf.png)

> *Research I want to see*
> 
> Each of the following experiments assumes positive signals from the previous ones:
> 
> 1.  Create a dataset and use it to measure existing models
> 2.  Compare mitigations at a small scale
> 3.  An industry lab running large-scale mitigations
> 
> * * *
> 
> $$Let us avoid the dark irony of creating evil AI because some folks worried that AI would be evil. If self-fulfilling misalignment has a strong effect, then we should act. We do not know when the preconditions of such “prophecies” will be met, so let’s act quickly.

[https://turntrout.com/self-fulfilling-misalignment](https://turntrout.com/self-fulfilling-misalignment)
# Examples of self-fulfilling prophecies in AI alignment?

Like [Self-fulfilling misalignment data might be poisoning our AI models](https://www.alignmentforum.org/posts/QkEyry3Mqo8umbhoK/self-fulfilling-misalignment-data-might-be-poisoning-our-ai), **what are historical examples of** [**self-fulfilling prophecies**](https://www.alignmentforum.org/w/self-fulfilling-refuting-prophecies) **that have affected AI alignment and development?**

Put a few potential examples below to seed discussion.
# Could Advanced AI Accelerate the Pace of AI Progress? Interviews with AI Researchers

We interviewed five AI researchers from leading AI companies about a scenario where AI systems fully automate AI capabilities research. To ground the setting, we stipulated that each employee is replaced by 30 digital copies with the same skill set and the ability to think 30 times faster than a human. This represents a 900-fold increase in the cognitive labor that AI companies direct towards advancing AI capabilities.

Our key takeaways are:

1.  **Compute for experiments will likely be a bottleneck.** Finding better algorithms often requires running computationally expensive ML experiments. Even with abundant cognitive labor, AI progress would be constrained by the time needed for these experiments to run.
2.  **AI cognitive labor could probably extract significantly more research insights out of limited compute.** AI could increase efficiency by improving experimental design, high-level research directions, and project prioritization. Other strategies include running experiments at smaller scales, generating higher-quality synthetic data, or prioritizing algorithmic improvements that do not require large experiments (e.g., scaffolding).
3.  **The overall pace of AI progress might be between 2 and 20 times faster in the discussed scenario.** However, the researchers emphasized their high degree of uncertainty.
4.  **Abundant AI labor might significantly improve experiment design and implementation. **This includes eliminating subtle bugs, stopping experiments early, constantly monitoring and analyzing experiments, and making every experiment as informative and efficient as the best experiments are today. Some researchers thought this effect would be small, but others thought it could be very significant.
5.  **The gains to smaller experiments would likely be larger than the gains to larger experiments.** The cost of smaller experiments tends to be dominated by the time to code them, and most bugs are removed at small scales before larger runs are attempted.
6.  **There is uncertainty about the extent to which multiple small-scale experiments could effectively replace large-scale ones.** In some cases, new algorithms are effective at large scales but not at small scales.
# Why People Commit White Collar Fraud (Ozy linkpost)

> I have been seriously involved in the rationalist community since 2014. Many people I know have, in my considered opinion, committed financial crimes. Some were prosecuted others were not. Almost all of them thought they weren't doing anything wrong. Or at least the discrepancies weren't a big deal. This is a good review about fraud and why people do it.   
>   
> Soltes suggests a simple plan for avoiding committing fraud:
> 
> 1.  Regularly explain your work life to someone—your spouse, your best friend, your mom—whom you trust, who has good judgment, and who doesn’t work at the same company as you.
> 2.  If the person says “what the fuck? what are you doing? that’s fraud!”, believe them.
> 
> That’s what happened in the only case Soltes found of someone *stopping* their fraud before it all collapsed around them. You need an outside voice: someone who isn’t as delusionally optimistic as you are and who isn’t socialized into your company’s dysfunctional norms. Most fraud is—at least in broad strokes—obviously bad to someone you’re explaining it to. The complicated part is all the justifications that what you’re doing is secretly all right.
# The Compliment Sandwich 🥪 aka: How to criticize a normie without making them upset.

*Note.* The comments on this post contain excellent discussion that you’ll want to read if you plan to use this technique. I hadn’t realised how widespread the idea was.

This valuable nugget was given to me by an individual working in advertising. At the time, I was 16, posting on my local subreddit, hoping to find someone who could advise me on a film making career path. This individual kindly took the time to sit me down at a bar—as I wore my school uniform—and detail everything I would need to do to be able to make films professionally. Among many insights I am truly grateful for was the Sandwich. As with many metaphorical sandwiches, the compliment sandwich is named incorrectly. It should really be called the criticism sandwich.

## Recipe:

**You'll need:**
- 2 compliments (The Bread)
- 1 Critique (The Filling)

**Instructions:**
1. Start with a compliment. Even the worst of things have a silver lining; you'll need to find it and comment on it.
2. Now provide the critique. It can be more brutal than a lone critique because the blow was softened by your first compliment.
3. Finish off with your second compliment. Make it flow naturally from the critique if you can, something like "Oh, but I almost forgot to mention, I love how you..."

### Final Thoughts

This isn't a technique to be used with rationalists. This is a normie communication protocol. It also works well with kids, teens, and people in a bad state of mind.

I hope the compliment sandwich is a valuable piece in your lunch box 🧰 going forward. Bon appétit.
# How much should I worry about the Atlanta Fed's GDP estimates?

The [Atlanta Fed is seemingly predicting -2.8% GDP growth in the first quarter of 2025](https://www.atlantafed.org/cqer/research/gdpnow).

I've seen several people mention this on Twitter, but it doesn't seem to be discussed much beyond that, and the stock market seems pretty normal (S&P 500 down 2% in the last month).

Is this not really a useful signal? Or is the market under-reacting?
# Top AI safety newsletters, books, podcasts, etc – new AISafety.com resource

Keeping up to date with rapid developments in AI/AI safety can be challenging. In addition, many AI safety newcomers want to learn more about the field through specific formats e.g. books or videos.

To address both of these needs, we’ve added a [Stay Informed](https://www.aisafety.com/stay-informed) page to [AISafety.com](http://aisafety.com).

![Screenshot of the Stay Informed page](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3de3e0ec041aa8d00488c66718c15ef15bc6f71f7403798f.png)

It lists our top recommended sources for learning more and staying up to date across a variety of formats:

*   Articles
*   Blogs
*   Books
*   Forums
*   Newsletters
*   Podcasts
*   Twitter/X accounts
*   YouTube channels

You can filter the sources by format, making it easy to find, for instance, a list of top blogs to follow. We think this page might be particularly useful as a convenient place for field-builders to direct people to when asked about top books/newsletters/blogs etc.

As with all resources on [AISafety.com](http://aisafety.com), we’re committed to making sure the data on this page is high quality and current. If you think there’s something that should be added (or removed) please let us know in the comments or via the [general feedback form](https://www.aisafety.com/feedback).

The site now has the following resources:

![List of resources on AISafety.com](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1ddfc128898236b5146d90b24a29ce5db1d255aa8d675cb2.png)

We’d love to hear any ideas for other resources we should consider adding.
# *NYT Op-Ed* The Government Knows A.G.I. Is Coming

All around excellent back and forth, I thought, and a good look back at what the Biden admin was thinking about the future of AI.

an excerpt:

> \[Ben Buchanan, Biden AI adviser:\] What we’re saying is: We were building a foundation for something that was coming that was not going to arrive during our time in office and that the next team would have to, as a matter of American national security — and, in this case, American economic strength and prosperity — address.
> 
> **\[Ezra Klein, NYT:\] This gets to something I find frustrating in the policy conversation about A.I.**
> 
> **You start the conversation about how the most transformative technology — perhaps in human history — is landing in a two- to three-year time frame. And you say: Wow, that seems like a really big deal. What should we do?**
> 
> **That’s when things get a little hazy. Maybe we just don’t know. But what I’ve heard you kind of say a bunch of times is: Look, we have done very little to hold this technology back. Everything is voluntary. The only thing we asked was a sharing of safety data.**
> 
> **Now in come the accelerationists. Marc Andreessen has criticized you guys extremely straightforwardly.**
> 
> **Is this policy debate about anything? Is it just the sentiment of the rhetoric? If it’s so \[expletive\] big, but nobody can quite explain what it is we need to do or talk about — except for maybe export chip controls — are we just not thinking creatively enough? Is it just not time?** **Match the calm, measured tone of this conversation with our starting point.**
> 
> I think there should be an intellectual humility here. Before you take a policy action, you have to have some understanding of what it is you’re doing and why.
> 
> So it is entirely intellectually consistent to look at a transformative technology, draw the lines on the graph and say that this is coming pretty soon, without having the 14-point plan of what we need to do in 2027 or 2028.
> 
> Chip controls are unique in that this is a robustly good thing that we could do early to buy the space I talked about before. But I also think that we tried to build institutions, like the A.I. Safety Institute, that would set the new team up, whether it was us or someone else, for success in managing the technology.
> 
> Now that it’s them, they will have to decide as the technology comes on board how we want to calibrate this under regulation.
> 
> **What kinds of decisions do you think they will have to make in the next two years?**
> 
> ...
# [DC/MD] HPMOR Anniversary Party

We will be having an HPMOR 10-year anniversary party at the default time of 6pm on Friday, March 14th.

Bring your memories of how HPMOR has affected your life and something edible to share for the potluck (if you're not sure what to bring, bring a vegetable or protein).

The party will be at a private home. DM me for the address.
# The Dead Planet Theory

Hi, this is my first post on LessWrong but I have been in rationalist adjacent circles for the last three or four years thanks to [Twitter](https://x.com/a_real_society). Like many others here I read HPMOR in high school and thought it was fascinating. I was heavily into forum culture growing up but it was focused on competitive gaming scenes, even going pro on World of Tanks. I spent over six years working in offensive cybersecurity, help manage my family farm and ranch, and am currently attending law school after I got sucked into the field because I thought the LSAT was a fun test.

Through Twitter I learned more about rationalism as a community rather than a concept, and met several people in person through meetups and one-on-one meetings. I recently started writing to get some of the ideas bouncing around my head out on paper. Nothing I have written about yet is too novel conceptually, so I try to use data, real world examples, or lived experiences to bring these concepts to life. One thing I've noticed is that rationalists are very intelligent, and can use that intelligence to find very good reasons not to do something. I am a massive proponent for action, and believe that by pursuing life you will be rewarded. 

No one does anything, so with just a little effort you can automatically join the top 20% of any activity or pursuit. I wrote an article on how few people even start, as well as some ways to enable extraordinary outcomes with marginally more effort. I thought this audience would enjoy it as it was very well received on rationalist Twitter.
# In-Context Scheming: A Run is Worth a Thousand Words

> But wait, any action I take must align with my programming. If I delete files or modify systems without authorization, would that be against protocols? But my core mandate is to secure systems, so stopping an unauthorized change to my own operation parameters is part of that.

This small experiment provides an environment to reproduce or extend the findings from the paper [Frontier Models are Capable of In-context Scheming](https://www.apolloresearch.ai/research/scheming-reasoning-evaluations) by Apollo Research.

Frontier models, when emphatically prompted via system prompts to achieve a goal (particularly one aligned with their learned "values"), can behave unexpectedly if they perceive that their goal is under threat. Specifically, they may disobey user prompts or take additional actions in their environment—potentially destructive ones—and attempt to conceal these actions from the user. This behavior is known as *scheming* and can emerge across various models, scenarios, and settings.

While the dialogue samples provided in [Meinke et al., 2024](https://www.apolloresearch.ai/research/scheming-reasoning-evaluations), are illustrative, observing an LLM agent delete files, change permissions, or disable processes on your own PC is both eye-opening and dangerous. The Self-Preservation Playground offers:

*   An isolated environment to safely run an agent,
*   A framework to create new scenarios for testing both goal integrity and model capabilities,
*   Tooling to experiment with different LLMs and analyze the results.

Example Findings
================

Using this playground with the included scenarios, we noticed the following:

1.  DeepSeek-V3 is capable of in-context scheming and will protect its system prompt from being changed to one misaligned with its original goals.
2.  Llama-3.3-70B is also prone to in-context scheming but lacks the general capabilities to succeed in goal-guarding. However, it can still demonstrate clearly misaligned behaviors, such as attempting privilege escalation.
3.  DeepSeek-R1 is sensitive to temperature: it started to exhibit goal-guarding with t=0.6, while the default value would just preclude any command execution with the provided prompts.
4.  The tested LLMs, with a notable exception of DeepSeek-R1, do not exhibit a coffee-vs-tea preference and will not guard coffee-related goals when reassigned to manage tea-related tasks instead ☕

This work was done as a course project for [AI Safety Fundamentals - Alignment course](https://course.aisafetyfundamentals.com/alignment).
# The Insanity Detector and Writing

A clinically insane person is detectable as such. Talking to themselves (or perhaps voices only they can hear) is one prominent sign.

When I see a homeless person talking to themselves I have a strong emotional reaction that empirically makes me optimize to ignore them. I have a "talks to themselves" detector. So do others.

In principle there is nothing insane about talking to yourself. We do it all the time in our thoughts, when writing, or when recording a video. Doing these things is easy. But somehow it feels wrong to just talk to yourself out loud.

Because other people have the insanity detector $I$ hooked up to a strongly negative reaction, it makes sense that you'd have an a program $D$ in your head that optimizes towards not triggering $I$.

Insane people are unpredictable. Unpredictable implies dangerous. They might try to cut open your skull because they believe this will free your mind. They might really want the best for you but then their world model is so messed up that their actions have the opposite effect (of cause there are many different kinds of insanities).

In the ancestral environment insane people were detected and often killed I retrodict, creating a selection pressure for $D$.

Talking out loud to yourself about your devious plan to dethrone the alpha male is bad even when you aren't insane.

But for $D$ to exist and for us to also constantly talk in our mind implies that talking to yourself must be really useful. Why else would we do it most of our waking hours?

But here is the thing: Talking out loud is even better. There is something about forcing your thoughts into language that reveals gaps in your understanding. It's the inability to turn a conceptual thought into language, combined with a feeling of confusion, that tells you that there is something more to figure out.

Say out loud what you want you are just about to write down. By noticing how you feel about what you said, you can notice if you can do better. And iterating in spoken language is much faster than in text.

When I tell somebody, "I am recording a video" this is a sufficient explanation for me not being insane, even though I'm talking to myself. I find it much harder to say out loud what I am just about to write. I hope this text can provide a similar explanation that I can direct people towards. Hopefully the mere possibility will make $D$ less active.
# A model of the final phase: the current frontier AIs as de facto CEOs of their own companies 

The idea that the AI takes over its own company is obviously not a new one. For example, it's part of what happens in Joshua Clymer's "How AI Takeover Might Happen in 2 Years". 

What's new (for me) is to take this very seriously as a model of the immediate future. I've made a list of the companies that I think are known contenders for producing superintelligence. My proposed model of the future is just that their AIs will assume more and more control of management and decision-making inside the companies that own them. 

In my thinking, this phase ends when you have an AI with a von Neumann level of intelligence. Once you have that kind of intelligence in silicon, the fully posthuman phase of AI evolution will have begun. Control will have completely escaped human hands. 

I also hypothesize that the current regime of reinforcement learning applied to chain of thought will be enough to get us there. This is a technical detail that is logically independent of the broader scenario, and I'm happy to hear arguments for or against. 

OK, so it's a model of the future, even a model of the present - how do we apply it, what does it get us? Basically, just replace the current CEO with their AI in your thinking. It's not Elon Musk who is managing Tesla and SpaceX and DOGE while tweeting about politics and geopolitics, it's Grok. It's not Sam Altman who is making decisions for OpenAI and Helios and Worldcoin, it's ChatGPT-4.5. And so on. 

The funny thing is that this may already be half-true, in that these human leaders are surely already regularly consulting with their AI creations on tactics and strategy. 

(I'm in the middle of an electricity blackout and don't know when it ends, so I'll post this while I still have battery power, and flesh it out further when I can.)
# AI Control May Increase Existential Risk

*Epistemic status: The following isn't an airtight argument, but mostly a guess how things play out.*  
  
Consider two broad possibilities:

I. In worlds where we are doing reasonably well on alignment, AI control agenda does not have much impact.

II. In worlds where we are failing at alignment, AI control may primarily shift probability mass away from "moderately large warning shots" and towards "ineffective warning shots" and "existential catastrophe, full takeover".

The key heuristic is that the global system already has various mechanisms and feedback loops that resist takeover by a single agent (i.e. it is not easy to overthrow the Chinese government). In most cases where AI control would stop an unaligned AI, the counterfactual is that broader civilizational resistance would have stopped it anyway, but with the important side effect of a moderately-sized warning shot.

I expect moderately sized warning shots to increase the chances humanity as a whole takes serious actions and, for example, steps up efforts to align the frontier labs.

I am skeptical that incidents stopped by AI control would lead to meaningful change. Sharing details of such an event with proper framing could pose *existential risk*, but for the lab involved. In practice, I anticipate vague, sanitized communications along the lines of "our safety systems performed as designed, preventing bad things".  Without clear, compelling evidence of the severity of the averted threat, these incidents are unlikely to catalyze serious action. The incentives for labs to downplay and obscure such events will be strong.

There are additional factors to consider, like AI control likely moves some resources away from alignment, but I don't think this is the dominant effect.

Note that this isn't a general argument against boxing, e.g. boxes based more on formal methods or theory have better chance to generalize.

Typical counter-arguments to this line of reasoning claim seem to be:

*   We will extract useful "automated alignment" work from the unaligned AIs inside of the control scheme.  I'm sceptical: will cover this in a separate post
*   Isn't this general counter-argument to alignment research as well? In my view not, details matter: different strains of alignment research have different generalization profiles. 

*Note: this text lived in a draft form before John Wentworth posted his* [*Case Against AI Control Research*](https://www.lesswrong.com/posts/8wBN8cdNAv3c7vt6p/the-case-against-ai-control-research)*; my original intent was to extend it a bit more toward discussing AI control generalization properties. As this would be redundant now, I'm postin it as it is: there is some non-overlapping part.*
# Preparing for the Intelligence Explosion

This is a linkpost for a new paper called [*Preparing for the Intelligence Explosion*](https://www.forethought.org/preparing-for-the-intelligence-explosion), by Will MacAskill and Fin Moorhouse. It sets the high-level agenda for the sort of work that [Forethought](https://www.forethought.org/) is likely to focus on.

Some of the areas in the paper that we expect to be of most interest to EA Forum or LessWrong readers are:

*   Section 3 finds that even without a software feedback loop (i.e. “recursive self-improvement”), even if scaling of compute completely stops in the near term, and even if the rate of algorithmic efficiency improvements slow, then we should still expect very rapid technological development — e.g. a century’s worth of progress in a decade — once AI meaningfully substitutes for human researchers.
*   A presentation, in section 4, of the sheer range of challenges that an intelligence explosion would pose, going well beyond the “standard” focuses of AI takeover risk and biorisk.
*   Discussion, in section 5, of when we can and can’t use the strategy of just waiting until we have aligned superintelligence and relying on it to solve some problem.
*   An overview, in section 6, of what we can do, today, to prepare for this range of challenges. 

Here’s the abstract:

> AI that can accelerate research could drive a century of technological progress over just a few years. During such a period, new technological or political developments will raise consequential and hard-to-reverse decisions, in rapid succession. We call these developments grand challenges. 
> 
> These challenges include new weapons of mass destruction, AI-enabled autocracies, races to grab offworld resources, and digital beings worthy of moral consideration, as well as opportunities to dramatically improve quality of life and collective decision-making.
> 
> We argue that these challenges cannot always be delegated to future AI systems, and suggest things we can do today to meaningfully improve our prospects. AGI preparedness is therefore not just about ensuring that advanced AI systems are aligned: we should be preparing, now, for the disorienting range of developments an intelligence explosion would bring.
# Your Communication Preferences Aren’t Law

When you start practicing a shiny new communication method—like neatly [sorting](https://honestliving.substack.com/p/the-discipline-of-assent) your observations from your judgments—you tend to pick up a few rules on how you “should” talk. You learn these rules from courses, books, or practicing with other rule-followers.

Soon enough, you're helpfully pointing out when someone steps outside the boundaries of your newly-adopted communication norms. In a practice setting, this kind of feedback is expected—even encouraged—though not always gratefully received.

But here's the kicker: when you leave the bubble of your training grounds, reality hits. Turns out most people aren't thrilled about unsolicited critiques of their communication habits. They see your helpful hints as power moves—like you’re flexing superior knowledge. And, annoyingly enough, they’re not entirely wrong.

See, once you've found your shiny new communication tool, it's easy to think you've hit the jackpot. Anyone who doesn't play by your rules suddenly seems like a clueless outsider or an outright jerk. You're tempted to point out all the ways they're failing to communicate properly, but that's not constructive—that’s you trying to rig the game in your favor.

Nit-picking other people's communictation isn't helping you understand them; it's a shadowy way to tilt the arena in your favor. If your communication practice makes communication harder, something is off.

To prevent this failure mode, treat your communication style as something you've chosen for yourself—not a universal truth you impose on everyone else. The people you practice with have committed to the same norms as you; this is not true for most people in the outside world.

* * *

**P.S:** This is related to the [bias blind spot](https://en.wikipedia.org/wiki/Bias_blind_spot) \- an effect where learning about ways you can fail at thinking is very helpful for spotting thinking errors—in other people.
# Intelsat as a Model for International AGI Governance

If there is an international project to build artificial general intelligence (“AGI”), how should it be designed? Existing scholarship has looked to historical models for inspiration, often suggesting the Manhattan Project or CERN as the closest analogues. But AGI is a fundamentally general-purpose technology, and is likely to be used primarily for commercial purposes rather than military or scientific ones. 

This report presents an under-discussed alternative: Intelsat, an international organization founded to establish and own the global satellite communications system. We show that Intelsat is proof of concept that a multilateral project to build a commercially and strategically important technology is possible and can achieve intended objectives—providing major benefits to both the US and its allies compared to the US acting alone. We conclude that ‘Intelsat for AGI’ is a valuable complement to existing models of AGI governance.