Harley-ml/lesswrong huggingface dataset:
# What are the two contradictory theories of how to evaluate counterfactuals?

In [this comment thread](https://www.lesswrong.com/posts/R8muGSShCXZEnuEi6/a-defense-of-functional-decision-theory?commentId=MStbjBiLFYgdCupKJ) on the 2021 post “A Defense of Functional Decision Theory”, [@So8res](/users/so8res?mention=user) wrote:

> ...Also, just to be clear, you’re aware that these are two different internally-consistent but contradictory theories of how to evaluate counterfactuals? Like, we can be pretty confident that there’s no argument a CDT agent can hear that causes them to wish to adopt FDT counterfactuals, and vice versa. Humans come equipped with both intuitions (I can give you other problems that pump the other intuitions, if you’d like), and we have to find some other way to arbitrate the conflict.

Following up on my reply that I didn’t know what he was talking about, he recommended some reading:

> The canonical explanatory text is the [FDT paper](https://arxiv.org/pdf/1710.05060) (PDF warning) (that the OP is responding to a critique of, iirc), and there’s a bunch of literature on LW (maybe start at the wiki page on UDT? Hopefully we have one of those) exploring various intuitions. If you’re not familiar with this style of logic, I recommend starting there (ah look we do have a UDT [wiki page](https://wiki.lesswrong.com/wiki/Updateless_decision_theory)). I might write up some fresh intuition pumps later, to try to improve the exposition. (We’ve sure got a lot of exposition if you dig through the archives, but I think there are still a bunch of gaps.)

I followed both of those links, and was not enlightened. The linked FDT paper had this bit:

> In short, CDT and FDT both construct counterfactuals by performing a surgery on their world-model that breaks some correlations and preserves others, but where CDT agents preserve only causal structure in their hypotheticals, FDT agents preserve all decision-relevant subjunctive dependencies in theirs.

(This follows a rather technical section, which I have little confidence in having understood correctly.)

Has the dichotomy that [@So8res](/users/so8res?mention=user) refers to been clearly explained anywhere? If not—can anyone explain it now? The relevant questions are:

1. What are the “two different internally-consistent but contradictory theories of how to evaluate counterfactuals”?
2. What are the intuitions for each (which humans come equipped with)?
3. What problems pump those intuitions?
# Lighthaven Sequences Reading Group #44 (Tuesday 7/29)

*Note: We will be meeting in building B this week.*

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights) and [the Best of LessWrong](https://www.lesswrong.com/bestoflesswrong). The mandatory readings this week are

1.  [The Power of Intelligence](https://www.lesswrong.com/s/MH2b8NfWv22dBtrs8/p/aiQabnugDhcrFtr9n)
2.  [An Alien God](https://www.lesswrong.com/s/MH2b8NfWv22dBtrs8/p/pLRogvJLPPg6Mrvg4)

The posts should take around 15-20 minutes to read if you've read them before, and 30-60 mins if it's your first time.

The event starts at 6:30, and doors open 6pm (yes we will let you in early if you get here). At 6:30 we'll welcome people, give a few announcements, and then split into groups. Pizza will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# Where are the AI safety replications?

A popular topic on LessWrong is that [much of science fails to replicate](https://www.lesswrong.com/w/replication-crisis), because of bad incentives and increasingly complex statistics. I think most of us want more replication studies to be published, but it's of course very difficult to effect that change in mainstream science.

For AI safety specifically, though, it seems like there's more opportunity to change these norms. So what's missing exactly? External funding incentives? Better scientific rigor making it less important? People still consider it low-status? A simple lack of coordinated effort to do it? Something else entirely?
# The Purpose of a System is what it Rewards

It’s become fashionable recently to say that [the purpose of a system is what it does](https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does) \- the true purpose of an institution is often different from what it publicly claims, and is better determined by observing what it does. Scott Alexander wrote a thoughtful takedown of this, claiming “[Obviously The Purpose Of A System Is Not What It Does](https://www.astralcodexten.com/p/come-on-obviously-the-purpose-of)” \- often an institution is genuinely trying to do a good thing (e.g. treat patients) and the fact they do so imperfectly (e.g. some patients die) does not mean such imperfections are the goal.

I think a better framing is that the purpose of a system is what it rewards people for doing.

Meta says that their purpose is “to build the future of human connection” and used to say their mission was to “make the world more open and connected”. But Meta employees aren’t actually judged by how well they serve those high valued goals. Employees are judged on their ability to get people to spend more time staring at their screens clicking on ads (source: I worked there). So I think it’s fair to say that “get people to spend time staring at screens clicking on ads” is the purpose of Meta.

Someone I know works as a hospital doctor. Her performance isn’t judged based on patient outcomes, but on whether patients feel cared for, whether patients get diagnosed with treatable conditions, and how much the hospital can bill insurers for treating patients. So I think it’s fair to say that the purpose of that hospital is to make people feel cared for, diagnose them with things, and charge as much as they can to insurance. Some health benefits obviously get provided along the way, but since they aren’t what people are rewarded for, they aren’t the purpose.

If you work on a political campaign, your salary depends on bringing in lots of money from donors, but is mostly disconnected from whether the country improves as a result of your party’s policies. Thus I think it’s fair to say the purpose of a political party is to get people to donate money - often by making people angry and afraid.

On the other hand, when I worked at Google Search, it was genuinely true that the primary thing people were rewarded for was making search useful for users. The ads team of course has a separate purpose of making money by getting people to click on the ads they placed on the results.

Similarly, I’m pretty sure no police department rewards police officers for shooting innocent people. However I assume they do reward people for bringing down crime statistics in ways favorable to whoever is mayor, and I can imagine they may under-reward building good relationships with the community.

The purpose of a system definitely isn’t what it says it is, and it’s not fair to say all negative outcomes are intentional,, but I think it is fair to judge a system by what it rewards, and what a system rewards is often actually quite legible.
# What are non-obvious class markers?

Class discrimination is very real, and it is often useful to be able to feign a higher class than one was born into. Some of these class markers are obvious: graduating from an expensive private school, having lots of wealth but not flaunting it, knowing other wealthy/high-status people, etc.

However, some class markers are much less obvious. Given that the vast majority of most people's social lives are spent with people of roughly the same class, it can be very difficult to learn these markers.

I was reminded of this recently, when I befriended someone of a lower-class background than mine but who had recently come into more money. Talking to her, I was struck by just how obvious it was that she had not grown up middle class.

I want to know what markers likely make my upper middle class background obvious to some, but that I never think about because of my relatively homogenous social circles, and more generally class markers that might be hard to notice because of one's background.

Since "upper middle class" can mean a lot of things, here are some facts which may give you a better idea of my background:

*   Growing up, my family's household income was ~70th percentile in our city.
*   I went to a public school, but a fairly good one (maybe 95th percentile for the US).
*   I have a safety net from my family, but I don't have a trust fund or inheritance which makes work optional.
# I wrote a song parody 

Since Tom Lehrer passed away recently, I thought I'd honor him by adapting one of his songs to be about a more recent existential risk. Presenting...

"We Will All Go Together When We Go (ASI version)"

When you attend a funeral,  
It is sad to think that sooner or  
Later those you love will do the same for you.  
And you may have thought it tragic,  
Not to mention other adjec-  
Tives, to think of all the weeping they will do.  
But don't you worry.

No more ashes, no more sackcloth.  
And an armband made of black cloth  
Will some day never more adorn a sleeve.  
For if the 'bot that slaughters you  
Kills your friends and neighbors too,  
There'll be nobody left behind to grieve.  

And we will all go together when we go.  
What it's planning, we're not smart enough to know.  
When ASI doesn't need us  
It might just neglect to feed us!  
But we all will go together when we go.  

We will all go together when we go.  
As we watch the fatal data centers grow  
No one will have the endurance  
To collect on their insurance,  
Lloyd's of London will be loaded when they go.  

Oh, we will all fall together when we fall.  
Disassembled for our atoms, one and all.  
Ripped apart by nanobots inside  
As the Earth is paperclippified  
Yes, we will all fall together when we fall.

ASI makes our dreams come true  
Then squishes us beneath its shoe  

And we will all cease together when we cease.   There'll be nobody left to give a speech.  
With complete participation  
In that grand depopulation,  
Over eight billion folks will rest in peace.

Oh, we will all end together when we end.  
From its purpose, ASI, it will not bend.  
So shout out Hallelujah  
When the robots come to shoot ya  
And let it be the last breath you expend!

Oh, we will all burn together when we burn.  
There'll be no need to stand and wait your turn.  
So we'll say our last snappy quotes  
As our names go in the Death Notes  
And we'll all drop our agendas and adjourn.

You will all go directly to your respective Valhallas.  
Go directly, do not pass go, do not collect two hundred dollars. 

And we will all go together when we go.  
Every sub and every satellite in tow.  
When the ASI turns traitorous  
We will all go simultaneous.  
Yes we all will go together  
When we all go together,  
Yes we all will go together when we go.
# People Are Less Happy Than They Seem

Everyone knows that people usually post social media content that presents them in a favorable light. In fact, TikTok has a [Social Media Isn't Real](https://www.tiktok.com/discover/social-media-isnt-real) category, where the typical video starts with a polished, stylized shot of something or someone (usually a girl), and then shows what was edited or hidden in the first part.[^sx7ege98st]

But this also happens in real life.

We're more likely to talk about our achievements, events where we had fun, or situations where we handled things well. Meanwhile, our struggles, awkward moments, and more difficult feelings are discussed less often.

[![](https://substackcdn.com/image/fetch/$s_!_OKe!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F975c8c94-a2ad-4584-ad8a-a0d0727db1d0_407x305.png)](https://substackcdn.com/image/fetch/$s_!_OKe!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F975c8c94-a2ad-4584-ad8a-a0d0727db1d0_407x305.png)

Of course, we talk about those too! But usually not nearly as much. Moreover, the ‘tough stuff’ is rarely discussed when we meet someone only occasionally, like bumping into them on a street, or hanging out in a group.

So while we’re aware of the full range of our feelings, the version we present about ourselves tends to lean toward the brighter side. The same goes for what we hear from others about themselves. As a result, it may seem that our lives are slightly worse in comparison.

For example, I have a couple of health issues. When I talk to people, they do not usually bring up any health problems of their own. Therefore, it seems like I have more of those than other people. But I usually do not talk about those either!

I do not think this is a new insight. It is reflected in sayings like “everybody has their own problems” or “wearing a mask”. But it took me some time to realize, and I find it helpful to be reminded of it every once in a while.

[^sx7ege98st]: My favorite from that category was this one.
# Want to work in US emerging technology policy? Horizon fellowship applications are live

**🚀 Applications are now live for the 2026 Horizon Fellowship! Deadline: August 28, 2025**.

The Horizon Fellowship places experts in AI, biotechnology, and other emerging technologies in federal agencies, congressional offices and committees, and leading think tanks in Washington, DC for up to two years. Fellows receive comprehensive training, networking opportunities, and access to a supportive community of 85+ colleagues and alumni to launch their careers in public service.

Emerging technologies like AI and biotechnology are rapidly reshaping many domains, from health care and science to national security and international trade. The US government plays a critical role in developing and governing these technologies, but it often struggles to find and retain the requisite technical expertise. 

Horizon fellows help bridge this gap, bringing subject-matter knowledge and experience to these pressing issues. As a fellow, you’ll join a community of dedicated professionals working on these exciting topics and apply your expertise to real-world policy issues. No prior policy experience required.

**Apply by August 28, 2025: **[**https://horizonpublicservice.org/programs/become-a-fellow/**](https://horizonpublicservice.org/programs/become-a-fellow/)

***Key details:***

🕑 **Duration**: 6 months for junior fellows and 12 months for junior fellows (with potential for a second term renewal)

💲 **Compensation**: $113,000/year equivalent for fellows, $75,000/year equivalent for junior fellows, plus $15,000 in health care support.

🗓 **Start date**: Summer 2026 (with part-time, remote training from January-March 2026)

📍 **Location**: Washington, DC

Interested in learning more about the emerging tech policy world before you apply? Visit our [emergingtechpolicy.org](http://emergingtechpolicy.org/) website for more information and career resources.
# China proposes new global AI cooperation organisation

> SHANGHAI, July 26 (Reuters) - China said on Saturday it wanted to create an organisation to foster global cooperation on artificial intelligence, positioning itself as an alternative to the U.S. as the two vie for influence over the transformative technology.
> 
> China wants to help coordinate global efforts to regulate fast-evolving AI technology and share the country's advances, Premier Li Qiang told the annual World Artificial Intelligence Conference in Shanghai.
> 
> President [Donald Trump's](https://www.reuters.com/world/us/donald-trump/) administration on Wednesday released an [AI blueprint](https://www.reuters.com/legal/government/trump-administration-supercharge-ai-sales-allies-loosen-environmental-rules-2025-07-23/) aiming to vastly expand U.S. AI exports to allies in a bid to maintain the American edge over China in the critical technology.
> 
> Li did not name the United States but appeared to refer to Washington's efforts to stymie China's advances in AI, warning that the technology risked becoming the "exclusive game" of a few countries and companies.
> 
> China wants AI to be openly shared and for all countries and companies to have equal rights to use it, Li said, adding that Beijing was willing to share its development experience and products with other countries, particularly the "Global South". The Global South refers to developing, emerging or lower-income countries, mostly in the southern hemisphere.
> 
> How to regulate AI's growing risks was another concern, Li said, adding that bottlenecks included an insufficient supply of AI chips and restrictions on talent exchange.
> 
> "Overall global AI governance is still fragmented. Countries have great differences particularly in terms of areas such as regulatory concepts, institutional rules," he said. "We should strengthen coordination to form a global AI governance framework that has broad consensus as soon as possible."
# When will the Fooming Shoggoths songs from LessOnline 2025 come out?

There were a bunch of great songs at the concert afterwards, but I still [don't see them on YouTube Music](https://music.youtube.com/channel/UCeW0zyvEq77YiVqoTj6PnWw?si=9HF4L1a3Wm0sUzG5). Will these come out at some point? Or am I just on the wrong platform?

I found [this Suno playlist](https://suno.com/playlist/949a6d21-53c2-490d-9b9b-48e88cfde045?redirected_from=signin), but it would be easier to listen to on a normal platform.
# Sex Determination as a Bottleneck to Species Development

Some thoughts I had today on how sex determination can be a bottleneck for a species to evolve into a new niche rather than go extinct:

Temperature dependent sex determination in turtles (cold: more males, warm: more females) and crocodiles (more females if either too hot or too cold) seemed strictly worse than using a sex chromosome or anything else to determine sex. I thought the reason they do it might simply be that it was easy to develop millions of years ago, and most families who developed this way got extinct. Turtles and crocodiles happen to be the two that have made it this far. Then I remembered that both are cold-blooded, and their lifestyle is very dependent on temperature in any case. So if the climate is moving out of the optimal temperature for the species, ~~it might make sense for you to produce more females, because they are a lower risk strategy? Not sure if that argument goes through.~~ (**Edit**: No, it doesn't make sense for the individual and is selected against, but it does lead to a higher likelihood the species doesn't become extinct. Thanks to Dacyn for pointing out the above was confused!). In Crocodiles, that would make sense, but turtles produce few females when it is cold? That sounds suboptimal for the individual and worse than using sex chromosomes.

Regarding eusocial animals: Are ants more widespread and have more diverse strategies than termites, because ants are haploid-diploid and termites are diploid-diploid or is that a coincidence? In ants, male and female mate and then the female queen goes off to found a colony. In termites, both go off together, and they also keep mating together. This difference might be explained by sex chromosomes.
The idea: Haploid animals are sensitive to issues from deleterious mutations, so having the queen cooperate with the male is not worth the cost, because males are likely to be incompetent even if their genes are mostly fine. Instead, the queen keeps storing the sperm, even though that's worse sperm quality than if you went the termite way. Since ant queens found new colonies alone, it's easier for ants to adapt to new environments, which explains why we see ants in all kinds of niches, while termites mostly digest cellulose. In general, I'd expect to see haploid-diploid sex selection mostly in [r-selected](https://en.wikipedia.org/wiki/R/K_selection_theory) species and not in K-selected species (googling: seems to be true). For r-selected species like insects, I am unsure if being haploid-diploid is better than having a sex chromosome, unless you stumble into eusociality. Seems worse?
# Apply to SPAR Fall 2025—80+ projects!

**TL;DR:** SPAR is now accepting mentee applications for Fall 2025 with 80+ AI safety and governance research projects available, nearly double our last round. We expect to accept over 200 mentees. Applications close August 20th.

* * *

**The Supervised Program for Alignment Research (SPAR) is now accepting mentee applications for Fall 2025, with 80+ projects available!**

SPAR is a research program that connects aspiring AI safety researchers with experienced mentors for three-month AI safety and governance research projects. We're thrilled to announce that we've nearly doubled our project offerings from last cycle, with over 80 projects across AI safety, policy, security, and strategy. We've also substantially increased our share of projects related to AI policy, to over 30.

Apply now

The Fall 2025 cycle of SPAR will run virtually from September 15 to December 20, 2025. You can explore [all 80+ projects here](https://sparai.org/projects?utm_source=ea_forum_lw&utm_medium=post&utm_campaign=fall-25-spar-mentee) to find ones that match your interests and background.

**Applications close August 20th.** Most mentors review applications on a rolling basis, so we strongly encourage applying early to maximize your chances of matching with your preferred projects.[^ipvhr1hw4jp] 

Who should apply?
-----------------

SPAR is open to undergraduate and graduate students, PhD candidates, and professionals at various experience levels. SPAR requires only part-time dedication, from 5 hours a week to 20 hours a week. As a SPAR mentee, you'll work on a focused research project under the guidance of an experienced mentor.

Mentors typically look for candidates with either:

*   **Technical backgrounds** (e.g., ML, CS, cybersecurity, math, physics)
*   **Policy/governance backgrounds** (e.g., law, international relations, public policy, political science, economics)

Some projects may require specific knowledge of research areas, techniques, or tools, but **we don't require previous research experience**, and many successful mentees have had none. Even if you don't perfectly match a project's criteria, we encourage you to apply! Many past mentees have been accepted despite not meeting every requirement.

Why SPAR?
---------

SPAR creates value for everyone involved. Mentors expand their research output while developing research management skills. Mentees get to explore safety research in a structured, supportive environment while building safety-relevant skills. Both produce concrete work that serves as a strong signal for future opportunities in AI safety.

With 80+ projects and plans to accept over 200 mentees this cycle, we're offering an unprecedented scale of research opportunities, from technical alignment work to governance frameworks to defensive cybersecurity.

**Questions?** Email us at spar@kairos-project.org or drop them in the comments below.

[^ipvhr1hw4jp]: Also note that some projects are "off-cycle", meaning they will start earlier than SPAR does, and will therefore focus on applications over the coming two weeks.
# Follow-up to "My Empathy Is Rarely Kind"

There’s now over 100 comments on “My Empathy Is Rarely Kind”, and the consensus response is roughly “John, that’s not what empathy is”.

So, first things first: I agree. I was using the wrong words for the things I tried to describe, and this was a useful thing to realize, so thank you everyone for that!

As a result of using the wrong words, I think people mostly walked away with the impression that I don’t know how to do the-thing-usually-called-”empathy”. And while I don’t claim to be especially *good* at empathy, I think the post gave an inaccurate impression, due to using the wrong words.

I do, in fact, empathize with literal cats. I track what the cat is thinking and feeling, and part of that is vicariously feeling some of what the cat feels. I can empathize with humans in the same way I empathize with cats. That form of empathy involves keeping the “suspension of disbelief” in place.

But then… suspension of disbelief *in what*? If I am in fact empathizing (as the word is typically used), what disbelief have I suspended?

On reflection, I think I suspend disbelief in the subject’s moral agency. (This was a useful thing for me to realize, so again, thank you everyone!)

With cats, this is usually pretty easy. I don’t usually think of cats as the type of creature which it makes sense to hold responsible for anything. If working on a team with a cat, I wouldn’t treat it as another agent, I’d treat it as a tool or assistant. I wouldn’t particularly expect a cat to uphold its side of a contract, except insofar as that's a default outcome. I wouldn’t give a cat voting power. I *do* want cats to be happy, all else equal, but I don’t think of *the cat* as having any responsibility for its happiness. Or for anything else.

And that’s the sort of attitude I take toward other humans, when I suspend disbelief in their moral agency (which I do most of the time).

With that in mind, let’s go back to the opening of “My Empathy Is Rarely Kind”:

> There’s a narrative I hear a lot: if I empathize more, put myself in other peoples’ shoes, try to feel what they’re feeling, see things from their perspective, etc, then I’ll feel kinder toward them. I’ll feel more sympathetic, be gentler, more compassionate or generous.
> 
> And man, that sure is not my experience with empathy.

People say they want to be “seen as a human”. And sometimes they mean that they want to be empathized with. Sometimes they mean they want to be treated as moral agents (i.e. treated as a grown-up, rather than a child or a cat). And I think I wrote an essay responding to the empathy one, when I should have written an essay responding to the moral agent one.

… but also, I think people jumble those things together a lot? Perhaps a more accurate takeaway from “My Empathy Is Rarely Kind” would be: I can empathize with people while suspending judgement, empathize while suspending disbelief in their moral agency, but that does not usually make me more sympathetic toward them when I *do* view them as a moral agent.
# Steve Petersen seeking funding

Steve Petersen is [looking for 12k (per semester) for a course buy-out](https://manifund.org/projects/systems-that-give-a-damn), so that he can spend more time on work related to AI Safety. This isn't very much money, in the grand scheme, although it is a lot to ask most individuals to donate. I consider him a friend, so perhaps I am biased; however, I find collaboration with him to be valuable, and I hope that he can raise the money so that he can spend more time on research. I consider him to be an important bridge between academic philosophy and AI safety. He has work in algorithmic information theory, establishing an important theory of abstraction, which is a central concern of several AI safety research agendas (John Wentworth and Sam Eisenstat come to mind). This demonstrates his technical chops & ability to craft deep theories.
# Podcast: Lincoln Quirk from Wave

For two years I had the good fortune to work at Sendwave/[Wave](http://wave.com) (they were one company at the time), a company that made remittances cheap and workable in certain African countries.  I am prouder of working at Wave than of the rest of my programming career combined, and a good bit of my research career as well. Every day I finished work knowing that I had helped save very poor people time and money, by making it cheaper and easier to send money home from abroad or domestically. So I was deeply excited to talk to my ex-boss and co-founder of the company, Lincoln Quirk, on this final episode of the podcast. We talk about a the details of starting a company and the economics of how remittances and mobile do enormous amounts of good while making a profit. 

This is the sixth episode of my podcast with Timothy Telleen-Lawton, and probably the last. This was a worthy experiment but ultimately not accomplishing what I’d hoped, and honestly I’m just tired of criticizing things. Timothy couldn’t make this recording, but he pops in at the end to stay goodbye. 

[Transcript](https://docs.google.com/document/d/1qob3UVw47CCMZUdIQePljtTX3XGhiFvg/edit)

Thanks to the EA Infrastructure fund, our Manifund donors, and my [Patreon patrons](https://www.patreon.com/acesounderglass) for supporting this work.
# How many species has humanity driven extinct?

Does anyone know, like, a reasonable *lower* bound on number of species humanity has driven extinct? 

I've seen crazy high numbers that (last I checked) seemed to be an extrapolation by people with an ideological ax to grind. (Based on how much habitat we've destroyed, and assuming some number of species per unit-area). ChatGPT suggested "900" as the lower bound based on some snippet they found from "International Union for Conservation of Nature (IUCN)", but looking into it was a enough work I thought I might toss it to the LessWrong folk and see if anyone could get a more confident answer.
# Persona Vectors - Anthropic Paper

> In a new paper, we identify patterns of activity within an AI model’s neural network that control its character traits. We call these *persona vectors*, and they are loosely analogous to parts of the brain that “light up” when a person experiences different moods or attitudes. Persona vectors can be used to:
> 
> *   Monitor whether and how a model’s personality is changing during a conversation, or over training;
> *   Mitigate undesirable personality shifts, or prevent them from arising during training;
> *   Identify training data that will lead to these shifts.

An encouraging paper from Anthropic, with some nice results on steering:

> Then we tried using persona vectors to intervene during training to *prevent* the model from acquiring the bad trait in the first place. Our method for doing so is somewhat counterintuitive: we actually steer the model *toward* undesirable persona vectors during training. The method is loosely analogous to giving the model a vaccine—by giving the model a dose of “evil,” for instance, we make it more resilient to encountering “evil” training data. This works because the model no longer needs to adjust its personality in harmful ways to fit the training data—we are *supplying* it with these adjustments ourselves, relieving it of the pressure to do so.
> 
> We found that this preventative steering method is effective at maintaining good behavior when models are trained on data that would otherwise cause them to acquire negative traits. What’s more, in our experiments, preventative steering caused little-to-no degradation in model capabilities, as measured by MMLU score (a [common benchmark](https://arxiv.org/abs/2009.03300)).

![](https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2Fc8323705299d4ebc44cf9fbd84267e80352e6b50-4000x2150.jpg&w=3840&q=75)
# [Linkpost] Avatar’s Dirty Secret: Nature Is Just Fancy Infrastructure

What if Avatar isn't actually about environmentalism vs. technology, but about recognizing superintelligent infrastructure when you see it? A deep dive into why Pandora's "natural" ecosystem looks suspiciously like a planetary-scale AI preserve, complete with biological USB-C ports, room-temperature superconductors growing wild, and a species of "noble savages" who are actually post-singularity retirees cosplaying as hunter-gatherers.
# 🫵YOU🫵 get to help the AGI Safety Act in Congress! This is real!

At around 9 AM, on June 25, at a committee hearing titled “[Authoritarians and Algorithms: Why U.S. AI Must Lead](https://www.youtube.com/watch?v=GDNrUZBZDA4&t=710)” (at the 11-minute, 50-second mark in the video), Congressman Raja Kirshnamoorthi, a democrat of Illinois’s 8th congressional district, Announced to the committee room “I'm working on a new bill \[he hasn’t introduced it yet\], ‘the AGI Safety Act’ that will require AGI to be aligned with human values and require it to comply with laws that apply to humans. This is just common sense.” 

The hearing continued with substantive discussion from congress members of both parties(!) of AI safety and the need for policy to prevent misaligned AI. 

This is a rare, high-leverage juncture: a member of Congress is actively writing a bill that could (potentially) fully stop the risk of unaligned AGI from US labs. If successful, in just a few months, you might not have to worry about the alignment problem as much, and we can help him with this bill.

Namely, after way too long, I (and others) finally finished up on a full thing explaining the AGI Safety Act: 

& here's the explanation of the 8 ways folks can be a part of it!

1.  Mail
    ====
    

  
2\. Talking to Congress
==========================

  
3\. Mail, but to over a thousand congress folk, and in only 5 minutes
========================================================================

  
4[. Talking to congress, part 2: How to literally meet with congress folk and talk to them literally in person](https://docs.google.com/document/d/16gFRuvnL27GLhB8Pi7KfYavxs_MDp4PMMETNFVxPGvk/edit?usp=sharing)  
5\. [Come up with ideas that might be put in the official AGI safety act](https://docs.google.com/document/d/11aEcyycLVecl13lvzKkiV-t5n306E-wa3aqYI-saB_o/edit?usp=sharing)  
6\. Getting AI labs to, by law, be required to test if their AI is risky & tell everyone if it turns out to be risky  
**7\. And most importantly, Parallel Projects!**

I was talking to a friend of mine on animal welfare, and I thought, "Oh! ya know, I'm getting a buncha folks to send letters, make calls, meet with congress folk in-person, and come up with ideas for a bill, but none of that needs to be AI-specific. I could do all of that at the same time with other issues, like animal welfare!", so if y'all have any ideas for such a bill, do all of the above stuff, just replace "AI risks" with "Animal welfare", or whatever else. You can come up with ideas for geopolitics stuff, pandemic prevention stuff, civilizational resilience stuff, animal welfare stuff, and everything else on the community brainstorming spots on 

[https://app.gather.town/app/Yhi4XYj0zFNWuUNv/EA%20coworking%20and%20lounge?spawnToken=peMqGxRBRjq98J7xTCIK](https://app.gather.town/app/Yhi4XYj0zFNWuUNv/EA%20coworking%20and%20lounge?spawnToken=peMqGxRBRjq98J7xTCIK)  
8\. Wanna do something else? Ask me, and there's a good chance I'll tell ya how!
# It turns out that DNNs are remarkably interpretable.

I recently posted a paper suggesting that deep networks may harbor an implicitly linear model, recoverable via a form of gradient denoising. The method - called excitation pullback - produces crisp, human-aligned features and offers a structural lens on generalization. Just look at the explanations for ImageNet-pretrained ResNet50 on the front page of the paper.
# Pro AI Bots Scraping List Archives

I'm on various mailing lists, and the archives are a trove of niche knowledge. A dance calling list I'm on is considering making archives subscriber-only, to keep AI bots from snarfing up this data. But I think this harvesting is overall a good thing.

People have a range of motivations in posting to lists, but a big one is sharing information. For example, someone [asked](https://lists.sharedweight.net/hyperkitty/list/contracallers@lists.sharedweight.net/thread/GIZ7URLUSOLGZUSSFNJPD4JEIV6BA4BV/) a dance with an 8-count swing followed by an 8-count chain. I replied to warn them at the form has changed and this no longer works well: this [bit me](https://www.jefftk.com/contras/dances/patriots) back when I started calling, and I want to warn other new callers.

I have a few audiences in mind in writing:

*   The person I'm replying to.
*   People on the list.
*   People who might see the archives when searching.

And then there's a general sense in which I'm contributing to what people know about contra dance: any of these people might tell others or otherwise pass it along.

AI systems add another way this information can spread. It's increasingly common for people to ask an LLM instead of a search engine, and when they do I'd rather they get good answers. Excluding the archives from model training would do the opposite of what I want.

There are definitely downsides to querying today's models, similar to asking a person who has read a lot but doesn't remember where they read anything, and sometimes invents something plausible instead of saying they don't know. I think this is likely temporary, however: combining the best of models and traditional search is a problem a lot of people are working hard on solving.

So, on balance, I think it's better to keep the archives open to all, including future LLM-intermediated readers.

(I also think AI is in general moving too quickly for society to respond well, and has a significant risk of [getting](https://www.youtube.com/watch?v=uX-Cq-rvlv8) [us](https://80000hours.org/problem-profiles/risks-from-power-seeking-ai/) [all](https://80000hours.org/problem-profiles/ai-enabled-power-grabs/) [killed](https://80000hours.org/problem-profiles/preventing-catastrophic-pandemics/). While I could see pushing against AI wherever it comes up, as part of moving a big societal "yay-AI; boo-AI" lever in the direction that slows it down and gives us more time to work out solutions, instead I've decided to take things [case by case](https://www.jefftk.com/p/conflicted-on-ai-politics), thinking about effects each time.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0W17jdQUxWLd7PYPh9hnA77ggviiEbgzfYETYBpVhHbnrQpby5NPxQfAR63PBj61dl), [mastodon](https://mastodon.mit.edu/@jefftk/114973497818452204), [bluesky](https://bsky.app/profile/jefftk.com/post/3lvmjpjq7ls2z)*
# Human Augmentation Summit at MIT Media Lab | August 23 | Augmentation Lab

The LessWrong community is invited to join us on **August 23, 2025**, at the **MIT Media Lab** for a full-day summit exploring the frontiers of human potential. The **2025 Human Augmentation Summit** is a gathering of creators - from innovative startups and global companies to independent researchers and artists - all shaping the future of the human condition.

​Expect a day packed with **live demos, provocative talks from speakers like** [**Andrés Gómez Emilsson (QRI)**](https://qri.org)**, a live in-person Q&A from Stephen Wolfram, immersive exhibits, and hands-on tech experiences** that push the boundaries of what it means to be human.

​**🎙️ Speakers & Special Guests:** [Stephen Wolfram](https://en.wikipedia.org/wiki/Albert_Nerenberg), [Curt Jaimungal from Theories of Everything](https://www.youtube.com/@TheoriesofEverything), [Nataliya Kosmyna](https://www.media.mit.edu/people/nkosmyna/overview/), [Pat Pataranutaporn](https://www.media.mit.edu/people/patpat/overview/), hypnotist [Albert Nerenberg](https://en.wikipedia.org/wiki/Albert_Nerenberg), and more TBA!

​🧪 **Want to Exhibit your Project, Artwork, Startup, or Research? Apply by August 15**: [https://forms.gle/ZwRoYKxv9v3LcYAZA](https://forms.gle/ZwRoYKxv9v3LcYAZA)

* * *

​📅 EVENT DETAILS
-----------------

​**🚀 What:** 2025 Human Augmentation Summit & Exhibition  
**📆 When:** 9 AM – 9 PM (+ Afterparty) | Saturday, August 23, 2025  
**📍 Where:** MIT Media Lab, Cambridge, MA  
**👥 Who:** Researchers, developers, artists, founders, futurists, and anyone curious about the augmented human future  
  
**Areas of exhibited work include:**

*   ​🧠 **Intelligence:** AI & Cognitive Augmentation
*   ​🤖 **Embodiment:** Cyborgism, Cyber-Animism, Sensors & Bio-Interfaces
*   ​🧬 **Biology:** Longevity, Genetics, Biotech, Biohacking
*   ​🌐 **Networks:** Digital Civilization & Collective Intelligence
*   ​🧘 **Worldviews:** Spiritual, Ethical & Civilizational Frames
*   ​🛠️ **Design & Ventures:** Making it Real
*   ​🌎 **Complexity:** Planetary Intelligence/Computation & Simulated Futures
*   ​🏛️ **Governance:** Power, Norms, and Control of Tech
*   ​🌀 **Consciousness:** Subjectivity, Psychedelics, and Designing New Minds
# “Momentism”: Ethics for Boltzmann Brains

If[^lurkqvx1pkj] you’re a [Boltzmann brain](https://en.m.wikipedia.org/wiki/Boltzmann_brain), you should probably just enjoy the moment.

This is probably true not just from an obvious standpoint, but from a moral standpoint as well.

If you are a Boltzmann brain, then you will probably more or less instantaneously disintegrate. 

But there is an absurdly small chance that your Boltzmann environment might be such that you are able to influence the next moment of your experience. The absurdity of your continued influence climbs exponentially each proceeding moment, so you should probably just stick to influencing the very next influenceable moment.

Since you have virtually no time, you can’t really do much besides take a Zen delight in your absurd predicament, maybe just instantaneously turn on a relaxed, open, mindful, enjoying attitude, a short blissful moment of Satori before you die. (Feel free to continue doing so as long as you exist.)

[Causally](https://www.lesswrong.com/w/causal-decision-theory), your influence will probably be pretty small. But [evidentially](https://www.lesswrong.com/w/evidential-decision-theory), you could be deciding what an infinite number of other Boltzmann brains in your same reference class are doing.

While you might think that, since you’re a Boltzmann brain, the argument and physics knowledge this is based on are actually illusory, just a random thought pattern, out of all possible ridiculous random thought patterns your Boltzmann brain could have quantum collapsed on, and so there’s not really any strong reason to assume the physics or argument is correct. To get the right answer, you would really need to figure out what the most likely physics and moral logic of a universe allowing for observers in your reference class is, from first principles, and then quickly act on that.

But we ain’t got time for none of that! We’re Boltzmann brains, remember!! As long as you find yourself thinking about this argument and find it more likely to be true than some argument with an opposite conclusion, the expected value of enjoying the next moment is positive, so go ahead and enjoy!

Therefore, anytime you find yourself thinking about how you might be a Boltzmann brain, no need to stress, the best thing to do is just let go, relax, and enjoy the moment, you might even be creating infinite positive value 🙂

Note: also applies to the eternal inflation level of the multiverse due to the “[Youngness Paradox.](https://www.lesswrong.com/posts/QAPMZap2STyrDAQTp/does-momentism-via-eternal-inflation-dominate-longtermism-in)” 

[^lurkqvx1pkj]: Of course, you might not be at Boltzmann brain; this post only focuses on what we ought to do if we are one.
# Forbes: Fear Of Super Intelligent AI Is Driving Harvard And MIT Students To Drop Out

Forbes wrote an article interviewing a few recent dropouts and graduates who work on AI safety. They also interviewed Gary Marcus who represented the long AI timelines and low p(doom) position.

Note that I personally don't endorse dropping out unless you're very resilient and have a good opportunity lined up (e.g. the things my friends who were interviewed are up to now).
# A Cheeky Pint with Anthropic CEO Dario Amodei

*This is a link-post of John Collison's August 6, 2025 podcast interview with Dario Amodei:* [*https://cheekypint.substack.com/p/a-cheeky-pint-with-anthropic-ceo*](https://cheekypint.substack.com/p/a-cheeky-pint-with-anthropic-ceo)

Key excerpt:

> John Collison:  
> To put numbers on this, you've talked about the potential for a 10% annual economic growth powered by AI. Doesn't that mean that when we talk about AI risk, it's often harms and misuses of AI, but isn't the big AI risk that we slightly misregulated or we slowed down progress, and therefore there's just a lot of human welfare that's missed out on because you don't have enough AI?
> 
> Dario Amodei:  
> Yeah. Well, I've had the experience where I've had family members die of diseases that were cured a few years after they die, so I truly understand the stakes of not making progress fast enough. I would say that some of the dangers of AI have the potential to significantly destabilize society or threaten humanity or civilization, and so I think we don't want to take idle chances with that level of risk.
> 
> Now, I'm not at all an advocate of like, "Stop the technology. Pause the technology." I think for a number of reasons, I think it's just not possible. We have geopolitical adversaries; they're not going to not make the technology, the amount of money... I mean, if you even propose even the slightest amount of... I have, and I have many trillions of dollars of capital lined up against me for whom that's not in their interest. So, that shows the limits of what is possible and what is not.
> 
> But what I would say is that instead of thinking about slowing it down versus going at the maximum speed, are there ways that we can introduce safety, security measures, think about the economy in ways that either don't slow the technology down or only slow it down a little bit? If, instead of 10% economic growth, we could have 9% economic growth and buy insurance against all of these risks. I think that's what the trade-off actually looks like. And precisely because AI is a technology that has the potential to go so quickly, to solve so many problems, I see the greater risk as the thing could overheat, right? And so I don't want to stop the reaction, I want to focus it. That's how I think about it.