Harley-ml/lesswrong huggingface dataset:
# [GDPval] Models Could Automate the U.S. Economy by 2027

[OpenAI's new GDPval benchmark](https://openai.com/index/gdpval/) measures AI capabilities on real-world tasks from the sectors contributing most to the U.S. GDP. Given a task on GDPval, a human industry expert compares the model deliverable to a deliverable by industry experts and chooses the preferred one. Model performances are thus reported as win rates.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/74d3ee6a1d53031a709731165be880c467d122cd84da0941.png)

Example GDPval tasks from full set. Figure from the paper.

In the spirit of [METR's task horizon study](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/), here is a plot on the model performances from GDPval against model release dates.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/af88ec7d639aa858c0c169d865f1443e92764e4a21b47f03.png)

The plot above uses a logit scale. Other scales can be found in the appendix below. Since only the best models concern us, the regression only uses models that outperforms previous models, labelled as the green "Frontier models" in the plot. There are only three datapoints, so this extrapolation is highly uncertain even if the $R^2$ is high.

Under this regression, a hypothetical model release today would have already outperformed an industry expert in an average task in GDPval. By mid-2026, human experts would prefer AI deliverables 75% of the time. By late-2027, human experts would prefer AI deliverables 95% of the time.

On job loss: GDPval contains tasks that are *currently* economically-useful. The usual argument holds: there could be novel tasks in the future that are economically-useful and that models do not perform well in.

The more interesting point is that most of the current U.S. GDP could be automated as early as 2027.

*Thanks to Nikola Jurkovic for helpful comments.*

Appendix
========

Repo Link
---------

[Here.](https://github.com/jeqcho/gdpval-plots)

Next Steps
----------

Fill in the data gap between GPT-4o and o3-high. This will involve running Sonnets 3.5, 3.7, 4, and possibly o1, on GDPval. Evaluate Sonnet 4.5, which is forecasted to score above 50%.

Extra Plots
-----------

Extrapolated dates

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/84867eee4fff011d72bb36c0c62ca1fb47dd417ed0cfff89.png)

Raw win rate on the y-axis

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/585875197793d742e4694861df165a80aa3a1a930b7a83d4.png)

Log win rate on the y-axis

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/bfeff05a8d7b3e29ea83aeb4cd5f3c797484f817a45059b6.png)

Odds on the y-axis

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/851aaf2993351c1f811726d1728feec03a1ef154f27fb998.png)

Table of Extrapolated Dates
---------------------------

The table [here](https://github.com/jeqcho/gdpval-plots/blob/main/data/model_performance_projections.csv) includes results from regressing on all models reported in the paper, and not just the frontier ones.
# The famous survivorship bias image is a "loose reconstruction" of  methods used on a hypothetical dataset

[File:Survivorship-bias.svg - Wikimedia Commons](https://commons.wikimedia.org/wiki/File:Survivorship-bias.svg)

![File:Survivorship-bias.svg](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b2/Survivorship-bias.svg/960px-Survivorship-bias.svg.png?20221011173327)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/73581e7efce42abd07da347d43fc7d34caf9f6b5ebc19e95.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ecbbce1257dd783451282a00f7b7e75a52871bdd0ed57eb4.png)

A later reprinting of the "unillustrated report" referenced:

[A\_Reprint\_Plane_Vulnerability.pdf](https://www.plover.com/~mjd/pictures/blog/misc/survivorship-bias/A_Reprint_Plane_Vulnerability.pdf)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/db2f6a624570735bc1e6e6e23ca27af4c0df0205061bafe6.png)

Even a 20mm autocannon round only has a ~50% chance of destroying a plane on an engine hit, as opposed to the ~100% assumption used for the illustration (I think the flak round statistics were per-fragment-hit and not per-shell)! Weird how few people were suspicious about a 100% loss rate for engine hits to a two-engine plane. 

Speaking of which, there is no source for the example data, which is noted as "hypothetical" in the report. The hypothetical data set was for a 1000-plane raid on a single objective, which implies a strategic bombing mission context, and thus that the planes were 4-engine strategic bombers like the B-17 instead of the 2-engine bomber in the illustration (this is a nitpick, but I thought it was neat). There is also no exact hit location data, with hits being aggregated by category.

It's still a nice-looking illustrations though, and I don't think the contributor did anything wrong here. It's just a reminder to look more closely at anything that looks too good.

Inspired by this tweet:

[Liron Shapira on X: "A plot showing zero planes returned after being shot in various large regions doesn’t prove which areas are best to reinforce since we don’t know the distribution of shots, but does prove that someone made a fake exaggeration of how much survivorship bias dominates other effects. https://t.co/7UGoyzv2zn" / X](https://x.com/liron/status/1972981807219101935)

Fighter pilots do in fact aim for specific parts of a plane with MG and autocannons, while flak rounds are shot from the ground.
# Lectures on statistical learning theory for alignment researchers

This is a recorded series of lectures and homework assignments discussions on the topic of statistical learning theory, co-organized by Gergely Szucs ([@Yegreg](https://www.lesswrong.com/users/yegreg?mention=user)) and [@Alex Flint](https://www.lesswrong.com/users/alexflint?mention=user). The homework assignments are available [here](https://drive.google.com/drive/folders/1HrHnX3XwMy1g1Yjs6EwHBaG02FY5bdD8?usp=sharing), and lecture notes are available [here](https://drive.google.com/drive/u/2/folders/1qx5pftRuNhejfCcDoTxaUKS0AaSwSHyZ).

The lectures are intended for those who want to do research on the [learning-theoretic agenda](https://www.lesswrong.com/posts/ZwshvqiqCvXPsZEct/the-learning-theoretic-agenda-status-2023) and need to catch-up on the necessary background knowledge. As complementary resources, consider looking into my [MATS lectures](https://www.lesswrong.com/posts/NWKk2eQwfuGzRXusJ/video-lectures-on-the-learning-theoretic-agenda), the [LTA reading list](https://www.lesswrong.com/posts/fsGEyCYhqs7AWwdCe/learning-theoretic-agenda-reading-list) and the (not quite finished) [sequence on infra-Bayesianism](https://www.lesswrong.com/posts/wzCtwYtojMabyEg2L/what-is-inadequate-about-bayesianism-for-ai-alignment) by [@Brittany Gelb](https://www.lesswrong.com/users/brittany-gelb?mention=user).

If you feel ready to start with actual research, consider applying to my track in the [PIBBSS fellowship](https://pibbss.ai/fellowship/).

**Privacy note:** *According to what I know, everyone who appear in the recording consented to the recording becoming available online. However, if there was some mistake, s.t. you are in the recording but don't want to be there, DM me and I will take it down ASAP.*

*H/t to my beloved spouse* [*@Marcus Ogren*](https://www.lesswrong.com/users/marcus-ogren?mention=user) *for placing the lectures on YouTube.*
# Uncertain Updates: September 2025

Does the calendar say October? Well, like the good comptroller I am, I waited for the period to end before reviewing the books.

I don’t normally do that? You didn’t know I was a comptroller? Okay, fine, look, I could have posted this last week, but I had two good reasons not to.

First, [last week’s Autofac Era essay](https://uncertainupdates.substack.com/p/the-autofac-era) was ready to go. It came together quickly after sitting on the ideas for a couple months, and I was excited to get it out.

Second, I was very close to finishing revisions on Chapter 4 of [the book](https://www.fundamentaluncertainty.com/), and I’m happy to say they’re now complete! They took a bit longer than expected, both because the original version of the chapter had more factual mistakes than I had realized that needed fixing, and because I had to completely rewrite one of the sections based on the feedback that, to be correct, I’d really need to talk about the Löbian obstacle instead of issues with Bayesian self trust.

That meant I had to understand the Löbian obstacle well enough to write about it, which ate up plenty of time because I didn’t understand Löb’s theorem as well as I needed to. Much thanks to Claude for being smart enough to be able to read the original papers and keep pointing out my mistakes as I tried to explain the ideas by just continually checking if my words literally made sense compared to what was in the papers.

As ever, I’m hopeful that revisions on future chapters go more quickly. We shall see.

Other Happenings
================

I’ll be at [the Curve conference](https://thecurve.goldengateinstitute.org/) this weekend in Berkeley. If you see me there, say hi!

I’m giving a talk about the origins of the post-rationalists later this month. Depending on how it goes, I might write something up here, too.

I have more posts about AI in the pipeline. Make sure to subscribe if you want to get those as soon as they are released.

[Subscribe now](https://uncertainupdates.substack.com/subscribe?)

And if you really want to know “What’s happening?”, you should [follow me](https://x.com/gworley3) on X, the everything app, where I sometimes post interesting things that go under appreciated.

[

![](https://substackcdn.com/image/fetch/$s_!_Uka!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F271812b4-19fe-4a4b-9f4e-a84ea67a8844_1214x518.png)



](https://substackcdn.com/image/fetch/$s_!_Uka!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F271812b4-19fe-4a4b-9f4e-a84ea67a8844_1214x518.png)
# But what kind of stuff can you just do?

A phrase that has been popping up lately is: "You can just do stuff" Every time I read it a part of me goes "Yeah! You can just *do* *stuff*!". And then I forget about it again. 

Part of this is that "doing" and "stuff" are pretty vague. So, why not *just do stuff*, like make a list of stuff that you can just do? 

This is for own inspiration as much as for the people reading it. And I'm very happy to hear further suggestions!

You can just:

*   Tell your dad you love him
*   Call your friends and suggest to meet up
*   Go for a walk
*   Tell the barista they look cute
*   Quit your job
*   Ask for help
*   Read a book
*   Hand out cookies on the street
*   Ask *that person* for their phone number
*   Paint your bedroom wall green
*   Ask the lady in the park what her wisest lesson in life is
*   Apologize
*   Dance in your living room
*   Hold that hug until the *other* person lets go
*   Sign up for a singing class
*   Call in sick
*   Switch off your phone
*   Bake bread
*   Invite people over for dinner
*   Give a compliment to a stranger
*   Listen to completely new music
*   Take a trip to the next city over
*   Share a secret you've been holding onto
*   Go swimming
*   Try a new flavor of ice-cream
*   Show up at someone's doorstep unannounced
# </rant> </uncharitable> </psychologizing>

There's a bit of an implicit norm that has accumulated over the past few years, but inconsistently, and AFAICT no one has publicly argued for it. 

Sometimes, when arguing online, you will notice yourself going into a mode that is more like a rant than like careful arguing. Or, you know is not really passing your interlocutor's Intellectual Turing Test. Or, you know is psychologizing someone, and there's decent odds you're wrong (perhaps only in a subtle way that isn't core to *your* point but is going to feel violating to the person you're psychologizing).

Sometimes, the correct action is *"just don't do those things."*

Sometimes, IMO, the correct action is "do those things, but carefully."

Sometimes, IMO, the theoretically ideal correct action all-else-equal is *"do those things carefully"*, but you're busy and it's realistically a choice between *"do them uncarefully"* or *"don't do it at all",* and it's actually better to do it uncarefully than not-at-all.

In such case, I think it's noticeably better to include a short disclaimer like "I recognize I am being uncharitable right now, and maybe am wrong about this, and am sorry."

The "I may be wrong" and "I'm sorry" parts are both pretty useful (especially if you don't have established trust with the interlocutor)

The first, because it's true (it's generally a good epistemic move to have at least two real hypotheses, and if you don't, IMO you should regard your conclusion as sus). The second, because uncareful rants and uncareful psychologizing and uncharitable paraphrases do have a decent chance of damaging the epistemic commons (not merely hurting people's feelings). 

["Rudeness"](https://www.lesswrong.com/posts/kJBKJJsurYiHadfQp/rudeness-a-useful-coordination-mechanic) is a useful concept. It's a way of agreeing "here are some actions that people will realistically do sometimes, sometimes for good reasons, but, it would be bad if people did them willy-nilly." It sets incentives better IMO if the expectation is "if you do the rude thing, you do take some reputational hit, and if you do it too much without somehow making up for it, longterm consequences will accumulate."

(Title of this post should probably actually be something like: "</costlyButOwnedRantThatIAcceptResponsibilityFor>", to bake the responsibility more into the meme. But, that title sucks and I haven't thought of a better one.

I think it's a reasonable counterargument to some previous-debates-about-norms, that "adding friction to people thinking out loud is often way, way, worse than you might naively expect." But, I think adding a quick disclaimer is just not that hard. 

I think this is a useful thing to do even if there's not a coordinated social norm about it. But, I do separately think it'd be good if it were considered "extra rude" to do uncareful rants, and uncharitable psychologizing, without briefly noting them as such.

*See:* [*Generalized Hangriness: A Standard Rationalist Stance Toward Emotions*](https://www.lesswrong.com/posts/naAeSkQur8ueCAAfY/generalized-hangriness-a-standard-rationalist-stance-toward).
# No, That's Not What the Flight Costs

It turns out that airlines largely do not make their money from selling tickets. Instead, airlines are primarily in the business of selling credit card rewards.

Rewards programs are intangible assets, so the financial statements of major airlines do not generally give valuations for them. However, there are a couple public sources for how valuable these rewards programs are:

1.  During COVID, many US major airlines applied for loans under the CARES Act. Some of them used their rewards programs as collateral for these loans, and both United and American publicly disclosed their reward programs' appraised values for this purpose.
2.  On Point Loyalty, a consultancy firm focused on loyalty programs, published appraisals of most airlines' rewards programs in 2023 and in 2020. Their methodology is proprietary and it is unclear what data their numbers are based on, but the estimates agree with the CARES Act disclosures.

Major US airlines also trade publicly, so we can see what portion of the airlines' value is due to their reward programs. Here's a chart of the EOY 2023 enterprise value of the top six US airlines, their On Point reward program valuation, and the difference between those two amounts (figures in millions of USD):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/209e97e44713b98a7b0a0b60e20b4a6f24b4ec83c8334a28.png)

It turns out that the rewards programs constitute the majority of the value of every major US airline. The entire business of selling flight tickets is, to an extent, a *loss leader* for the airlines' real business of selling miles.

It's not just On Point's valuations being wrong either, the United and American CARES Act filings tell the same story:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/32419840ebd9db908f31143df0071fc66286aa303a4a872a.png)

In fact, the only reason airlines can afford to operate the way they do is because American consumers subsidize the industry every time they make a credit card purchase. About 15% of US interchange (the fee that businesses pay in order to accept card payments) is ultimately paid to airlines in return for loyalty program benefits.

You pay for your flights with every purchase you make, not just when you buy the tickets.

Note: a previous version of this post incorrectly used market capitalization rather than [enterprise value](https://en.wikipedia.org/wiki/Enterprise_value) as the total value of the airlines' business, failing to take debt into account. Thank you to [@Mis-Understandings](https://www.lesswrong.com/users/mis-understandings?mention=user) for pointing this out.
# Open Thread Autumn 2025

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/w/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# We’ve automated x-risk-pilling people

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8a9fcc466c08735fd48caaa54e8c630b839102c16ce3b979.png)

After a message, the chatbot asks how helpful it was, on the scale from "Not at all" to "Completely changed my mind". n=24, median=5, average=4.46.  
One of the zeros is someone who added a comment that they had already been convinced.

Something that I've been doing for a while with random normal people (from Uber drivers to MP staffers) is being very attentive to the diff I need to communicate to them on the danger that AI would kill everyone: usually their questions show what they're curious about and what information they're missing; you can then dump what they're missing in a way they'll find intuitive.

We've made a lot of progress automating this. A chatbot we've created makes arguments that are more valid and convincing than you expect from the current systems.

We've crafted the context to make the chatbot grok, as much as possible, the generators for why the problem is hard. I think the result is pretty good. Around a third of the bot's responses are basically perfect.

We encourage you to go try it yourself: [https://whycare.aisgf.us](https://whycare.aisgf.us). Have a counterargument for why AI is not likely to kill everyone that a normal person is likely to hold? Ask it!

If you know normal people who have counterarguments, try giving them the chatbot and see how they'll interact/whether it helps.

*We're looking for volunteers, especially those who can help with design, for ideas for a good domain name, and* [*for funding*](https://aisgf.us/fundraising)*.*
# Do One New Thing A Day To Solve Your Problems

People don't explore enough. They rely on cached thoughts and actions to get through their day. Unfortunately, this doesn't lead to them making progress on their problems. The solution is simple. Just do one new thing a day to solve one of your problems. 

Intellectually, I've always known that annoying, persistent problems often require just 5 seconds of actual thought. But seeing a number of annoying problems that made my life worse, some even major ones, just yield to the repeated application of a brief burst of thought each day still surprised me. 

For example, I had a wobbly chair. It was wobbling more as time went on, and I worried it would break. Eventually, I decided to try actually solving the issue. 1 minute and 10 turns of an allen key later, it was fixed. 

Another example: I have a short attention span. I kept wasting all my time constantly refreshing pages in the hopes of the tiniest dopamine hit. This sucked. I couldn't read anything longer than a page without getting distracted, I couldn't practice skills for more than a moment, or write anything longer than a tweet. So I started to do one new thing a day to solve this. 

First, I decided to block out two hours in the morning free from electronics. This sorta worked. Then, I tried going to the library after the two hours were up. This sorta worked, too, and I used the computer sensibly there. But then the swarm of attention-hijacking processes out to get you which infest the internet got their hooks into me again, and I regressed. So I tried blocking out my time to use electronics in two hour blocks where I alternately can/can't use electronics. This kinda worked, but then I got cocky and thought myself immune to tight feedback loops and regressed. So I banned myself from electronics one day a week. And that sorta worked too. 

All of these things synergized to effectively [pause my feedback loops,](http://bewelltuned.com/pause_your_feedback_loops) and were independent enough that even when my attention is hijacked again and I start doomscrolling, I can recover within a week or so instead of the months it might have taken before. And now I have enough time to be bored, I find myself doing things like talking to my family more, going to new libraries to read any book that catches my fancy, pick up my notepad to write down ideas when they catch my attention, set up co-working sessions with my team. 

None of the major issues in my life are fully solved. Or even mostly solved. Wait, scratch that, the biggest one is basically solved, partially due to trying new things out to solve my problems. But even for the rest, I've made so much more progress in the last year than before because I made the serendipitous choice to spend just one month trying new things every day to solve my problems. 

I can only imagine what doing that for a year would look like for me.  

What would it look like for you?
# What I've Learnt About How to Sleep

1.  Sleeping too much can tire me out as much as sleeping too little.
2.  Waking up in a sleep cycle, about 1 and a 1/2 hours, is tiring.
3.  Switching off digital devices an hour before bed helps me sleep.
4.  My body has an internal clock and I can just decide when to wake up by setting an intention to do so before you go to bed.
5.  The costs of missed sleep often show up not the next day, but the day after that.
6.  Earplugs work, but I have to roll them thin before inserting them, otherwise they won't seal themselves in your ear.
7.  Blocking out light works. Pro-tip: use tin-foil. It's good for more than just blocking out telepaths.
8.  Melatonin isn't meant for putting you to sleep immediately. You take it several hours before your intended bed time, to shift your sleep schedule.
9.  I regularly get migraines if I stay awake >2-3 hours past my normal bed time.
10.  I can quickly fix my sleeping schedule if I want to.
11.  Shifting your sleep schedule too quickly can lead to poor sleep.
12.  Consistency matters a lot for good sleep.
13.  After I fix my sleep schedule, it's glaringly obvious when I feel tired. You want to go to sleep on time.
14.  I can just decide to go to sleep. Yes, even if I'm doom scrolling on twitter late at night.
15.  Consciously weighing the value of going to sleep vs. continuing what I'm doing right now helps me realize I want to go to sleep. Is sacrificing tomorrow worth what you're doing today?
16.  I start reading OOD things when I'm awake too late: this includes things I've always wanted to read, but never got around to. Like textbooks, blog posts, papers etc.
17.  I can cough so much that I can't fall asleep for more than an hour for weeks.
18.  I become funnier when I'm severely sleep-deprived.
19.  Absent corrective efforts, my sleep schedule drifts forwards each night.
20.  When I sleep late, the next day I feel constrained by my sleep schedule, as I have fewer hours to stay awake. So I decide to sleep later to "have more time".
21.  Magnesium gives me crazy dreams. Like, spelunking caverns whilst hiding from the dragon god in Demon Souls, fighting zombies for chocolate in supermarkets, and finding a space-warping trash stream that accelerates till its velocity hits a singularity. Yes, that was a single dream.
22.  I can just change clothes at night if I'm too hot/cold in bed.
23.  Caffeine doesn't make me more awake, but does stave off tiredness for several hours.
24.  When I get mentally exhausted, I can take a 10-minute nap to get a second wind.
25.  Going to the gym at night doesn't make me that tired.
26.  Eating before bed makes it harder to sleep.

There are probably other things I'm missing, but I can't really think of what at the moment. So ends my attempt at a listicle. Man, this 500 word requirement is a bit tedious. Ah well, such requirements are useful and we had to set the limit somewhere.
# a quick thought about AI alignment

Epistemic status: I am new to AI alignment and still just learning the lit; please forgive if this is obvious or well-trodden ground, but I hadn't yet come across this point so far.

*“Before the law, there was no sin”*  (attr. Romans 5:13)

In nature, animals often do harm and cause suffering to animals of other species in the course of meeting their own needs. Many also do harm to (some) other members of the same species to advance their own (or kin) fitness. This is unsurprising from an evolutionary standpoint, and is not subject to ethical judgement. 

Humans, uniquely, arrived at ethical principles to prohibit harming other humans for their own gain. Humans, uniquely, arrived at the idea that we should prohibit or limit causing harm or suffering to animals of other species for our own gain. These advances seem attributable to humans' exceptional intelligence and the emergence of conceptual thought and rationality. It remains an uphill battle for such principles to become accepted and enacted by humans. This difficulty seems to stem from two limitations: insufficient application of rationality, and the fact that human brains are not very far diverged from our pre-human ancestors. We still experience contrary motivations and impulses baked in by millennia of selection pressure to optimally exploit all resources, including other animals and conspecifics.

So the thought occurred to me that if intelligence and rationality are what led humans to at least *try* to override exploitative impulses in favor of honoring rights and extending compassion to other creatures, then by extrapolation, if AGI were to become even *more* intelligent than humans, wouldn't it if anything trend even further in that direction? And an intelligence with a non-biological physical substrate might not have (or more easily escape) the conflicting instincts embedded in our animal brains. In short, it occurred to me that maybe the default expectation should be that AGIs would be *more* aligned with (rational) human values than humans are.

This is by no means a strong argument, and I'm not at all inclined to defend this thesis. I just thought the idea was interesting to consider.  I will be happy to receive pointers to where this may have been previously articulated or debated.
# How likely are “s-risks” (large-scale suffering outcomes) from unaligned AI compared to extinction risks?

I’ve been reading about existential risks from advanced AI systems, including the possibility of “worse-than-death” scenarios sometimes called suffering risks (“s-risks”). These are outcomes where a misaligned AI could cause immense or astronomical amounts of suffering rather than simply extinguishing humanity.

My question: **Do researchers working on AI safety and longtermism have any informed sense of the relative likelihood of s-risks compared to extinction risks from unaligned AI?**

I’m aware that any numbers here would be speculative, and I’m not looking for precise forecasts, but for references, models, or qualitative arguments. For example:

Do most experts believe extinction is far more likely than long-lasting suffering scenarios?

Are there published attempts to put rough probabilities on these outcomes?

Are there any major disagreements in the field about this?

I’ve come across Kaj Sotala’s “Suffering risks: An introduction” and the work of the Center on Long-Term Risk, but I’d appreciate more recent or deeper resources.

Thanks in advance for any guidance.
# Lighthaven Sequences Reading Group #52 (Tuesday 10/7)

*Note: We will be meeting in building F this week.*

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights) and [the Best of LessWrong](https://www.lesswrong.com/bestoflesswrong). The mandatory readings this week are

1.  [How much do you believe your results?](https://www.lesswrong.com/posts/nnDTgmzRrzDMiPF9B/how-much-do-you-believe-your-results)

These posts should take around 15-20 minutes to read if you've read it before, and 30-60 mins if it's your first time.

The event starts at 6:30, and doors open 6pm (yes we will let you in early if you get here). At 6:30 we'll welcome people, give a few announcements, and then split into groups. Pizza will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# LLMs as a limiter of social intercourse

Ever since watching [this clip](https://youtu.be/7ky8yzS4VZY) of a Larry David interview, the phrase "social intercourse" has stuck with me. He uses it to describe things like dinner parties where people mix and mingle.

I thought about it the other day. I was in Cancun on vacation and was texting with a friend. I went on Google Maps to figure out what part of Cancun my hotel was in and noticed something odd: it said "Gulf of America" where I was expecting "Gulf of Mexico".

My first thought was to ask my friend whether it was a Trump thing that I missed. My second thought was that that's a lazy question: I could easily just look it up. But my third thought was that this second thought is one of a [midwit](https://knowyourmeme.com/memes/iq-bell-curve-midwit).

When I put my [Jedi cloak](https://www.figma.com/community/file/1037853630295134216/iq-bell-curve-midwit-meme-template) on I realize that information isn't the only consideration to think about here. Social intercourse is also a consideration. Talking to friends is nice, actually. So I asked, and it lead to a fun little exchange.

This makes me want to walk back a bit on the advice I previously had to "[give it a google](https://www.lesswrong.com/posts/d6yNW5T6J9rtnGizc/give-it-a-google)". Not that it's always bad advice, just that one should consider social intercourse in addition to knowledge seeking.

Consider the first example from that post; the one about poker. For me personally, I began my poker journey with such googling. It wasn't a bad idea; and yes, it's far superior to just winging it. But my cousin Jason plays poker. I could have reached out to him. He would have been able to do a fine job of teaching me, and it would have been a good excuse to catch up. We don't get to talk often enough.

So I guess what I'm trying to do here isn't really to walk back on the "give it a google" idea. It's more that I'm trying to expand and generalize it. The idea isn't to literally use Google. It's to do *some* form of initial research, and asking a friend (or family member or librarian or neighbor...) will often suffice.

And furthermore, I'm trying to say that when the initial research involves talking to a human, you get the benefit of social intercourse.[^sh5qri3g9hi] This should be part of your utilitarian calculus. Or, more pragmatically, you should probably just have a heuristic saying to "be mor social" alongside related heuristics such as "touch grass".

What about LLMs? Well, I kinda see LLMs as an extension of all of this.

Like, there's the set of things you could research by googling, and then there's the set of things you could research by prompting. The latter set is larger and is growing more quickly.

I think there's a failure mode with the first set where you don't give enough weight to social intercourse. Where you ask google how to fold your laundry instead of calling your mom. I think there are similar failure modes with the second set and that these failure modes are naturally more dangerous.

[^sh5qri3g9hi]: Amongst other things. Like, I dunno, maybe networking.
# Are We Leaving Literature To The Psychotic?

Those who have fallen victim to LLM psychosis often have a tendency to unceasingly spam machine-generated text into the text corpus that is the internet. There are many different reasons for this, but a popular one seems to be the impression that by doing so, they can shape the next generation of LLMs. They are likely correct. We are fooling ourselves if we believe that AI companies will be fully successful at filtering out questionable content in the near term.

The psychotic spammers are (unfortunately) very likely on to something here. I would be shocked if their spam had no effect on future LLMs. I’m not sure if any AI safety organization has put thought into what “optimal” spam would look like, and I’m certainly not suggesting we start counter-spamming the internet with sanity-inducing rationalist memes or anything, but…has any serious thought been put into how internet users \*should\* be shaping our collective text corpus to maximize chances of alignment-by-default?
# A Snippet On The Epistemically Hygienic Containment Of Faith-In-Reason-Itself

**Epistemics**: *I'm experimenting with "snippets" that are low now-effort material from the deep past. Maybe ancient comments? Maybe things I wrote with myself as an audience and was kind of embarrassed to post anywhere but didn't want to throw away? This is the former, written in a specific era, for a specific audience, with specific feelings and questions authentically expressed, and a* [*Gricean*](https://en.wikipedia.org/wiki/Cooperative_principle) *response offered in good faith. Shorn of such context, maybe it is bad?*

Arguably, most thoughts that most humans have are either original or good but not both. People seriously attempting to have good, original, pragmatically relevant thoughts about nearly any topic normally just shoot themselves in the foot. This has been [discussed](https://www.lesswrong.com/posts/aHaqgTNnFzD7NGLMx/reason-as-memetic-immune-disorder) [ad](https://www.lesswrong.com/posts/AdYdLP2sRqPMoe8fb/knowing-about-biases-can-hurt-people) [nauseum](https://www.lesswrong.com/posts/m5AH78nscsGjMbBwv/making-your-explicit-reasoning-trustworthy).

This place is not good for cognitive children, and indeed it MIGHT not be good for ANYONE! It could be that "speech to persuade" is simply a cultural and biological adaptation of the brain which primarily exists to allow people [to trick other people](https://www.edge.org/conversation/hugo_mercier-the-argumentative-theory) into giving them more resources, and the rest is just a [spandrel](https://en.wikipedia.org/wiki/Spandrel_(biology)) at best.

It is admirable that you have restrained yourself from spreading links to this website to people you care about and you should continue this practice in the future. One experiment per family is probably more than enough.

--

HOWEVER, also, you should not try to regulate speech here so that it is safe for dumb people without the ability to calculate probabilities, detect irony, doubt things they read, or otherwise tolerate cognitive "ickiness" that may adhere to various ideas not normally explored or taught.

There is a possibility that original thinking is valuable, and it is possible that developing the capacity for such thinking through the consideration of complex topics is also valuable. This site presupposes the value of such cognitive experimentation, and then follows that impulse to whatever conclusions it leads to.

Regulating speech here to a level so low as to be "safe for anyone to be exposed to" would basically defeat the point of the site.
# How long do AI companies have to achieve significant capability gains before funding collapses?

It’s an open secret that essentially all major AI companies are burning cash and running at massive losses.   
  
If progress is slow enough such that it requires X years of continued funding to achieve AI capabilities at least useful enough to produce a net ROI, at what value of X will the economics collapse, resulting in a major downscaling or total collapse of these companies?
# Non-copyability as a security feature

It seems hard to imagine that there's anything humans can do that AIs (+robots) won't eventually also be able to do. And AIs are cheaply copyable, allowing you to save costs on training and parallelize the work much more. That's the fundamental argument why you'd expect to see AI displace a lot of human labor.

But both AIs and humans are vulnerable to being tricked into sharing secrets, but so far AIs are more vulnerable, and there's not really any algorithms on the horizon that seem likely to change this. Furthermore, if one exploits the copyability of AIs to run them at bigger scale, then that makes it possible for attackers to scale their exploits correspondingly.

This becomes a problem when one wants the AI to be able to learn from experience. You can't condition an AI on experience from one customer and then use that AI on tasks from another customer, as then you have a high risk of leaking information. By contrast, humans automatically learn from experience, with acceptable security profiles.
# Experiment: Test your priors on Bernoulli processes.

I have run 1,000,000 experiments. Each experiment consists of 5 trials with binary outcomes, either $L$ (for left) or $R$ (for right).

However, I'm not going to tell you how I've picked my experiments. Maybe I'm just flipping a fair coin each time. Maybe I'm using a biased coin. Or maybe I'm doing something completely different, like dropping a bouncy ball down a mountain and checking whether it hits a red rock or a white rock first--and different experiments are conducted on different mountains. I might be doing some combination of all three.

You do get one guarantee, though: All the experiments are Bernoulli processes. In particular, the *order* of the trials is irrelevant.

Your goal is to guess the marginal frequencies of the fifth trial. For each $k = 0, 1, \dots, 4$, you need to tell me the frequency that the fifth trial is an $R$ given that $k$ of the outcomes of the first four trials are $R$.

For example, if every experiment is just flipping a fair coin, then the fifth trial will be an $R$ with probability $1/2$, no matter what the first four are. However, if I'm using biased coins, then the frequency of $R$ will increase the more $R$s seen.

To help you in your guessing, I have provided a [csv of all the public trials](https://raw.githubusercontent.com/cooljoseph1/prior-testing/refs/heads/public/public.csv). As an answer, please provide a list like \[0.3, 0.4, 0.5, 0.6, 0.7\] of your frequencies--the kth element of your list is the marginal frequency over the experiments with $k$ of the first four trials being $R$.

I haven't yet looked at the frequencies myself, but I will do so shortly after posting this. If you want to test your guesses against others, I have created a [market on Manifold Markets](https://manifold.markets/JosephCamacho/how-good-are-your-priors-calculate?r=Sm9zZXBoQ2FtYWNobw). I will resolve the market before I reveal the correct frequencies, which will happen in around two weeks, but maybe earlier or later depending on trading volume.

Good luck!
# RiskiPedia

RiskiPedia is a collaborative, data-driven, interactive encyclopedia of risks. Well, it is far from an encyclopedia right now; it is just launching, and mostly consists of half-baked pages I’ve created to exercise the MediaWiki extensions that make the pages interactive.

Try it at [https://riski.wiki](https://riski.wiki) \-\- you can explore the chances you'll [end up in the emergency room tomorrow](https://riski.wiki/wiki/Unexpected_ER_Visit) or the risk you'll get [mauled by a grizzly](https://riski.wiki/wiki/Grizzly_Bear_Attacks_in_the_USA) if you spend all summer hiking in Glacier National park.

I’m working on it because I’ve been frustrated that risks are typically presented as binary “safe” or “dangerous”, usually with no mention of **how** dangerous or safe. I hope it will help people think more clearly about risks, so at least a few of them stop worrying about things that aren't very risky and maybe start worrying more about the things that actually are risky. Like [driving](https://riski.wiki/wiki/Driving).

I’m hoping to recruit early contributors from the rational thinking community to help launch RiskiPedia by:

1.  Creating more pages about risks that people care about, or risks that people **should** care about.
2.  Fact-checking the risks that are already there.
3.  Helping with wiki administration: approving users, deleting spam (if it appears), helping to get consensus on policies and procedures and all the other Wiki-community stuff that I should probably know more about before launching this thing.
4.  If you're a total geek, help out with the behind-the-scenes coding (it is all open source php and javascript up on [github](https://github.com/RiskiPedia)).

Anybody can create an account and contribute, don't feel like you need permission to jump in and help out.

RiskiPedia is not a business; if it is wildly successful there might be a RiskiPedia Foundation to support it. For now, I'll be funding anything that absolutely needs funding (like the server that it runs on) out of my own pocket.
# Pause House, Blackpool

Are you passionate about pushing for a global halt to AGI development? An international treaty banning superintelligent AI? Pausing AI? Before it’s too late to prevent human extinction?

Would you like to live with a group of like-minded people pushing for the same?

Do you want to do much more, but don’t have the financial means to support yourself volunteering?

Then [apply to stay at Pause House](https://docs.google.com/forms/d/e/1FAIpQLSdVGXftQM0rt2eGt1FiodPB1k0NZFzbryG8N5S0V1mZpsHDQQ/viewform?usp=sharing&ouid=104406273905203370907). I (Greg Colbourn), am offering free accommodation, (vegan) food, and a small stipend for those who need it (£50/week). In exchange I ask for a commitment to spending at least 20 hrs a week on work related to pushing for a Pause. This could be either campaigning/advocacy, or raising public awareness.

If you have an income, then I ask that you pay ~cost price (£20/day).

Pause House is located in Blackpool, UK, next door to [CEEALAR](https://ceealar.org/) (which I founded in 2018); 1hr from Manchester, 3.5hrs from London. It has a large communal work/social/events room, dining room, well equipped kitchen, toilet, laundry and small courtyard downstairs. And a meeting room and 12 bedrooms in the upper floors, 4 with en suite bathrooms (first come, first served!). The other 8 bedrooms all have sinks in them, and the use of 2 shared showers and 2 shared toilets.
# Postrationality: An Oral History

[@Gordon Seidoh Worley](https://www.lesswrong.com/users/gordon-seidoh-worley?mention=user) is going to be giving a presentation about the history of the postrationalist movement, as he remember its, as part of the Int/a Speaker Series.

The event takes place online and you can sign up to attend on Luma:

[https://luma.com/6l2wiq7v?tk=GulAmK](https://luma.com/6l2wiq7v?tk=GulAmK)
# Humans Are Spiky (In an LLM World)

Assessments of "general" vs "spiky" capability profiles are secretly assessments of "matches existing infrastructure" vs "doesn't".

Human societies contain human-shaped roles because humans were the only available workers for most of history. Packaging tasks into human-sized, human-shaped jobs was efficient.

Given LLMs, the obvious thing to do is to try to drop them into those roles, giving them the same tools and affordances humans have. When that fails to work, though, we should not immediately conclude that the failure is because LLMs are missing some "core of generality".

When LLM agents become more abundant than humans, as seems likely in the very near term, the most effective shape for a job stops being human-shaped. At that point, we may discover that human capability profiles are the spiky ones.