Harley-ml/lesswrong huggingface dataset:
# Job Openings: SWE, PM, and Grants Coordinator to help improve grant-making

Survival and Flourishing Corp is growing our team to support the continued development and scaling of our grant evaluation software and processes. We're looking for talented individuals who want to join us in our mission of improving philanthropic decision-making and impact.

[Full-Stack Software Engineer (S-Process)](https://survivalandflourishing.com/careers/full-stack-engineer)  
Annual compensation: **$250,000 - $350,000**  
[**SWE Application Form**](https://docs.google.com/forms/d/e/1FAIpQLScFq7uUslkPX2dzhlVChwJa3KJUR7KrPkolctafd3sIEjCpcQ/viewform?usp=header)

[Product Manager (S-Process)](https://survivalandflourishing.com/careers/product-manager)  
Annual compensation: **$150,000 - $200,000**  
[**PM Application Form**](https://docs.google.com/forms/d/e/1FAIpQLScyLR6vawc2EwizTjswvFO81awDEjQLjI9CYzIyhjVXiiOyAQ/viewform?usp=header)

[Grants Coordinator (S-Process)](https://survivalandflourishing.com/careers/grants-coordinator)  
Annual compensation: **$60,000 - $80,000**  
[**GC Application Form**](https://docs.google.com/forms/d/e/1FAIpQLSex4CIW0kylXhiCXIPlOMfckstU7-pri09iOfI3oBqQOnzJFw/viewform?usp=header)

Why work with us
----------------

SFC is a small, mission-driven team working to improve how philanthropy operates. We've developed grant evaluation software that has been used by several philanthropists to date, including Jaan Tallinn, Blake Borgeson, and the Future of Life Institute, through the [S-Process](https://survivalandflourishing.fund/s-process) of the [Survival and Flourishing Fund](https://survivalandflourishing.fund/).

Our main focus now is on making the app more information-dense and less cognitively taxing for philanthropists, enabling them to engage in more impactful giving. We have a clear roadmap of high-value features and are excited to expand the team so we can build more, faster.

### What we offer

*   Competitive compensation with room for negotiation based on experience
*   Fully funded individual health, vision, and dental insurance
*   Majority premium coverage for additional dependents
*   Available health/wellness discounts and perks through the health insurance provider
*   6% Safe-Harbor 401(k) matching
*   9 weeks off per year, including PTO and holidays
*   12-week paid family leave
*   Fully remote work with 2-4 in-person team workweeks per year in the Bay Area
*   Opportunity to work on meaningful projects that improve philanthropic impact
# Steven Adler reports that NVIDIA is attempting to stifle pro-export-control speach

> For months now, I’ve heard about widespread fear among think tank researchers and policy experts who publish work against NVIDIA’s interests.
> 
> NVIDIA is the most valuable company on Earth, the dominant supplier of AI chips, and is currently valued at 4.5 trillion dollars. I worry that fear of its potential retaliation—rumors of attempted firings and media hit pieces—is distorting some of the most important AI policy debates.
> 
> Based on what I’ve heard, there is no smoking gun of wrongdoing, but certainly a pattern of fear that impacts policy discussion, justified or not. This fear isn’t just limited to think tank staffers either: I’ve heard many concerning claims that the major AI companies disagree with NVIDIA’s policy stances and know that its talking points are flawed, but are unwilling to speak out because NVIDIA has the power to undermine their business.
> 
> To date, these details have mostly dripped out in isolated places online, and so I’ve done the legwork to document and consolidate them to one place, in addition to speaking with a wide range of sources for further substantiation.

[continued](https://stevenadler.substack.com/p/the-45-trillion-dollar-elephant-in)
# I handbound a book of Janus's essays for my girlfriend

  
My girlfriend loves Janus's writing on simulators and AI, so I made her a physical copy she could keep on her shelf. Not print-on-demand or a binding service, but handbound from raw materials. A unique Artifact.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3ef1cc4a138a249cfe892cea9947cc00b6ac150d948d10c7.jpg)

For the cover I laser cut brass inlays into burgundy cloth over bookboard. Most books just have foil imprints or plain impressions but I wanted something that felt substantial, something that would look awesome on her shelf rather than being Paperback #784. The text block was done with French link stitches using waxed linen thread and a golden ribbon bookmark to match with the brass on the cover.

The text block itself is printed on 28/70 flecked cream paper. I typeset the whole thing with Next.js and Paged.js, which was honestly *way* more pleasant than fighting with LaTeX. [Here's the PDF](https://drive.google.com/file/d/1XnElW-ILt9B3Aq19K8F-WIWsgM6Cz7Vx/view?usp=sharing) if anyone wants to see the layout. You would not believe how much time I spent thinking about things like margin ratios and footnotes and chapter breaks.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e60bad5c33230a1c3cb7c3cc63594bf7790c14036db304cf.jpg)

This is only the second book I've made and it came out way better than I expected. The brass looks amazing against the burgundy cloth and the whole thing feels hefty and satisfying to hold.

I did the binding and most of the typesetting, but Fiora edited Janus's essays to clean them up for the print format, made numerous suggestions on the design, and even wrote a [preface](https://www.lesswrong.com/posts/5EJQGYvohJpvEZwKd/preface-to-simulacra-and-simulators-dreams-of-the-predictive) for the whole thing.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/edd869bd12c88966282a61d4ad738f49aaef2d8a7e9bde4a.jpg)
# Space colonization and scientific discovery could be mandatory for successful defensive AI

*Epistemic status: quick draft of a few hours thought, related to a few weeks cooperative research *

In a multipolar ASI offense/defense scenario, there seems to be a good chance that intent-aligned, friendly AI will not colonize space. This could for example happen because we intent-align defensive AI(s) with institutes under human control, such as companies, police forces, secret services, militaries or military alliances, governments, or supragovernmental organizations. The humans controlling these entities might not support space colonization, space colonization might be outside their organization’s mandate, or there might be other organizational constraints prohibiting space colonization.

If an offensive AI (either unaligned, or intent-aligned with a bad actor) escapes into space, it might be able to colonize the resources it finds there. For example, it could build a laser with a beam diameter exceeding earth's and use it against us. Or, it could direct a meteorite at us large enough to cause extinction. In these scenarios, it seems impossible for earth-bound defensive AI to successfully ward off the attack, or for us, and the defensive AI(s), to recover from it.

Therefore, if:

1.  We end up in a multipolar ASI offense/defense scenario (e.g. because no pivotal act was performed), and
2.  Defensive AI is intent-aligned with humans who do not effectively colonize space, and
3.  Offensive AI escapes into space, and
4.  Escaped offensive AI can mobilize space resources to build a decisively large weapon,

It seems to follow that offense trumps defense, possibly leading to human extinction.

More generally, a *minimum viable defense theorem* could be formulated for multipolar ASI offense/defense scenarios:

*If mobilizing resources can lead to a decisive strategic advantage, any successful (system of) defensive AI(s) should at least mobilize sufficient resources to win from any weaponry that could be constructed from the unmobilized resources.*

One could also imagine that weaponizing new science and technology could lead to a decisive strategic advantage. A version of this theory could therefore also be:

*If inventing weaponizable science and technology leads to a decisive strategic advantage, any successful (system of) defensive AIs should at least invent and weaponize sufficient science and technology to successfully defend against any weaponry that could be constructed from the uninvented science and technology.*

These results might be seen as a reason to:

*   Support a pause.
*   Perform a pivotal act (if ASI can be aligned).
*   Make sure we align (if ASI can be aligned) defensive, friendly ASI with entities which intent to occupy sufficient strategic space in domains such as space colonization and weaponizable science.
# Selected Graphics Showing Progress towards AGI

| Except from the upcoming post: *Beyond Human Wisdom: Can We Survive the Rise of AGI?* Coming soon 😊(PM me if you'd be keen to provide feedback on the draft). |
| --- |

### Motivation

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/99b0eb6f239995666284c3acaffe50ed2d6717c581f6624a.png)

Motivating image —  [`View on Life Itself`](https://lifeitself.org/learn/wisdom-gap)

| **The****🅂🅄🅅****Triad (the challenge)** |
| --- |
| **🅂peed** \\- in absolute terms and relative to the speed of governance **🅄ncertainty** - regarding the situation and strategy **🅅ulnerability** - many catastrophic threats that are hard or costly to defend against |

### Graphs on Progress

| ![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/97f167e4c232553b5390629a6c6a99e74727e994cdeefb53.png) [` View on Metaculus `](https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/) [` View Goodheart Lab's forecast aggregator `](https://agi.goodheartlabs.com/) ![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/f484bf41f1ea0e3f1decc3dcc545cc85e131312e78f87dba.png) [` A Definition of AGI based on Cattell-Horn-Carroll theory `](https://www.agidefinition.ai/) ![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/8cd9c6da4e7bfc70a3009cccdc88dd589cd883f367e05d8f.png) ![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/b8cf5835d33ce2f9725b1fcbb5c5b86b7a27d1042c776219.png) [` View on Dewi Erwan's Github `](https://dewierwan.github.io/ai-training-diffusion/?utm_source=bluedot-impact) ![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/b58937ca11b189c59eaa09d5581367c126642aa627c5ffc6.png) [`View at International Scientific Report on the Safety of Advanced AI`](https://assets.publishing.service.gov.uk/media/6716673b96def6d27a4c9b24/international_scientific_report_on_the_safety_of_advanced_ai_interim_report.pdf) ![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/ed4b192b512154ade3e017a619ad3bfad335de1c1db1916c.png) [` View on OpenAI Blog `](https://openai.com/index/gdpval/) ![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/b437cde5f64eaca0dcf2dfcfd704065afeef82449759713c.png) [` View on METR `](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) |
| --- |
# The If Anyone Builds It, Everyone Dies march assurance contract should indicate how many signatures it has received

Looking at the If Anyone Builds It, Everyone Dies website, I really like the materials it provides in extending on the book. I also like the idea of an assurance contract for a large event ([dismissals from PauseAI movement leadership notwithstanding](https://www.youtube.com/watch?v=5yw8H9lFiAk)). However, I see a key problem with how it's set up: there is no indication of how many signatures it has received.

Adding a signature indicator would serve many useful roles:

1.  It would provide evidence of how close or far we are from reaching the goal, which would improve in planning ahead if, say, it's close to getting enough signatures and we should start planning a date
2.  It would function well as a motivator for future attention on getting more signatures (people like to see numbers go up)
3.  It would ensure greater transparency to the community on how well or poorly the campaign is going
4.  The lack of an indicator may cause people to be suspicious that e.g. it has received unusually few signatures, or the signature list is not in an organized database, or something similar

I wonder why this has not already been done, and if there are any technical obstacles for the site in setting it up.
# Lighthaven Sequences Reading Group #54 (Tuesday 10/21)

*Note: We will be meeting in building F this week.*

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights) and [the Best of LessWrong](https://www.lesswrong.com/bestoflesswrong). The mandatory readings this week are

1.  [Your Cheerful Price](https://www.lesswrong.com/posts/MzKKi7niyEqkBPnyu/your-cheerful-price)

These posts should take around 15-20 minutes to read if you've read it before, and 30-60 mins if it's your first time.

The event starts at 6:30, and doors open 6pm (yes we will let you in early if you get here). At 6:30 we'll welcome people, give a few announcements, and then split into groups. Pizza will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# An epistemic theory of populism [link post to Joseph Heath]

_The following is an AI-generated summary_

# Populism Fast and Slow - Summary

Joseph Heath argues that existing academic definitions of populism—whether 
treating it as an ideology or merely an electoral strategy—are inadequate. He 
proposes understanding populism through the lens of **dual-process cognitive 
theory** (Kahneman's "fast and slow" thinking).

## Core Argument

Populism is a political strategy that privileges **intuitive cognition** 
(System 1: fast, effortless, "common sense") over **analytical reasoning** 
(System 2: slow, effortful, requiring expertise). This creates a fundamental 
divide between "the people" who rely on intuition and "elites" who employ 
analytical thinking.

## Why This Matters

Many evolved cognitive intuitions work poorly in modern large-scale societies. 
For example:

- People intuitively overestimate punishment's effectiveness (due to 
  regression-to-mean bias)
- They misunderstand trade and immigration economics
- They struggle with collective action problems

When experts develop views contradicting these intuitions through analytical 
reasoning, it creates lasting resentment. Populist politicians exploit this gap 
by championing common sense views on issues where elite consensus is strongest.

## Why Populism Thrives Now

Social media accelerates communication, favoring fast intuitive responses over 
slow analytical ones. It also removes elite gatekeepers, allowing direct appeals 
to popular intuition.

## Specific Features Explained

1. **Crime/immigration stances** - Intuition favors punishment over expert 
   consensus on effectiveness
2. **Poor handling of collective action problems** - Intuition suggests blaming 
   others rather than coordinated restraint
3. **Stream-of-consciousness speaking style** - Demonstrates lack of verbal 
   inhibition, perceived as "honesty"
4. **Illiberalism** - Difficulty with abstract liberal principles requiring 
   cognitive decoupling
5. **Conspiracy thinking** - Natural cognitive bias toward conspiracism 
   requires active analytical suppression

## Why the Left Struggles

The rebellion is against *cognitive* elites, not economic ones. The left's 
progressive agenda requires more cognitive inhibition and control (e.g., 
language policing, navigating complex bureaucracies), intensifying the very 
burdens that fuel populist resentment.
# 21st Century Civilization curriculum

[I’ve just released](https://x.com/richardmcngo/status/1980531594139238622?s=46) a curriculum on foundational questions in modern politics, which I drew up in collaboration with [Samo Burja](https://x.com/SamoBurja?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor). I’ve copied the introductory text and the section headings below; you can find the full curriculum at [www.21civ.com](https://www.21civ.com/).

[**Sign up here**](https://forms.gle/aNUqFsrNFxipkUMm7) **by 27 October to join the first cohort of discussion groups** (which will meet weekly to discuss each of the 11 sections of the curriculum).

This curriculum is about Western civilization, and how it enables citizens of the Western world to live together in a just, orderly way. But it’s also about the 21st century, which has been characterized by the continual decline of many aspects of that civilization.

Despite our superior technology, there are many things that Western countries could do in the past that we can’t today—e.g. rapidly build large-scale infrastructure, maintain low-crime cities, and run competent bureaucracies. More importantly, it feels like there are no adults in the room: modern elites often seem unvirtuous and even unserious by historical standards. This curriculum focuses on explaining what changed, and how to orient to the world we now find ourselves in.

Discussions of large-scale political issues can be unsettling or jarring. So the curriculum intertwines discussion of what’s happening on a factual level, with readings on how to develop a healthy emotional and ethical stance towards politics (culminating in the final week’s focus on cultivating virtue). It also strongly prioritizes honesty and clarity of writing (even on topics often considered taboo), which is one reason why most readings are informal blog posts or essays rather than academic papers.

Given the sheer scope of the topics covered by the curriculum, it does not aim at comprehensiveness; nor does it try to give detailed strategies for solving civilizational decay. Indeed, given the accelerating development of AI (as discussed in week 10) the coming decades are likely to be extremely unpredictable. Readers should instead think of the curriculum as a starting point for informed, realistic discussions about how we as a civilization can steer ourselves through the coming turmoil.

### **Contents**

*   Week 1: Western Culture
*   Week 2: Civilizational Decay
*   Week 3: The Managerial State
*   Week 4: The Open Society and its Discontents
*   Week 5: The Psychology of Modern Elites
*   Week 6: The Unprotected Class
*   Week 7: World on Fire
*   Week 8: Industry and Money
*   Week 9: Sociopolitics
*   Week 10: Technology and Civilization
*   Week 11: Reviving Virtue

[Read the full curriculum here](http://www.21civ.com/).
# Remarks on Bayesian studies from 1963

In 1963, Mosteller and Wallace published [Inference in an Authorship Problem](https://gwern.net/doc/statistics/bayes/1963-mosteller.pdf), which used Bayesian statistics to try to infer who wrote some of the disputed *Federalist* Papers. (Answer: Madison) Anyway, at the end they have a list of "Remarks on Bayesian studies" which is astonishing to read 62 years later:

> 1.  Study of variation of results with different priors is recommended. Bracketing the prior is often easy. When founded in data, the choice of the prior has a status like that of the data distribution—subjectivity tempered with empiricism.
> 2.  Where possible, priors should be empirically oriented. Planning ahead for the collection of data suitable for estimating the form and underlying constants of prior distribution is useful and important, and very likely not hard to do once one gets in the frame of mind of preparing to use Bayes' theorem. The remark is all the more germane in the field of repetitive studies,where there has never been much excuse for the failure to collect such data.
> 3.  In any method of inference, data distributions matter enormously, and their study and choice requires the development of new and systematic methods.
> 4.  Statisticians need to provide richer sets of priors and data distributions than we now have, with a view to their manageability as well as their utility in studies of data. For example, mixture distributions can give satisfactory tail properties, but are grisly mathematically.
> 5.  The moment one leaves the simplest application of Bayes' theorem, the applier finds himself involved in a welter of makeshifts and approximations. This trouble and its cures require systematic developments. The prospect of useful exact treatments can be neglected.
> 6.  Simple Bayesian methods are needed that can be applied without appeal to high speed computers.
> 7.  Users of Bayes' theorem will find that many statistical jobs are readily handled by standard devices, but not by available Bayesian techniques. To avoid such devices for the sake of consistency reduces the quality of the research. On the other hand, time spent developing Bayesian procedures for a standard problem can be a profitable pursuit.
> 8.  In summary, for large scale data analysis, Bayesian methods require new studies of theoretical and empirical distributions and their approximation and estimation comparable in extent to those provided by statisticians up to 1935.

Some things never change!
# White House OSTP AI Deregulation Public Comment Period Ends Oct. 27

The White House's Office of Science and Technology Policy [has issued a request for information (RFI)](https://www.federalregister.gov/documents/2025/09/26/2025-18737/notice-of-request-for-information-regulatory-reform-on-artificial-intelligence) relevant to regulations that "unnecessarily hinder the development, deployment, and adoption of artificial intelligence (AI) technologies within the United States."

If you have anything to say about that topic that you want the U.S. government to hear, consider [submitting a public comment](https://www.federalregister.gov/documents/2025/09/26/2025-18737/notice-of-request-for-information-regulatory-reform-on-artificial-intelligence#open-comment) before 11:59 p.m. Eastern Time on Monday 27 October 2025.

The emphasis on removing regulations is pursuant to the [White House's AI Action Plan](https://www.whitehouse.gov/wp-content/uploads/2025/07/Americas-AI-Action-Plan.pdf), which was the subject of [a previous RFI](https://www.federalregister.gov/documents/2025/02/06/2025-02305/request-for-information-on-the-development-of-an-artificial-intelligence-ai-action-plan).

I submitted [a one-page commment](https://files.nitrd.gov/90-fr-9088/Zack-Davis-AI-RFI-2025.pdf) to that one, which did get dutifully included in [the 18,480 page PDF of 10,067 responses](https://www.nitrd.gov/coordination-areas/ai/90-fr-9088-responses/). Interestingly, the new RFI on deregulation only has 243 comments at press time for this post; perhaps informative comments are less likely to get drowned out in the noise this time.
# Any corrigibility naysayers outside of MIRI?

> It's plausible that humanity could make a corrigible ASI by 2035 if the planet was united around that goal and being very careful.

Are there any knowledgeable people outside MIRI who might disagree with me on this statement and be interested in arguing with me about it? I'm more interested in the corrigibility bit than the ASI bit. (Like, if Gary Marcus might argue that we're not getting ASI by 2035 regardless, but that's not what I'm looking for.)
# Statement on Superintelligence - FLI Open Letter

> We call for a prohibition on the development of superintelligence, not lifted before there is
> 
> 1.  broad scientific consensus that it will be done safely and controllably, and
> 2.  strong public buy-in.

Sign on [the main website](https://superintelligence-statement.org/) or [Eko](https://action.eko.org/a/sign-the-statement-on-superintelligence)[^oginbvw23yo].

[^oginbvw23yo]: Looks like the latter is lower checking/privacy preserving, and gets added to the total figure?
# Is terminal lucidity real?

As a neuroscientist, I became fascinated by the supposed phenomenon of 'terminal lucidity', where people with severe dementia suddenly become lucid again shortly before death. It sounds impossible given how catastrophically damaged their brains are, but it keeps being reported across cultures and centuries. If it's real, it could suggest new treatment methods for Alzheimer's (not just another ineffective amyloid-clearing antibody), yet it's shockingly understudied.
# SecureBio is Hiring Software Engineers

I've had a lot of people reach out to me who are interested in working on biosecurity, but have a background in software engineering / computer science. A lot of these conversations have looked something like:

> A: I'd be really excited to work on biosecurity, it seems [really important](https://forum.effectivealtruism.org/posts/33t5jPzxEcFXLCPjq/the-four-pillars-a-hypothesis-for-countering-catastrophic) and [relatively neglected](https://forum.effectivealtruism.org/posts/7wBZG8skNp8o4Mb9v/ajeya-s-quick-takes?commentId=dMzz4FxHbovLaxqL9). Are you hiring for software engineers at SecureBio?
> 
> Me: I wish we were, you seem really great! But I don't know when we will be, depends on funding and some strategy questions.

This has now changed, and SecureBio is now hiring for two different software engineering roles that don't require a biology background:

*   NAO: [Bioinformatics Engineer](https://securebio.org/careers/bioinformatics-engineer-2025/). An in-person role on the [project I lead](https://naobservatory.org/), developing a metagenomic biosurveillance system to detect [stealth pathogens](https://www.gcsp.ch/publications/securing-civilisation-against-catastrophic-pandemics).
    
*   AI: [Software Engineer, AI Safety and Biosecurity](https://securebio.org/careers/software-engineer-ai-safety-and-biosecurity/). A remote role on the [project Seth leads](https://securebio.org/ai/), evaluating frontier AI systems, with a focus on biosecurity and misuse risk.
    

Consider applying?

Happy to answer questions!

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02cYEEZLS6boYHCz1VkncYzch77Z1rcD3FUbPJakSgGmwBzUQuw1T1j5R7t4zzPjfCl), [lesswrong](https://lesswrong.com/posts/myxjkEopX3C9RDdp4), [the EA Forum](https://forum.effectivealtruism.org/posts/8zyMEBhDch9PCuyu5), [mastodon](https://mastodon.mit.edu/@jefftk/115423878465681941), [bluesky](https://bsky.app/profile/jefftk.com/post/3m3ujyhx4x223)*
# AI Timelines and Points of no return

In this essay, I introduce two **Points of No Return (PNR)**:

*   **The Hard PNR**. The moment where we have AI systems powerful and intelligent enough that they can prevent humanity from turning them off.
*   **The Soft PNR**. The moment where we have AI systems that we *will not want to* turn off. For instance, because they are too intertwined with the economy, military systems, geopolitics, or romantic relationships.

I believe the Soft PNR is underrated (especially in rationalist communities).

I also believe that it is roughly as irrecoverable as the Hard PNR, and that it is quite likely to come first.

Cheers!
# LW Reacts pack for Discord/Slack/etc

Edit: Consider using [the SVGs from the wiki page](https://www.lesswrong.com/w/lesswrong-reacts).

Ever wanted to say Scout Mindset to someone on a chat platform, but didn't want to have to type 13 characters?

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a0dc7f9babb4b32bffa64a84b5c18f5b196d808037c3002a.png)

Or wanted your server to have magically better epistemics because of a few images?

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a644ea4247b2b8c8195a2f817d5e52cb6100c2d374dc3662.png)

The LW Reactions Pack is here to help! With 63 images (or 37 for the cut down budget set of good ones if you're limited), we guarantee you will have more reaction options than you started with.

*   [Full Pack (63)](https://drive.google.com/drive/folders/1hPRbALWKE9lQ0JA97NNdhWn2UNlWlfOi?usp=sharing)
*   [Important only (37)](https://drive.google.com/drive/folders/1NhEtVqHYnN-oHCwUzgVFMiXwsvQ_pySy?usp=sharing)
# Why is OpenAI releasing products like Sora and Atlas?

Recently, OpenAI has been releasing products that don't seem to advance the goal of creating AGI or Superintelligent AI. The social video product Sora is one such product. Though at least it involves a new AI model that might teach them lessons about developing capabilities. Another is the AI-enabled browser Atlas.

*Why is OpenAI spending the manpower, compute, and management focus to create and support these products instead of using those resources to race toward more powerful AI?*

I am most interested in public sources from OpenAI or answers from those who have direct knowledge. If you have a different speculation from mine, share that too.

Speculation on possible reasons:

*   Investors want OpenAI to have more revenue or more products. I could imagine that not all investors fully believe the vision of reaching AGI, or suspect that reaching superintelligence will destroy their investment, as capitalism is fully disrupted.
*   These products enable collecting data that they can use to train AIs. (I'm not clear how that would be the case.)
*   They are "dogfooding" their coding assistant AI and need projects to use it on, so they have made these products. (But even if you've developed it, that doesn't mean it is worth the costs of releasing and supporting it.)
*   Their employees want to develop cool products and see the fruits of their work. Creating and releasing these products keeps the employees happy, so they will continue to be maximally productive in advancing AI.
*   OpenAI is bottlenecked on AI researchers. They cannot develop more powerful AI any faster than they are, so they can afford to hire developers and others to create products. Those products encourage the buildout of computing capabilities that will be available when the AI researchers need them.
*   AI development is slowing or hitting bottlenecks, so they want to capture the market share of products that can be developed with current levels of AI.
# Paper: Take Goodhart Seriously: Principled Limit on General-Purpose AI Optimization

**TL;DR:** This paper takes existing mathematical results to build the most general and rigorous case for why we should be very cautious about pushing optimization too far in General-Purpose AI systems, as it likely leads to catastrophic Goodhart failures, and ultimately loss of control.

Written by Antoine Maier, AI Security Researcher at the [General-Purpose AI Policy Lab](https://www.linkedin.com/company/gpai-policy-lab/), and Aude Maier, PhD student at the École Polytechnique Fédérale de Lausanne (EPFL).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/17ab43e864cd62405d0f01f1475c1b727f56470d34b0c07e.png)
# Introducing the Epoch Capabilities Index (ECI)

We at Epoch AI have recently released a [new composite AI capability index](https://epoch.ai/benchmarks/eci) called the Epoch Capabilities Index (ECI), based on nearly 40 underlying benchmarks.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0b3c0119fa9a71a2cdb9b559ed1c6f6e55374c1acd9b98c0.png)

Some key features...

*   **Saturation-proof**: ECI "stitches" benchmarks together, to enable comparisons even as individual benchmarks become saturated.
*   **Global comparisons**: Models can be compared, even if they were never evaluated on the same benchmarks.
*   **Difficulty-based task weighting**: ECI uses a simple statistical model (similar to those used in Item Response Theory) under which models deemed more capable if they score well on difficult benchmarks, and benchmarks are deemed more difficult if capable models are unable to score highly on them. 

ECI will allow us to track trends in capabilities over longer spans of time, potentially revealing changes in the pace of progress. It will also improve other analyses that would otherwise depend on a single benchmark for comparison.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9268c5ee28eebbf3ef1326fa038dba4af5f44b20e966d8e4.png)

You can find more details about our methodology and how to interpret ECI [here](https://epoch.ai/benchmarks/eci). We want this to be useful for others' research; comments and questions are highly valuable to us!

This release is an independent Epoch AI product, but it builds on research done with support from Google DeepMind. We'll be releasing a full paper with more analysis soon, so keep an eye out!
# TBC Episode with Max Harms - Red Heart and If Anyone Builds It, Everyone Dies

This is a linkpost to the latest episode of The Bayesian Conspiracy podcast. We talk with Max Harms on the air for the first time since [2017](https://www.thebayesianconspiracy.com/2017/03/29-fiction-and-fun-with-max-harms/)! He's got a new novel coming out - Red Heart. Order your copy [here](https://maxharms.com/redheart) or at [Amazon](https://amzn.to/3JlqAU6). We spend about the first half talking about [If Anyone Builds It, Everyone Dies](https://amzn.to/47s263s).   
  
Max is a great guy who helped us get the initial momentum for starting our local LessWrong community in Denver about a decade ago. He works for MIRI, writes [novels](https://crystalbooks.ai/) and [essays](https://utopiandreams.substack.com/), and is an awesome and interesting person. I expect many of you to also find much of the conversation interesting. 

As a bonus, the last five minutes of the episode is a clip of the first five minutes of the audiobook of Red Heart.
# Genius is Not About Genius

For a long time, I was confused by others belittling geniuses as morons, and glorying morons as geniuses. Then it hit me: genius is not about genius. It's about status. Duh. 

When you call someone a genius, you are making a claim that they should be viewed as high-status. And vice versa when you call someone a moron. 

Which wouldn't be a big issue when it comes to evaluating whether someone were a genius if it were not for a couple of unfortunate facts. First, it is hard to evaluate expertise, outside a few areas like weight-lifting, where everyone can verify that that sure is a big guy lifting a big rock. Which plays into the second issue: genius is spiky. You can be the world's greatest chip architect and still believe in nonsense anyone can see is false, like astrology. 

Together, this means that when someone is called a genius, you can be in a situation where:

1) Sees a bid being made to make someone high status,

2) For work whose value, you have to take on faith,

3) All the while, you see said genius spouting moronic statements. 

And then it becomes perfectly natural to reject this bid.

Only, there's a bit of an issue with this story. Why exactly are you seeing many inane statements by this supposed-genius? For instance, Newton is widely recognized as a genius, but we don't often hear of his (in retrospect) silly statements on theology. 

The answer is that the moron/genius is (often) a public intellectual. Public intellectuals opine on every topic under the sun. Worse yet, they opine on matters sacred and profane to you and I. Public intellectuals want their views to have influence. Which makes them dangerous. 

Naturally, the sacred takes precedence over mere truth. And recall that the sacred is partly characterized by being ~ every good thing, having no internal contradictions or incoherence. So we see that a public intellectual associated with the sacred gains a halo of virtue that washes away any sin. That silly remark they made? Irrelevant. 

Whereas the profane can never be good. Someone espousing a profane view, why, they must be a moron. Genius is forever out of their reach, no matter whether they've achieved feats no man ever has before. 

* * *

OK, that's about it for this post. But I wanted a little digression at the end on humour. Very often, it becomes impossible for haters to understand that the object of their hate could have been joking. Why is that? 

Well, because humour is composed of two things: status regulation and suprisal. If the object of your hate can regulate other people's status, that confers status on them in turn. But that's impossible, because they're literally The Devil, and how can The Devil be high-status? (Bad example, I know. I blame Milton.) And as for suprisal, that implies a certain level of intelligence. Again, one cannot ascribe a positive quality to the object of blackest hatred. So, hatred kills laughter. 

Somewhat ironic, given how profane humour can be.
# Emergent Introspective Awareness in Large Language Models

New Anthropic research ([tweet](https://x.com/AnthropicAI/status/1983584136972677319), [blog post](https://www.anthropic.com/research/introspection), [paper](https://transformer-circuits.pub/2025/introspection/index.html)): 

> We investigate whether large language models can introspect on their internal states. It is difficult to answer this question through conversation alone, as genuine introspection cannot be distinguished from confabulations. Here, we address this challenge by injecting representations of known concepts into a model’s activations, and measuring the influence of these manipulations on the model’s self-reported states. We find that models can, in certain scenarios, notice the presence of injected concepts and accurately identify them. Models demonstrate some ability to recall prior internal representations and distinguish them from raw text inputs. Strikingly, we find that some models can use their ability to recall prior intentions in order to distinguish their own outputs from artificial prefills. In all these experiments, Claude Opus 4 and 4.1, the most capable models we tested, generally demonstrate the greatest introspective awareness; however, trends across models are complex and sensitive to post-training strategies. Finally, we explore whether models can explicitly control their internal representations, finding that models can modulate their activations when instructed or incentivized to “think about” a concept. Overall, our results indicate that current language models possess some functional introspective awareness of their own internal states. We stress that in today’s models, this capacity is highly unreliable and context-dependent; however, it may continue to develop with further improvements to model capabilities.

See [the Transformer Circuits post](https://transformer-circuits.pub/2025/introspection/index.html) for more. Here's the first figure, explaining one of the main experiments:

![](https://transformer-circuits.pub/2025/introspection/injected-thoughts-blog.png)

Personally, I'm pretty excited by this work! (While I work at Anthropic, I wasn't involved in this research myself beyond commenting on some early drafts.)

I'm still worried about various kinds of confounders and "boring" explanations for the observations here, but overall this was a substantial update for me in favor of recent models having nontrivial subjective experience, and is the best attempt I know of yet to empirically investigate such things. I think the paper does a pretty good job of trying to distinguish "true" introspection from explanations like "it's just steering the model to say words about a topic".
# AISLE discovered three new OpenSSL vulnerabilities

The company post is linked; it seems like an update on where we are with automated cybersec.  
  
*So far in 2025, only four security vulnerabilities received CVE identifiers in OpenSSL, the cryptographic library that secures the majority of internet traffic. AISLE's autonomous system discovered three of them.* ([CVE-2025-9230](https://nvd.nist.gov/vuln/detail/CVE-2025-9230), [CVE-2025-9231](https://nvd.nist.gov/vuln/detail/CVE-2025-9231), and [CVE-2025-9232](https://nvd.nist.gov/vuln/detail/CVE-2025-9232))  
  
Some quick thoughts:

*   OpenSSL is one of the most human-security-audited pieces of open-source code ever, so discovering 3 new vulnerabilities sounds impressive. How much exactly: I'm curious about peoples opinions
*   Obviously, vulnerability discovery is a somewhat symmetric capability, so this also gives us some estimate of the offense side
*   This provides concrete evidence for the huge pool of bugs that are findable and exploitable even by current level AI - this is something everyone sane believed existed in my impression
*   On the other hand, it does not neatly support the story where it's easy for rogue AIs to hack anything. Automated systems can also fix the bugs, hopefully systems like this will be deployed, and it seems likely the defense side will start with large advantage of compute
*   It's plausible that the "programs are proofs" limit is defense-dominated. On the other hand, actual programs are leaky abstractions of the physical world, and it's less clear what the limit is in that case.
# Social media feeds 'misaligned' when viewed through AI safety framework, show researchers

In a [study](https://arxiv.org/abs/2509.14434?ref=foommagazine.org) from September 17 a group of researchers from the University of Michigan, Stanford University, and the Massachusetts Institute of Technology (MIT) showed that one of the most widely-used social media feeds, Twitter/X, owned by the company xAI, is recognizably misaligned with the values of its users, preferentially showing them posts that rank highly for the values of 'stimulation' and 'hedonism' over collective values like 'caring' and 'universal concern.'

*Continue reading at* [*foommagazine.org*](https://www.foommagazine.org/social-media-feeds-misaligned-when-viewed-through-ai-safety-framework-show-resarchers-2/) ...
# Boston Secular Solstice Celebration 2025

As the year comes to an end, the days have grown shorter and the nights have grown longer. Tonight we remember the ancient days of humanity's past, and sing of the future we might hope for. There is uncertainty in both directions, and it will take courage to act under uncertainty. We cannot know for sure what has been, and we cannot say what the future will be. There is this, however:

  
Tomorrow will be brighter than today.  
  
The Solstice celebration will be on December 27th. Doors open at 6:30; event starts at 7:00. Please arrive on time to avoid disrupting the somber and more serious parts of the program. RSVPs are appreciated so we can plan appropriately.  
  
If you have never been to a Secular Solstice in this style before, you should expect alternating speeches and songs, social pressure to join in the singing, and discussion of death in the speeches.  
  
This year we're at Connexion, about a half mile from Sullivan Square Station.
# 2025 NYC Secular Solstice & East Coast Rationalist Megameetup

On December 20th, New York City will have its Secular Solstice. From the evening of December 19th to the morning of the 22nd, NYC will host the East Coast Rationalist Megameetup. You can buy [tickets here](https://rationalistmegameetup.com/checkout).

What's Solstice?

Solstice is a holiday for people comfortable with uncomfortable truths and who believe in good. Secular Solstices take place in many cities around the world, but for us New York City is the best place for it. The first Solstice started here, amid towers that reach for the sky and glitter like stars in the night. Now a tradition spanning over a decade, rationalists from across North America - and sometimes further afield- will come together to sing about humanity's distant past and the future we hope to build.

And what's the megameetup?

For a decade now, New York City has annually hosted aspiring rationalists for a long weekend of skill sharing, socializing, and solsticing. Picture around a hundred people who try to make better decisions and think without bias (or read a lot of LessWrong and its adjacent blogosphere and fiction) in a couple of hotel conference rooms for a weekend, leaving behind only memories and pages of scrawled math equations. Timed to coincide with Secular Solstice, the megameetup is a mixing pot of people who have been attending rationalist meetups regularly for years and people who are meeting in-person rationalists for the first time.

In addition to sleeping space (Friday, Saturday, and Sunday nights) for those from out of town, we'll have:

*   Rationality practice games, both tested and experimental
*   Lightning talks on all manner of subjects
*   Lunch and dinner on Saturday, lunch on Sunday
*   A Secular Solstice afterparty on Saturday night
*   A lot of interesting people to talk to - hopefully including you!
*   *Watch this space for more additions*

We hope to see you there!
# Economics and Transformative AI (by Tom Cunningham)

Excerpt

> **Examples of economic implications from statistical structure.**
> 
> Here are a few brief cases in which the equilibrium economic effect of AI is determined by the underlying statistical structure of the domain. My conjecture is that these types of observations could be formalized in a common framework.
> 
> 1.  **The concentration of the market for AI depends on the dimensionality of the world.** If the world is intrinsically high-dimensional then the returns to model scale will be steadily increasing, and so we should expect high concentration and high markups. If instead the world is intrinsically low-dimensional then the returns to scale will flatten, and there should be low concentration (high competition) and low markups.
> 2.  **The effect of AI on scientific progress depends on the structure of the world.** I give this argument below: if the world has a simple latent structure then progress will be bottlenecked more by intelligence than by data, and so advances in AI will dramatically accelerate scientific progress, without being bottlenecked on more data collection.
> 3.  **The wages paid to an occupation depends on the work’s latent dimensionality.** If the work consists of tasks with high latent dimensionality then the returns to experience and ability will be high, and so wages will be high. As AI changes the incremental effect of human experience and intelligence we should expect it to change the structure of wages.
> 4.  **The demand for compute will depend on the self-similarity of the world.** If 7 billion people all have very different problems then there are few efficiencies we can make in inference (through caching and distillation) and the share of GDP paid to compute will be high. If instead they have similar problems then the returns to additional compute will fall rapidly (demand will be inelastic) and the share of income paid to compute will be small.