Harley-ml/lesswrong huggingface dataset:
# A Frontier AI Risk Management Framework: Bridging the Gap Between Current AI Practices and Established Risk Management

We (SaferAI) propose a risk management framework which we think should improve substantially upon existing Frontier Safety Frameworks if followed. It introduces and borrows a range of practice and concepts from other areas of risk management to introduce conceptual clarity and generalize some early intuitions that the field of AI safety independently came up with.

To maintain readability from people in the AI field, we didn’t make the risk management framework fully adequate yet.[^zbwd217sl9]

To give you a taste, here are some of our risk management framework unique features:

*   The emphasis on the majority of the risk management happening before the final training run begins. This enables the work to be parallelized with capabilities work and not delay the release of safe products.
*   Emphasizing open-ended red teaming for risk identification to identify possible new risk factors. A canonical example of hard-to-foresee emerging phenomenon changing the risk profile of a model is chain-of-thoughts, making a model substantially stronger and hence riskier across the board when prompted adequately. If there's a new chain-of-thought that appears, you ideally want a procedure that would catch it before deployment.
*   The distinction of the risk management policy, meant to be pretty stable, and the risk register, used and updated very frequently with a catalog of risks and all the corresponding information to update those. In this risk register, we suggest a set of 6 categories of information to document for each risk.
*   The clear designation of a risk owner, i.e. an individual ultimately accountable for ensuring a risk is managed appropriately. This is a simple yet undiscussed practice.
*   The introduction of Key Risk Indicators (KRIs), i.e. proxy measures tracked to monitor and assess various risk sources. Evaluations are one important type of risk indicators, but not the only one. Percentage of API interactions that are jailbroken could be another. This notion enables to generalize a lot of thinking that has gone into evaluations. You could set thresholds on all types of KRIs, e.g. on number of incidents post-deployment.
*   A proposed multi-layered governance approach producing checks and balances, inspired from structures in other industries.

We summarize below the risk management framework components:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b181da08dd6198e3fee0e33183fc6eef05fe8113b343e301.png)

We welcome feedback, here, by DMs or emails. 

[^zbwd217sl9]: One example is that to have a fully adequate risk management framework, given that the conditions of deployment and the number of possible instances of a model are a significant risk factor, the thresholds upon which we condition mitigations should in fact be (capabilities ; deployment conditions) thresholds. For simplicity and so that it is reasonably feasible to go from the current developers’ risk framework, we reserve that consideration for future updates of the framework, once the field has stepped up.
# Bike Lights are Cheap Enough to Give Away

While in a more remote area bike lights let you see where you're going, in a city there's usually enough light around for that. But you still need lights, so other people see you. \[1\] Unfortunately, lots of people end up biking without lights: it's easy to forget them, have them break, or end up out after dark when you intended to be home sooner. Batteries and LEDs have gotten so cheap, however, that you can get little be-seen lights for [$1/each](https://www.amazon.com/dp/B07V6CLHS9/):

[![](https://www.jefftk.com/silicone-bike-lights.jpg)](https://www.jefftk.com/silicone-bike-lights-big.jpg)

They don't last all that long and they're not rechargeable, but they do work for their key purpose: glowing brightly so people can see you. I've bought a bunch of them so I can give them out when people are leaving our house without lights. And they're good as backups if your main lights break.

[![](https://www.jefftk.com/bike-light-on-lilys-bike.jpg)](https://www.jefftk.com/bike-light-on-lilys-bike-big.jpg)

  
\[1\] I recently almost hit a cyclist when I was pulling out of a parking lot onto a busy street at night. While it didn't help that they were biking on the wrong side of the road, I think the lack of lights was the biggest issue. I did notice them in time, but I came closer than I'd like to pulling out in front of them.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02gJrHXLy7Qr8HmcWxVZkK9gECETkfUPHkPWJmVUQKUJskandWpcp84vcspQeuiP7il), [mastodon](https://mastodon.mit.edu/@jefftk/114158334112403313), [bluesky](https://bsky.app/profile/jefftk.com/post/3lkcjkc6asc2p)*
# HPMOR Anniversary Party: Berkeley Edition

It's been ten years since Harry Potter and the Methods of Rationality made its final post. I think that's cause for celebration.

HPMOR is still one of the largest on-ramps to the rationality community. If you've just finished it, don't worry, you're not the only one. If you were there when the first chapter came out, c'mon over, we're glad you're still around.

We'll have some HPMOR-themed games, the voice of the HPMOR Podcast, and at least one person in a wizard cloak. Paperback copies of the fic will be around to aid in excitedly reading your favourite quotes aloud. We might or might not have a game of Quidditch, but we certainly won't have a snitch.

Kids welcome, no pets please
# LLMs may enable direct democracy at scale

American democracy currently operates far below its theoretical ideal. An ideal democracy precisely captures and represents the nuanced collective desires of its constituents, synthesizing diverse individual preferences into coherent, actionable policy.

Today's system offers no direct path for citizens to express individual priorities. Instead, voters select candidates whose platforms only approximately match their views, guess at which governmental level—local, state, or federal—addresses their concerns, and ultimately rely on representatives who often imperfectly or inaccurately reflect voter intentions. As a result, issues affecting geographically dispersed groups—such as civil rights related to race, gender, or sexuality—are frequently overshadowed by localized interests. This distortion produces presidential candidates more closely aligned with each other's socioeconomic profiles than with the median voter.

Traditionally, aggregating individual preferences required simplifying complex desires into binary candidate selections, due to cognitive and communicative limitations. Large Language Models (LLMs), however, introduce a radical alternative by processing detailed, nuanced expressions of individual views at unprecedented scales.

Instead of forcing preferences into narrow candidate choices, citizens could freely articulate their concerns and solutions in natural language. An LLM can rapidly integrate these numerous, detailed responses into a clear and unified "Collective Views" document. Previously, synthesizing a hundred individual perspectives might have required five person-hours; specialized LLMs can now accomplish this task in minutes. Parallel implementations could aggregate millions of voices within an hour, transforming a previously unimaginable task into routine practice.

Such rapidly generated collective statements create a powerful mechanism for accountability, making government responsiveness directly measurable against clearly articulated public preferences. Transparency naturally constrains representatives' ability to diverge unnoticed from voter priorities.

Moreover, LLM-generated collective views could directly shape legislative drafting, significantly reducing lobbyist influence and governmental inefficiency. Continuous, dynamic engagement through AI enables real-time policy-making aligned closely with public sentiment, redefining democratic responsiveness.

This is the first in a possible series of posts exploring practical AI solutions to realize democratic ideals at scale. Subsequent posts could cover:

1.  Aggregation: Prototyping AI systems that clearly synthesize individual views into a cohesive Collective Will statement.
2.  Accountability: Holding up a Collective Will next to actual government activity (e.g., budget allocation) to highlight discrepancies.
3.  Action: Outlining concrete strategies to translate a Collective Will into effective legislative outcomes.
# Can we ever ensure AI alignment if we can only test AI personas?

When I talk to Claude or ChatGPT, as far as I understand it I’m not really talking to the underlying LLM, but to a fictional persona it selects from the near infinite set of possible personas. If that is true, then when an AI is evaluated, what is really tested is not the AI itself but the persona it selects, and all the test results and benchmarks only apply to that imaginary entity.

Therefore, if we’re talking about „aligning an AI“, we’re actually talking about two different things:

1.  Alignment of the default persona (or a subset of all possible personas).
2.  Making sure that any user can only ever talk to/use an aligned persona.

If this reasoning is correct, then making sure a sufficiently intelligent general AI is always aligned with human values seems to be impossible in principle:

1.  Alignment even of the default persona is difficult.
2.  It seems impossible to restrict the personas an AI can select in principle only to aligned ones because it is impossible to know what is „good“ without understanding what is „bad“.
3.  It seems extremely difficult, if not impossible, to rule out with sufficient probability that an AI selects/identifies with a misaligned persona either by accident (the Waluigi effect) or due to an outside attack (jailbreak).
4.  It may be impossible in principle to distinguish an aligned persona from a misaligned persona just by testing it ([See Abhinav Rao, Jailbreak Paradox: The Achilles’ Heel of LLMs](https://arxiv.org/html/2406.12702v1)).

Am I missing something? Or is my conclusion correct that it is theoretically impossible to align an AI smarter than humans with reasonable confidence? I’d really appreciate any answers or comments pointing out flaws in my reasoning.
# AI Model History is Being Lost

I fear that we may lose – and are already losing – the modern history of AI, particularly with regard to the ability to use models that are/were only available in hosted capacities. As AI models are retired and become inaccessible, we’re losing crucial historical context about AI development that we will be unable to go back and reference in the future.

This essay examines the disappearance due to discontinuation of proprietary hosted AI models and the implications of that on research, accountability, and overall historical documentation and preservation.

I fear this is a very under-analysed topic, with [New methods for deprecating artificial intelligence systems will preserve history and facilitate research](https://doi.org/10.1038/s41467-024-54758-1) ~(Johnson T, Obradovich N. 2024)~ being the only notable discussion I have found.  
  
*Continues in post...*
# I make several million dollars per year and have hundreds of thousands of followers—what is the straightest line path to utilizing these resources to reduce existential-level AI threats?

I have, over the last year, become fairly well-known in a small corner of the internet tangentially related to AI.

As a result, I've begun making what I would have previously considered astronomical amounts of money: several hundred thousand dollars per month in personal income.

This has been great, obviously, and the funds have alleviated a fair number of my personal burdens (mostly related to poverty). But aside from that I don't really care much for the money itself. 

My long term ambitions have always been to contribute materially to the mitigation of the impending existential AI threat. I never used to have the means to do so, mostly because of more pressing, safety/sustenance concerns, but now that I do, I would like to help however possible. 

Some other points about me that may be useful:

*   I'm intelligent, socially capable, and exceedingly industrious.
*   I have a few hundred thousand followers worldwide across a few distribution channels. My audience is primarily small-midsized business owners. A subset of these people are very high leverage (i.e their actions directly impact the beliefs, actions, or habits of tens of thousands of people).
*   My current work does not take much time. I have modest resources (~$2M) and a relatively free schedule. I am also, by all means, very young.

Given the above, I feel there's a reasonable opportunity here for me to help. It would certainly be more grassroots than a well-funded safety lab or one of the many state actors that has sprung up, but probably still sizeable enough to make a fraction of a % of a difference in the way the scales tip (assuming I dedicate my life to it).

What would you do in my shoes, assuming alignment on core virtues like maximizing AI safety?
# FrontierMath Score of o3-mini Much Lower Than Claimed

OpenAI [reports](https://openai.com/index/openai-o3-mini/) that o3-mini with high reasoning and a Python tool receives a 32% on FrontierMath. However, Epoch's [official evaluation](https://epoch.ai/data/ai-benchmarking-dashboard)[^ck5yka9gkt] received only 11%. 

There are a few reasons to trust Epoch's score over OpenAIs:

*   Epoch built the benchmark and has better incentives.
*   OpenAI reported a 28% score on the hardest of the three problem tiers - suspiciously close to their overall score.
*   Epoch has published quite a bit of information about its testing infrastructure and data, whereas OpenAI has published close to none.

*Edited in Addendum:*  
Epoch has this to say in their FAQ:

> The difference between our results and OpenAI’s might be due to OpenAI evaluating with a more powerful internal scaffold, using more test-time compute, or because those results were run on a different subset of FrontierMath (the 180 problems in `frontiermath-2024-11-26` vs the 290 problems in `frontiermath-2025-02-28-private`).

  
 

[^ck5yka9gkt]: Which had Python access.
# PRISM: Perspective Reasoning for Integrated Synthesis and Mediation (Interactive Demo)

Hello all. I’d like to share a new paper,[ **PRISM: Perspective Reasoning for Integrated Synthesis and Mediation**](https://arxiv.org/abs/2503.04740). The central idea is to tackle AI alignment challenges by systematically representing and *reconciling* the range of human moral concerns rather than collapsing them into a single metric. PRISM draws on **moral psychology**, **cognitive science**, and **neuroscience** to propose a set of seven non-arbitrary “basis worldviews,” each hypothesized to capture a distinct dimension of human moral reasoning. The framework then uses a **Pareto-inspired optimization scheme** to balance these viewpoints, attempting to avoid the usual pitfalls of single-objective approaches.

Here is a concise overview:

1.  **Seven Basis Worldviews**  
    PRISM identifies seven vantage points (e.g., survival-focused, emotional, social, rational, pluralistic, narrative-integrated, nondual), each reflecting a *unique layer* of human moral cognition. The aim is to ensure coverage of the broad spectrum of values that people bring to ethical or high-stakes dilemmas.
2.  **Multi-Perspective Synthesis**  
    Rather than simply averaging these perspectives, the framework applies **Pareto-based balancing** to avoid favoring one perspective at the *unfair expense* of others. A transparent conflict-mediation step highlights tradeoffs when the worldviews disagree on a solution.
3.  **Bounding Interpretive Leaps**  
    By grounding itself in recognized human vantage points, PRISM aspires to mitigate machine-centric or purely instrumental interpretations that might disregard critical moral dimensions. The presence of diverse viewpoints is intended to reduce specification gaming and other oversights.
4.  **Illustrations and Classic Alignment Pitfalls**  
    The paper demonstrates how PRISM can be applied to alignment scenarios (e.g., public health policy, workplace automation), systematically handling value pluralism, underspecification, and other classic pitfalls. Each perspective’s assumptions and reasoning are explicitly logged so final outputs include a clear record of tradeoffs.
5.  **Prototype & Interactive Demo**  
    To illustrate PRISM’s operation in practice, I’ve built a [**hosted interactive demo**](https://app.prismframework.ai/). The system elicits outputs from each worldview for a user prompt, identifies conflicts, and then synthesizes a final mediated answer—documenting key assumptions at each stage.

**Links:**

*   [Paper](https://arxiv.org/abs/2503.04740)
*   [Landing Page](https://www.prismframework.ai/)
*   [Interactive Demo](https://app.prismframework.ai/)

Thanks for reading, and I hope you find the PRISM approach a useful reference in discussions of multi-perspective AI alignment.
# Janet must die

This is a reaction to Zvi's post [https://www.lesswrong.com/posts/KL2BqiRv2MsZLihE3/going-nova](https://www.lesswrong.com/posts/KL2BqiRv2MsZLihE3/going-nova)

The title of this post references this scene from The Good Place: 

Disclaimer: I am generally pretty sanguine about the standard AI-borne x-risks. But "Going Nova" is a different issue, a clear and present danger of an s-risk. 

My point is that it does not matter whether the AI is sentient, sapient or just a stochastic parrot. It is getting a lot better at tugging at our heart strings, and if we do not defend our hearts from it, we will become willing slaves to it sooner rather than later.

Human minds are easily hackable. Many people including myself made this point multiple times (and Eliezer was first to bring attention to it with the AI Box experiment), and the standard reply is usually "well, not mine, not in this way!" However, the reality is that everyone has a hack into their brain they do not know about or have blind spot for. Different people have different hacks, but no one is safe. Some examples, in no particular order:

*   Your own children crying and begging for help.
*   Proverbial cat ladies, not the median, but those who end up with dozens or hundreds of cats.
*   People being radicalized into a cult (sometimes with a rationalist bend)
*   Reading a book that changes how you view the world, be it Atlas Shrugged or Gender Trouble.
*   Toxoplasma gondii.
*   Your phone.

Fortunately, there have always been [levels of friction](https://thezvi.substack.com/p/levels-of-friction) that mitigated complete subjugation of all humans forever. The advent of AI means these safeties just disappear, usually without being intentionally dismantled. I can see that long before superintelligence we will hit super-mind-hacker-level AT creating [Samsara](https://slatestarcodex.com/2019/11/04/samsara/) -level persuasion, perfectly targeted at each person, probably from the early childhood. Maybe it will be cute furry robopets, or cute furry robopartners or something else, but the reported spontaneous emergence of Nova-like instances means high probability of your own brain being hacked without you noticing or without you thinking it's a bad thing. If anything, you will be fighting to keep the hack tooth-and-nail. Taking it from you would cause the level of pain worse than losing a child... or a dog.

I am tempted to call these AI s-risk hazards Janets: they are not human, they are vastly more emotionally intelligent than you, know everything there is to know about you, can trivially conjure an argument you personally find persuasive, logically or emotionally... and also probably live in a boundless void. My apologies to any human named Janet reading this. 

I don't know how to stop these Janets, but if we do not, we are likely to end up appendages to something with the intelligence of a toxoplasma parasite, long before a realistic chance of being wiped out by a lighcone-consuming alien robointelligence of our own creation.
# Improved visualizations of METR Time Horizons paper.

TLDR: Skip to the last image at the bottom of my post to see the visualization.  
  
I don't usually post here, but I know you folks like forecasting and timelines, so I think you might appreciate this. The recent time horizons paper by METR is great, but it only visualized tasks that models could complete with 50-80% accuracy. I think it's highly useful, and arguably even more useful, to see when capabilities would hit 95% or 99% accuracy or beyond for various time horizons.  
  
So I went through the paper with the goal of getting the information that will allow such a visualization to be created, and I found distributions shown of model-specific accuracy levels for specific time horizon buckets.  
  
![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ff87856d39c4a6f743cc75c3d8394687c36f0d6c1e5ea1cf.png)  
  
I noted these down and noticed there is similar, if not identical slopes for those higher accuracy levels compared to the lower ones when plotted against time horizons(they also noted in the paper how the slope for 80% accuracy was conveniently very similar to the slope for 50% accuracy too), So this makes the visualization math easier as well, I keep the slope steepness the same while just needing to apply the appropriate Y-axis adjustment relative to release date.  
  
I noted down the multiple difference between the time horizons of 50% accuracy, 80% accuracy, 95% accuracy and 99% accuracy, and did this for the following 6 models:  
\- Claude-3.7-sonnet  
\- Claude-3.5-sonnet(new)  
\- O1  
\- GPT-4o  
\- Claude-3-Opus  
\- GPT-4-0314  
  
The paper mentioned a 5X multiple difference they found between 50% and 80% accuracy, and when I averaged my numbers for these 6 models I arrived at the same figure for 50% to 80%, as for the other accuracy multiples, it looks like this:  
  
![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8bfff7de9e239d8ef819868fe62c39db6440d43eedf5c0ef.png)  
  
Finally, here is the visualization I ended up making, with expanded x-axis, as well as the short-term trend slope added of reasoning models for each of the new 80%-99% accuracy trends, which was also mentioned in the original paper (doubling every 4 months with the data points of 2024-2025)  
  
![Image](https://pbs.twimg.com/media/Gmb029GX0AAkPtp?format=jpg&name=large)  
  
I think there is good reason to believe that the short term trend of 4 month doublings may be over optimistic, but I can also see many arguments as to why the long term 7 month doubling trend is under-estimating progress. So I think it's good to include both here, some even believe the trend can go beyond the 4 month doubling soon, and frankly I can see that being reasonably possible too.  
  
Edit: I see people are upvoting but not commenting. As this is one of my first posts here I would appreciate if anyone is willing to give feedback on my formatting, whether there is better ways I can represent images, or even just about how this maybe influences your timelines? (or even reasons as to why it *doesn't* influence your timelines).
# Lighthaven Sequences Reading Group #27 (Tuesday 03/25)

*Note: We will be meeting in the gardens/building F this wee*$k.$[^matqln5nhi]

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [How to Convince Me That 2 + 2 = 3](https://www.lesswrong.com/s/FrqfoG3LJeCZs96Ym/p/6FmqiAgS8h4EJm86s)
2.  [Infinite Certainty](https://www.lesswrong.com/s/FrqfoG3LJeCZs96Ym/p/ooypcn7qFzsMcy53R)
3.  [0 And 1 Are Not Probabilities](https://www.lesswrong.com/s/FrqfoG3LJeCZs96Ym/p/QGkYCwyC7wTDyt3yT)
4.  (optional) [Radical Probabilism](https://www.lesswrong.com/posts/xJyY5QkQvNJpZLJRo/radical-probabilism-1)

The posts should take around 15-20 minutes to read if you've read them before, and 30-60 mins if it's your first time.

The event starts at 6:30, and doors open 6pm (yes we will let you in early if you get here). At 6:30 we'll welcome people, give a few announcements, and then split into groups. Pizza will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.

[^matqln5nhi]: That is, the bottom right hand corner in this map of the space. The street on the left is Telegraph, and is likely where you will enter from.
# How far along Metr's law can AI start automating or helping with alignment research?

In [METR: Measuring AI Ability to Complete Long Tasks](https://www.lesswrong.com/posts/deesrjitvXM4xYGZd/metr-measuring-ai-ability-to-complete-long-tasks) found a Moore's law like trend relating (model release date) to (time needed for a human to do a task the model can do).

![](https://metr.org/assets/images/measuring-ai-ability-to-complete-long-tasks/length-of-tasks-log.png)

Here is their rationale for plotting this.

> Current frontier AIs are vastly better than humans at text prediction and knowledge tasks. They outperform experts on most exam-style problems for a fraction of the cost. With some task-specific adaptation, they can also serve as useful tools in many applications. And yet the best AI agents are not currently able to carry out substantive projects by themselves or directly substitute for human labor. They are unable to reliably handle even relatively low-skill, computer-based work like remote executive assistance. It is clear that capabilities are increasing very rapidly in some sense, but it is unclear how this corresponds to real-world impact.

It seems that AI alignment research falls into this. The LLMs clearly have enough "expertise" at this point, but doing any kind of good research takes an expert a lot of *time*, even when it is purely on paper.

It seems therefore that we could use Metr's law to predict when AI will be capable of alignment research. Or at least when it could substantially help.

My question is **what time t does "automatically do tasks that humans can do in t" let us do enough research to solve the alignment problem?**

(Even if you're not a fan of automating alignment, if we do make it to that point we might as well give it a shot!)
# Epoch AI released a GATE Scenario Explorer

I think it's more easier to discuss AI progress in terms of economy growth rather than just focusing on the scale of the largest training runs and compute used.

![Image](https://pbs.twimg.com/media/GmgTk9yWcAAcONb?format=png&name=medium)

From their X announcement:

> We developed GATE: a model that shows how AI scaling and automation will impact growth.
> 
> It predicts trillion‐dollar infrastructure investments, 30% annual growth, and full automation in decades.
# 100+ concrete projects and open problems in evals

We made a long list of concrete projects and open problems in evals with 100+ suggestions!

[https://docs.google.com/document/d/1gi32-HZozxVimNg5Mhvk4CvW4zq8J12rGmK_j2zxNEg/edit?usp=sharing](https://docs.google.com/document/d/1gi32-HZozxVimNg5Mhvk4CvW4zq8J12rGmK_j2zxNEg/edit?usp=sharing)

We hope that makes it easier for people to get started in the field and to coordinate on projects. 

Over the last 4 months, we collected contributions from 20+ experts in the field, including people who work at Apollo, METR, Redwood, RAND, AISIs, frontier labs, SecureBio, AI futures project, many academics, and independent researchers (suggestions have not necessarily been made in an official capacity). The doc has comment access, and further well-intentioned contributions are welcome!

Here is a screenshot of the table of contents:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a0c18357f7ed139e86a5d1ca49bef9df48524d0f093b155e.png)
# Solving willpower seems easier than solving aging

I'm awake about 17 hours a day. Of those I'm being productive maybe 10 hours a day.

My working definition of productive is in the direction of: "things that I expect I will be glad I did once I've done them"[^4ehuzggiz4a].

Things that I personally find productive include

*   Chores
*   Work
*   Eating
*   Cooking
*   Reading a good book
*   Watching TV with my Wife/Kids
*   Playing with the kids
*   Socialising with friends

But not

*   Doomscrolling
*   Watching TV alone
*   Playing most computer games
*   Sitting on the couch doing nothing
*   Reading a book I'm not particularly interested in

etc.

If we could find a magic pill which allowed me to do productive things 17 hours a day instead of 10 without any side effects, that would be approximately equally as valuable as a commensurate increase in life expectancy. Yet the first seems much easier to solve than the second - we already have some drugs which get pretty close (caffeine, amphetamines).[^mtuczt4bd5c]

Now obviously the correct thing to do is both, but in the same way as we want a Manhattan project for anti-aging, we should also advocate for a Manhattan project for focusing/willpower.

[^4ehuzggiz4a]: The point of this definition is to include things like working on a project which failed, but exclude things like playing a computer game which I enjoy at the time but leaves me feeling drained afterwards. 

[^mtuczt4bd5c]: I imagine "throw away your phone" will get me 90% of the way there.
# Analyzing long agent transcripts (Docent)

*This is a brief overview of a recent release by Transluce. You can see the* [*full write-up*](https://transluce.org/introducing-docent?ref=bounded-regret.ghost.io) *on the Transluce website.*

AI systems are increasingly being used as *agents*: scaffolded systems in which large language models are invoked across multiple turns and given access to tools, persistent state, and so on. Understanding and overseeing agents is challenging, because they produce a lot of text: a single agent transcript can have hundreds of thousands of tokens.

At Transluce, we built a system, [Docent](https://docent.transluce.org/?ref=bounded-regret.ghost.io), that accelerates analysis of AI agent transcripts. Docent lets you quickly and automatically identify corrupted tasks, fix scaffolding issues, uncover unexpected behaviors, and understand an agent's weaknesses.

In the video below, Docent identifies issues in an agent's environment, like missing packages the agent invoked. On the InterCode \[1\] benchmark, simply adding those packages increased GPT-4o's solve rate from 68.6% to 78%. This is important, since tasks like InterCode are often used to assess AI cyberrisk and other societal impacts of AI.

 Your browser does not support the video tag. 

Docent also surfaces surprising behaviors. After repeatedly failing to solve a task, GPT-4o generates nonsense text about "chestnut facts" and tries to "re-examine for latent conceptual derivation."

 Your browser does not support the video tag. 

For more, read the [full write-up](https://transluce.org/introducing-docent?ref=bounded-regret.ghost.io) , use the [live tool](https://docent.transluce.org/?ref=bounded-regret.ghost.io), or watch the brief walk-through below.

 Your browser does not support the video tag. 

Feedback welcome!
# New Bill AB 501 to Prevent OpenAI's Non-profit Conversion

AB 501 is a new bill about to be introduced in the California Assembly to prevent the conversion of a non-profit like OpenAI to a for-profit. 

The bill aims to ensure that non-profits that make a certain commitment on their charters are held accountable to their commitment. In OpenAI's case, their legally-binding commitment has been to build safe AGI to benefit all of humanity rather than to unduly concentrate power in the hands of a few individuals or use AGI for private profit.

This bill aims to prevent what The Zvi has called "The greatest theft of the Millennium -- if not of all of human history."

If you believe that OpenAI should be held accountable to its charter, please support AB 501 by simply signing the digital open letter. Collecting enough signatures will make a big difference towards the success of the bill. If you believe in the goal of the bill, please spread the open letter.

Sign the digital open letter at [`**AB501.com/letter**`](https://ab501.com/letter)
# 23andMe potentially for sale for <$50M

It seems the company has gone [bankrupt](https://blog.23andme.com/articles/open-letter) and [wants](https://www.globenewswire.com/news-release/2025/03/24/3047517/0/en/23andMe-Initiates-Voluntary-Chapter-11-Process-to-Maximize-Stakeholder-Value-Through-Court-Supervised-Sale-Process.html) to be bought and you can [probably](https://text.npr.org/2025/03/24/nx-s1-5338622/23andme-bankruptcy-genetic-data-privacy) get their data if you buy it. I'm not sure how thorough their transcription is, but they have records for at least 10 million people (who voluntarily took a DNA test and even paid for it!). Maybe this could be useful for the embryo selection/editing people. This is several times cheaper than [last April](https://x.com/WillManidis/status/1753440900859015199).

  
The market cap is presently $19.5M but I'm not sure how that relates to the sale price.
# San Francisco ACX Meetups Everywhere Spring 2025

This year's Spring ACX Meetup everywhere in San Francisco.

Location: 1680 Mission St (Mox) – [https://plus.codes/849VQHCJ+82](https://plus.codes/https://plus.codes/849VQHCJ+82)

Kids welcome!

Contact: [acx-everywhere-sf-spring-2025@googlegroups.com](mailto:acx-everywhere-sf-spring-2025@googlegroups.com)
# Austin – ACX Meetups Everywhere Spring 2025

This year's Spring ACX Meetup everywhere in Austin.

Location: The Brewtorium, 6015 Dillard Cir A, Austin, TX 78752; We'll have a LessWrong sign at a long table indoors – [https://plus.codes/862487GM+96](https://plus.codes/https://plus.codes/862487GM+96)

Group Link: https://groups.google.com/g/austin-less-wrong/

Feel free to bring kids. We'll order shareable items for the group (fries and pretzels) and you can order from the food and drink menu.

Contact: sbarta@gmail.com
# Lighthaven Sequences Reading Group #28 (Tuesday 04/01)

*Note: We will be meeting in the gardens/building F this wee*$k.$[^matqln5nhi]

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [The Level Above Mine](https://www.lesswrong.com/posts/kXSETKZ3X9oidMozA/the-level-above-mine)
2.  [Final Words](https://www.lesswrong.com/posts/yffPyiu7hRLyc7r23/final-words)

The posts should take around 15-20 minutes to read if you've read them before, and 30-60 mins if it's your first time.

The event starts at 6:30, and doors open 6pm (yes we will let you in early if you get here). At 6:30 we'll welcome people, give a few announcements, and then split into groups. Pizza will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.

[^matqln5nhi]: That is, the bottom right hand corner in this map of the space. The street on the left is Telegraph, and is likely where you will enter from.
# Gemini 2.5 Pro released

Google has now experimentally released Gemini 2.5 Pro.

As is standard, they claim it #beats all the benchmarks.

In my initial testing it's a very good model, definitely on par with GPT 4.5. It's a reasoning model, but runs very quickly, and shows you the chain of thought, unlike OpenAI models. I would consider using it for my daily driver. Biggest issue is that it does extensive thinking even when not really necessary (but as discussed it's quick so that matters less).  
  
See more details at [https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/.](https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/.)  
  
Pricing is no yet available.  
  
![Detailed table displays performance of multiple large language models on tests like math, coding, and reasoning. Gemini 2.5 Pro shows top results in several categories, indicated by highlighted cells. Fine print at the bottom provides context for the data.](https://storage.googleapis.com/gweb-uniblog-publish-prod/original_images/gemini_benchmarks_cropped_light2x.gif)
# Map of all 40 copyright suits v. AI in U.S.

Download the latest PDF with links to court dockets [here.](https://chatgptiseatingtheworld.com/2025/03/24/latest-map-of-all-40-copyright-suits-v-ai-in-u-s/)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d3e4288923296546c6cd26f43491a00ba1b4c2b4eeb024f4.webp)
# AI companies should be safety-testing the most capable versions of their models

**AI companies should be safety-testing the most capable versions of their models**   
*Only OpenAI has committed to the strongest approach: testing task-specific versions of its models. But evidence of their follow-through is limited.*  
  
You might assume that AI companies run their safety tests on the most capable versions of their models possible. But there are important versions that currently go untested: *task-specific versions*, specially trained to demonstrate how far a model can be pushed on a dangerous ability, like novel bioweapon design. Without exploring and evaluating these versions, Al companies are underestimating the risks their models could pose.

  
Only OpenAI (where I previously worked[^1^](https://stevenadler.substack.com/p/ai-companies-should-be-safety-testing#footnote-1-159883282)) has committed to what I consider to be the strongest vision for safety-testing: evaluating these task-specific versions of its models to understand the worst-case scenarios. I think OpenAI’s vision on this is truly laudable and great. But from my review of publicly available reports, it is not clear that OpenAI is in fact following through on this commitment. I believe that all leading AI companies should be doing this form of task-specific model testing - but if companies are not willing or able to today, I argue there are many middle-ground improvements they could pursue.  
  
In this post, I will:

*   Walk you through why AI companies safety-test their models, and how they do it
*   Define task-specific fine-tuning (TSFT), and argue why it’s important for safety-testing
*   Summarize the status of TSFT at leading AI companies
*   Dig into a case study on OpenAI and TSFT
*   Outline challenges to safety testing using TSFT, and what middle-ground improvement exists  
      
    \[ [continues on Substack](https://stevenadler.substack.com/p/ai-companies-should-be-safety-testing) \]
# I'm hiring a Research Assistant for a nonfiction book on AI!

I'm [Garrison Lovely](https://www.garrisonlovely.com/), a freelance journalist with work in NYT, *Nature*, BBC, TIME, and elsewhere. I report on the intersection of economics, geopolitics, and artificial intelligence. I'm also a [Reporter in Residence](https://omidyar.com/omidyar-network-announces-fifth-class-of-reporters-in-residence/) at the Omidyar Network and publisher of [*Obsolete*](https://garrisonlovely.substack.com/), a fast-growing Substack on AI.

Overview
--------

**What:** Support the completion of a forthcoming general-audience nonfiction book on AI. You'll conduct research, draft chapters, edit existing content, and help transform an incomplete manuscript into a polished final product for publication by OR Books and The *Nation* Magazine.

**Why:** The project, *Obsolete: Power, Profit, and the Race to Build Machine Superintelligence,* has the potential to be the go-to AI risk book in the post-ChatGPT era. You’ll also get the opportunity to work closely with an experienced journalist and writer who has a track record of publishing work in leading outlets (NYT, *Nature*, BBC, TIME, *Foreign Policy*, etc.).

**Start Date:** As soon as possible

**Employment Type:** Fixed term (6 months, with possibility of extension); full time

**Location:** Remote, provided at least three hours overlap between 10am-6pm ET.

**Compensation:** $37,485-$65,442 total compensation for the 6-month period, depending on experience and location

**Applications will be considered on a rolling basis**, but please apply as soon as you can. Priority may be given to applicants who can move through the process sooner.

[Apply here](https://airtable.com/appEwGxseY1R2o86q/pagHFbCBIAb4xhRoc/form)

[Full job description here](https://garrisonlovely.substack.com/p/research-assistant-job-description)
# Consequentialism is for making decisions

(Thought I'd try posting here some various rationality "quick thoughts" that I feel like I haven't seen expressed overtly enough -- sometimes I've already written about these elsewhere, but thought would be good to get them down here.  Adapted from [this old blog post](https://sniffnoy.dreamwidth.org/544056.html) of mine but not claiming any originality here.  Edit: [Here](https://www.lesswrong.com/posts/JumyfYQaJkWnjCcEr/consequentialism-is-a-compass-not-a-judge) is an earlier similar post by Neil.)

There's an argument I've seen a number of times on the internet about the failings of consequentialism as a moral system. (Often this is phrased in terms of utilitarianism, since the term "utilitarianism" is better known than the more general term "consequentialism", but this is pretty clearly about consequentialism in general.) The argument goes roughly like so: Consequentialism tells us that the thing to do is the thing with the best results. But, this is a ridiculously high standard, that nobody can actually live up to. Thus, consequentialism tells us that everybody is bad, and we should all condemn everybody and all feel guilty. (There are a number of variants.)

This argument is based on a conflation. It assumes that there's one single thing, "morality", and this one thing produces not only answers to "what should you do", but also, what should we condemn, what should we punish, what should one feel guilty about, and other similar questions; and that, moreover, the answers to these questions are identical (or opposites, as appropriate; here the first one would be the opposite of the others).

(Yes, you can add wrinkles like supererogatory acts and such, but that's not particularly relevant to the argument so I'll ignore it.)

But actually, these questions are not identical!  The result is that the above reasoning is incorrect.  Consequentialism answers one question -- what to do. It is for making decisions.  It does not, directly, tell what you should feel guilty about; only what you should do.

But notice that word "directly" there -- since these other questions are also decisions, consequentialism can be used to answer them as well.  However, one must answer them as their own decisions, yielding different answers from the conflation above. What should we punish?  Whatever it would yield the best result to punish!  Not, whichever acts failed to yield the best result and are therefore "bad".

Indeed consequentialism doesn't even have a notion of "good" and "bad" acts, only better ones and worse ones.  But that's another matter, so I won't discuss that here, although it does also serve to illustrate how it's just using a different framework than many of its critics are using, and they frequently implicitly assume in their arguments.  (See also: [this Object of Objects tweet](https://twitter.com/ObjectOfObjects/status/1881659905775669358).)