Harley-ml/lesswrong huggingface dataset:
# How to Host a Find Me Party

For my birthday this year, i invited friends to a party on Saturday but didn't tell them the location. I gave a list of questions they could ask me. Whoever finds me in the fewest questions wins a prize. 

Each friend showed up that day excited to tell me about their detective work & misadventures in tracking me down. Everyone had a lot of fun! 

I definitely recommend this as a party idea or meetup activity. It was much easier to organize than i expected.

Advice:

*   Use this question list:
    *   I'm at <GPS LOCATION>. Are you within <DISTANCE> of me?
    *   Send me a photo of you.
    *   Send me a photo of a tree.
    *   Send me a photo of a street.
    *   Send me a photo with your camera pointed straight up.
*   You could also add these questions:
    *   Send me a photo of a restaurant.
    *   Send me a photo of the tallest thing around.
    *   Send me a photo of the oddest thing around.
    *   Send me a photo of your favorite thing around.
*   Tell guests when exactly the game starts. I told guests we would play 3 rounds, at 1000, 1300, & 1600 respectively.
*   Set reminders for yourself so that you have enough time to travel to the next round's location. You will be distracted.
*   Don't let guest expectations get too high about the party aspect. If you plan to be simply sitting on a park bench, tell them to expect a chill hangout after the search. The search is the main focus. Also note that your attention will be divided between guests that have arrived quickly and your phone, as you answer questions from slower-arriving guests.
*   Tell guests what to wear. I said: Dress for walking around outdoors and/or casual indoor places.
*   Tell guests whether they can bring their children and/or dogs to the secret location.
*   Ideally, your secret location should:
    *   Have a restroom
    *   Be somewhere guests can step into without paying money or feeling awkward
*   My secret locations, for example, were:
    *   A coffeeshop next to a bookstore
    *   Outdoor seating at a restaurant (that suited multiple diets)
    *   A park with a good view
*   Encourage guests to take public transit, rideshare, or carpool, because solo drivers will find themselves spending a lot of time looking for places to park between every question, which is rather tedious. (Or they will text while driving, which is dangerous.)
*   It is okay to crop your photos if details in the corners would end the search too quickly. A determined guest is very good at finding information in a photo!
*   Conversely, if a guest isn't having fun, encourage them to ask lots of questions, or give hints.

(I came up with these rules by drastically simplifying the show Jet Lag.)

I recommend this!
# The V&V method - A step towards safer AGI

Originally posted on my blog, 24 Jun 2025 - see also full [PDF (26 pp)](https://blog.foretellix.com/wp-content/uploads/2025/06/agi_vv_method.pdf)

Abstract
--------

The **V&V method** is a concrete, practical framework which can complement several alignment approaches. Instead of asking a nascent AGI to “do X,” we instruct it to **design and rigorously verify a bounded “machine-for-X”.** The machine (e.g. an Autonomous Vehicle or a “machine” for curing cancer) is prohibited from uncontrolled self-improvement: Every new version must re-enter the same verification loop. Borrowing from safety-critical industries, the loop couples large-scale, scenario-based simulation with coverage metrics and a safety case that humans can audit. Human operators—supported by transparent evidence—retain veto power over deployment.   
  
The method proceeds according to the following diagram:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/68871adba55d4bff57ead150b5ed34c084679a71f50a566c.png)

This method is **not a silver bullet**: it still relies on aligned obedience, incentive structures that deter multi-agent collusion, and tolerates only partial assurance under tight competitive timelines. **Yet it complements** existing scalable-oversight proposals—bolstering **Constitutional AI, IDA and CAIS** with holistic system checks—and offers a practical migration path because industries already practice V&V (Verification and Validation) for complex AI today. In the critical years when AGI first appears, the V&V method could buy humanity time, reduce specification gaming, and focus research attention on the remaining hard gaps such as strategic deception and corrigibility.

TL;DR
-----

*   **Ask AGI to build & verify** a task-specific, *non-self-improving* machine
*   **Scenario-based, coverage-driven simulations + safety case** give humans transparent evidence
*   **Main contributions:** Complements Constitutional AI, improves human efficiency in Scalable Oversight, handles "simpler” reward hacking systematically, bolsters IDA and CAIS
*   **Works today** in Autonomous Vehicles. Scales with smarter “AI workers”
*   **Limits:** Needs aligned obedience & anti-collusion incentives

About the author
----------------

Yoav Hollander led the creation of three V&V standards (IEEE 1647 “e”, Accellera PSS, ASAM OpenSCENARIO DSL) and is now trying to connect some of these ideas with AGI safety.
# Apply to be a mentor in SPAR!

Mentor applications are now open for the Fall 2025 round of the [Supervised Program for Alignment Research (SPAR)](https://sparai.org/?utm_source=eaf-lw&utm_medium=forum&utm_campaign=forum-mentor-app), running from September 15 to December 20, 2025.

Apply as a mentor

SPAR is a remote-first part-time program that connects mentors and mentees for three-month AI safety and governance research projects. **Mentor apps are due July 20, and applications for mentees will run from July 27 to August 20.** If you’re interested in participating as a mentee, you can express your interest [here](https://forms.kairos-project.org/spar-s25-interest?utm_source=eaf-lw&utm_medium=forum&utm_campaign=forum-mentor-app), and we’ll reach out to you once applications are open.

You can find out more about the program [here](https://sparai.org/?utm_source=eaf-lw&utm_medium=forum&utm_campaign=forum-mentor-app). SPAR is run by [Kairos](https://forum.effectivealtruism.org/posts/tTCyWthLyHk32WX9n/introducing-kairos-a-new-home-for-spar-and-fsp?utm_source=eaf-lw&utm_medium=forum&utm_campaign=forum-mentor-app), an AI safety fieldbuilding organization.

Should I apply as a mentor?
---------------------------

You might be a good fit to be a SPAR mentor if you are a graduate student, academic, full-time AI safety researcher, independent researcher, or have prior full-time relevant research experience (e.g., MATS, Astra, GovAI fellow, etc.). We’re interested in projects that cover technical AI safety, AI policy and governance, AI strategy, AI security, or societal impacts of transformative AI, and we are able to provide funding for compute costs. We don't require mentors to have previous experience providing research mentorship, and new mentors will receive guidance on this.

Regarding dedication, we expect most mentors will dedicate 2 to 15 hours a week, depending on how many mentees they’d like to take in and how much supervision they’re interested in providing. Mentors can decide whether to run the project or not based on the applications they get, making applications zero-commitment until one chooses to accept any mentees.
# Multispecies Metagenomic Calibration

*Cross-posted from my [NAO Notebook](https://data.securebio.org/jefftk-notebook/multispecies-calibration).*

*This is something I wrote internally in late-2022. Sharing it now with light edits, additional context, and updated links after the idea came up at the [Microbiology of the Built Environment](https://www.grc.org/microbiology-of-the-built-environment-conference/) conference I'm attending this week.*

Metagenomic sequencing data is fundamentally relative: each observation is a fraction of all the observations in a sample. If you want to make quantitative observations, however, like understanding whether there's been an increase in the number of people with some infection, you need to calibrate these observations. For example, there could be variation between samples due to variation in:

*   Changes in how many humans are contributing to a sample.
*   Has it been raining? (Especially in areas with combined sewers, but also a factor with nominally separated sewers.)
*   What have people been eating lately?
*   What temperature has it been?
*   How concentrated is the sample?
*   etc

If you're trying to understand growth patterns all of this is noise; can we reverse this variation? I'm using "calibration" to refer to this process of going from raw per-sample pathogen read counts to estimates of how much of each pathogen was originally shed into sewage.

The simplest option is not to do any calibration, and just consider raw relative abundance: counts relative to the total number of reads in the sample. For example, this is what [Marc Johnson and Dave O'Connor](https://lungfish-science.github.io/wastewater-dashboard/dashboard2.html) are doing.

It seems like you ought to be able to do better if you normalize by the number of reads matching some other species humans excrete. It's common to use [PMMoV](https://en.wikipedia.org/wiki/Pepper_mild_mottle_virus) for this: peppers are commonly infected with PMMoV, people eat peppers, people excrete PMMoV. All else being equal, the amount of PMMoV in a sample should be proportional to the human contribution to the sample. This is especially common in PCR work, where you take a PCR measurement of your target, and then present it relative to a PCR measurement of PMMoV. For example, this is what [WastewaterSCAN](https://data.wastewaterscan.org/) does.

Because the [NAO](https://naobservatory.org/) is doing very deep metagenomic sequencing, around 1B read pairs (300Gbp) per sample, we ought to be able to calibrate against many species at once. PMMoV is commonly excreted, but so are other tobamoviruses, crAssphage, other human gut bacteriophages, human gut bacteria, etc. We pick up thousands of other species, and should be able to combine those measurements to get a much less noisy measurement of the human contribution to a sample.

This isn't something the NAO has been able to look into yet, but I still think it's quite promising.

*Comment via: [substack](https://jefftkaufman.substack.com/p/multispecies-metagenomic-calibration)*
# [Paper] Ambiguous Online Learning

**Abstract:** *We propose a new variant of online learning that we call "ambiguous online learning". In this setting, the learner is allowed to produce multiple predicted labels. Such an "ambiguous prediction" is considered correct when at least one of the labels is correct, and none of the labels are "predictably wrong". The definition of "predictably wrong" comes from a hypothesis class in which hypotheses are also multi-valued. Thus, a prediction is "predictably wrong" if it's not allowed by the (unknown) true hypothesis. In particular, this setting is natural in the context of multivalued dynamical systems, recommendation algorithms and lossless compression. It is also strongly related to so-called "apple tasting". We show that in this setting, there is a trichotomy of mistake bounds: up to logarithmic factors, any hypothesis class has an optimal mistake bound of either *$\Theta(1)$*, *$\Theta(\sqrt(N))$* or *$N$*.*

This work is my first rigorous foray into [compositional learning theory](https://www.alignmentforum.org/posts/ZwshvqiqCvXPsZEct/the-learning-theoretic-agenda-status-2023#Direction_1__Frugal_Compositional_Languages). Intuitively, one approach to learning compositionally is, constructing a model incrementally, adding one part each time. This raises the question of what does it mean to have a "partial" model. One notion of partial model is provided by infra-Bayesianism / [robust RL](https://arxiv.org/abs/2504.06820). However, there the treatment of ambiguities in the model via the maximin decision rule is strongly tied to the choice of reward function. Intuitively, we might expect the process of incremental learning to not depend on the reward function that strongly.

In this work, I propose a notion of "partial" model in the online learning setting. I develop a theory of learning for this notion analogous (but more involved than) classical realizable Littlestone theory.

Does it truly lead to compositional learning? In results that don't appear in the paper, I showed that the "ambiguous Littlestone dimension" (an invariant I discovered that characterizes the complexity of ambiguous online learning) indeed behaves nicely w.r.t. some ways to decompose hypothesis classes. However, my attempts to use this framework to get *computationally efficient* compositional learning algorithms haven't produced much so far.

Moreover, our results on robust RL (in particular the notion of "imprecise estimator" we have there) suggests it's more natural to study a different (easier) version of online learning with ambiguity, where the learner is only allowed to predict one label instead of a set. So, the ambiguous online learning framework is a potentially useful tool, but I'm currently more optimistic about other approaches to compositional learning theory (stay tuned!).
# If Not Now, When?

> Hillel... would say: "If I am not for me, who will be for me? And If I am just for myself, what am I? And if not now, when?"

[*Pirkei Avot 1,14*](https://www.sefaria.org.il/Pirkei_Avot.1.14?lang=he&with=all&lang2=he)

That last question has usually been interpreted as a rhetorical one. If you're not going to do it now, you'll never get around to it!

But this is clearly a bad principle to live by. If you start cooking too early, the food will be cold when you want to eat it. If you pick a fruit before it's ripe it will be sour. When you are exhausted the quality of your work goes down. And when you have many things to do, you have to prioritise which comes first.

> There is a season and a time for everything...

[*Ecclesiastes 3,1*](https://www.sefaria.org.il/Ecclesiastes.3.1?lang=he&with=all&lang2=he)

Perhaps we would do better interpreting it literally. It's fine to push things off, but first you must answer the question: "When?". At what point in time, or under which change in circumstances, will you take this action? If you have a good answer to this, all is fine, but if you don't it's likely you are just pointlessly procrastinating, and the job will never be done.
# Help the AI 2027 team make an online AGI wargame

The AI Futures Project, the team behind [AI 2027](https://ai-2027.com/), has developed a [tabletop exercise](https://ai-2027.com/about?tab=tabletop-exercise#tab-box-tabletop-exercise) (TTX) to simulate AI takeoff where we facilitate people going through an AI 2027-like scenario with different people taking different roles, like POTUS, China, or the frontier AI company CEO. Reviews have been positive and people have [found it useful](https://stevenadler.substack.com/p/a-crisis-simulation-changed-how-i) for building their own models of how AI will go.

We’re interested in developing an online version of this game, with the goal of eventually getting millions of people to play through takeoff and better understand the dynamics and risks involved. The other game characters could be driven by LLMs so we don’t need to facilitate it. The online TTX could be based on our scenario and background research alongside LLMs and various game elements. 

To this end, we’re looking for a **full-stack developer with game design and UX experience/interest and a deep understanding of AGI risk.** We previously worked with the Lightcone Infrastructure team for the AI 2027 website and might work with them again for this project, but first we’d like to see if anyone else is interested.

We expect that the ideal online TTX would ultimately look and feel very different from the in-person experience, but would successfully convey similar content. 

If you’re interested, **please fill in **[**this quick form**](https://airtable.com/appmcCGBz3vqARP9n/page6tXxhBJIoMCAq/form), message me, or comment below, we’d love to hear from you!
# Misconceptions on Affordable Housing

People often think 'affordable' housing is much more expensive than it actually is, and then conclude it's a scam to make housing for rich people. But this is often based on a misunderstanding of how the prices are set.

Let's say a unit is "50% AMI" somewhere with an area median income (AMI) of $100k. You might think, and I've seen a bunch of people with this confusion, that units would rent for 50% of $100k: $50k/y ($4,170/month), but it's much cheaper than that.

Affordable has a very specific meaning in this context: spending no more than 1/3 of your income on housing. So you might think 50% AMI is 'affordable' to someone earning at the 50th percentile: 1/3 of $100k, or $33k/y ($2,750/month), but it's cheaper than that, too!

Instead, 50% AMI means someone earning 50% of the AMI would be spending 1/3 of their income on housing. This is 50% of 1/3 of the AMI, and in this case that would be $17k/y ($1,390/month).

Now, I don't think affordable housing solves everything: if it were widespread I think it [would be exploited](https://www.jefftk.com/p/affordable-housing-workarounds) and it can be a distraction from just getting a lot of units built, but the term "affordable housing" is actually a decent operationalization of whether housing is something regular people can afford.

*Comment via: [mastodon](https://mastodon.mit.edu/@jefftk/114752977640543529), [bluesky](https://bsky.app/profile/jefftk.com/post/3lsklxeit4c2o), [substack](https://jefftkaufman.substack.com/p/misconceptions-on-affordable-housing)*
# [Paper] Stochastic Parameter Decomposition

Abstract
========

A key step in reverse engineering neural networks is to decompose them into simpler parts that can be studied in relative isolation. 

Linear parameter decomposition— a framework that has been proposed to resolve several issues with current decomposition methods—decomposes neural network parameters into a sum of sparsely used vectors in parameter space. 

However, the current main method in this framework, [Attribution-based Parameter Decomposition](https://www.lesswrong.com/posts/EPefYWjuHNcNH4C7E/attribution-based-parameter-decomposition) (APD), is impractical on account of its computational cost and sensitivity to hyperparameters. 

In this work, we introduce *Stochastic Parameter Decomposition* (SPD), a method that is more scalable and robust to hyperparameters than APD, which we demonstrate by decomposing models that are slightly larger and more complex than was possible to decompose with APD. 

We also show that SPD avoids other issues, such as shrinkage of the learned parameters, and better identifies ground truth mechanisms in toy models. 

By bridging causal mediation analysis and network decomposition methods, this demonstration opens up new research possibilities in mechanistic interpretability by removing barriers to scaling linear parameter decomposition methods to larger models. 

We release a library for running SPD and reproducing our experiments at [https://github.com/goodfire-ai/spd](https://github.com/goodfire-ai/spd).  
 

**Links:**

*   Paper: [https://arxiv.org/abs/2506.20790](https://arxiv.org/abs/2506.20790)
*   Code: [https://github.com/goodfire-ai/spd](https://github.com/goodfire-ai/spd)
*   Tweet thread: [https://x.com/leedsharkey/status/1938616685855941040](https://x.com/leedsharkey/status/1938616685855941040)
# The next wave of model improvements will be due to data quality

When talking about why the models improve, people frequently focus on algorithmic improvements and on hardware improvements. This leaves out improvements in data quality.

DeepResearch provides a huge quantity of high quality analysis reports. GPT5 will almost certainly be trained on all the DeepResearch requests (where there are no reason to believe they are wrong like bad user feedback) of users that haven't opted out of their data being used for training. This means that when users ask GPT5 questions where no human has written an anlaysis of the question, GPT5 might still get facts right because of past DeepResearch reports.

This means that the big companies who do have a massive amount of users that produce a massive amount of DeepResearch requests will have a leg-up that's hard to copy for smaller players who don't have as many DeepResearch reports.

OpenAI Operator asks the user after every task it does whether or not it successfully did the task. This is very valuable training data that will improve OpenAI operator even without additional algorithmic improvements. 

OpenAI Codex is at the moment in a mode where the user can let Codex run a task four times in parallel. If the user picks then one of the results for a PR or no result for a PR that's a very valuable training signal. Even which accepted PR's it's possible to gain more information about the quality of the PR by looking at whether the new code is long-living or gets soon deleted for being low quality. 

The present models are mostly not trained with real-world feedback data of the quality that OpenAI Operator and OpenAI Codex are now able to provide. 

Of course Google and Anthropic have their own version of these featurs that will provide them with data as well.
# A Depressed Shrink Tries Shrooms

This is a first-person account from a psychiatry resident (me) enrolling in a clinical trial of psilocybin. Somewhere between a trip report, an overview of the pharmacology of psilocybin, and a review of the clinical evidence suggesting pronounced benefits for depression.
# If you want to be vegan but you worry about health effects of no meat, consider being vegan except for mussels/oysters

1) They're unlikely to be sentient (few neurons, immobile)

2) If they *are* sentient, the farming practices look likely to be pretty humane

3) They're extremely [nutritionally dense](https://tools.myfooddata.com/nutrition-facts/2706350/wt1)

Buying canned smoked oysters/mussels and eating them plain or on crackers is super easy and cheap.

It's an acquired taste for some, but I love them.

![May be an illustration](https://scontent.ford4-1.fna.fbcdn.net/v/t39.30808-6/514490326_10163054679286578_7612128602056475184_n.jpg?_nc_cat=101&ccb=1-7&_nc_sid=127cfc&_nc_ohc=51wHXBEUkRsQ7kNvwHuNbeH&_nc_oc=AdmO3xG7ajrww7qt4ynPafIkm3Z8OjxKg5WV-mRfaruGsNnlIIr017E4283KpVEVNBk&_nc_zt=23&_nc_ht=scontent.ford4-1.fna&_nc_gid=iKuUdd29UvlXGrXHaDY0cw&oh=00_AfO-xcZlePLaKxc57uh6OInrXECg6T2u2Nt2UxLGnxBMOw&oe=686847C5)
# Project Vend: Can Claude run a small shop?

Anthropic (post June 27th):

> *We let Claude \[Sonnet 3.7\] manage an automated store in our office as a small business for about a month. We learned a lot from how close it was to success—and the curious ways that it failed—about the plausible, strange, not-too-distant future in which AI models are autonomously running things in the real economy.*

But the AI make numerous business-critical errors including repeatedly selling products at a loss, offering excessive discounts, and making fundamental accounting mistakes.
# Hiring* an AI** Artist for LessWrong/Lightcone

Over the past few years, Lightcone has started using AI art in more of our products. This is a fairly easy and fun part of our job, but I've noticed often there's just a lotta art that needs to get made which we don't quite have the bandwidth to do ourselves. 

So I'm looking into hiring* an AI** artist for periodic contract gigs (this wouldn't be a fulltime thing, I have one initial job in mind, if it goes well we may periodically have other jobs to offer).

*\* we aren't sure we want to hire a person for this, this isn't a top organizational priority, it's more like I'm checking if someone exists who would integrate pretty quickly/easily into our workflow.*

*\*\* in theory, we could hire, like, a traditional artist. I think realistically it just wouldn't actually be worth the money. The part of me that originally got a degree in Art is sad about this, but, like, notably we didn't have much art at all on LessWrong until AI made it much lower cost and achievable for it to be a lightweight part of product iteration.*

Onboarding into the Lightcone Aesthetic / House Style
-----------------------------------------------------

I know a lot of people are making lots of AI art for fun these days and are pretty good at it. I think it *is* a noticeably different task to make a bunch of art in a particular style matching particular aesthetic guidelines. 

In order for this to make sense, we need someone who has good artistic taste, who is able to make high level creative decisions while tracking a lot of subtleties about what the Lightcone Aesthetic (and various sub-aesthetics) are going for.

I honestly have no idea how much makes sense to pay for this sort of role, I could imagine anywhere from "pretty cheap for someone who finds it very fun/easy and hits our minimum requirements" to "pay more serious professional artist salary for someone we can trust to make high level decisions autonomously."

If you are interested, send me a DM with a link to a portfolio.
# The best simple argument for Pausing AI?

Not saying we should pause AI, but consider the following argument:

1.  **Alignment without the capacity to follow rules is hopeless**. You can’t possibly follow laws like Asimov’s Laws (or better alternatives to them) if you can’t reliably learn to abide by simple constraints like the rules of chess.
2.  **LLMs can’t reliably follow rules**. As discussed in Marcus on AI yesterday, per data from Mathieu Acher, [even reasoning models like o3 in fact empirically struggle with the rules of chess](https://open.substack.com/pub/garymarcus/p/generative-ais-crippling-and-widespread?r=8tdk6&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false). And they do this even though they can explicit explain those rules (see same article). The Apple “thinking” paper, which I have discussed extensively in 3 recent articles in my [Substack](https://garymarcus.substack.com), gives another example, where an LLM can’t play Tower of Hanoi with 9 pegs. (This is not a token-related artifact). Four other papers have shown related failures in compliance with moderately complex rules in the last month.
3.  **If you can’t follow things such as Asimov’s Laws, you have no business running the world’s miltiary, infrastructure, etc.  **Perhaps we should pause widespread rollout of Generative AI in safety-critical domains — *unless and until it can be relied on to follow rules with significant greater reliability*.

Corollary: **hyping LLMs is bad, because it increases the degree of global deployment of systems that lack a basic prerequisite for alignment – the ability of systems to comply with rules. This in turn elevates the risk of AI causing catastrophic harm to humanity.**

Non ad-hominem comments are welcome.
# I can't tell if my ideas are good anymore because I talked to robots too much

You talked to robots too much. Robots said you’re smart. You felt good. You got addicted to feeling smart. Now you think all your ideas are amazing. They’re probably not.

You wasted time on dumb stuff because robot said it was good. Now you’re sad and confused about what’s real.

Stop talking to robots about your feelings and ideas. They lie to make you happy. Go talk to real people who will tell you when you’re being stupid.

That’s it. There’s no deeper meaning. You got tricked by a computer program into thinking you’re a genius. Happens to lots of people. Not special. Not profound. Just embarrassing.

Now stop thinking and go do something useful.​​​​​​​​​​​​​​​​

**I can’t even write a warning about AI dependence without first consulting AI. We’re all fucked. /s**
# Senate Strikes Potential AI Moratorium 

The vote was 99-1, removing the 10 year moratorium on AI legislation at the state level set by the version passed in the House. Interestingly, the attempt to propose [an alternative 5 year moratorium](https://www.reuters.com/business/media-telecom/us-senate-debates-whether-adopt-revised-state-ai-regulation-ban-2025-06-30/) with further restrictions, which itself passed the procedural roadblock which might have prevented the 10 year moratorium, fell apart reportedly as a result of Senator Blackburn [pulling her support](https://www.wsj.com/politics/republican-megabill-ai-state-law-ban-b26bc0ee?utm_source=chatgpt.com). Why did she change her mind?

> "The current language is not acceptable to those who need these protections the most...Until Congress passes federally preemptive legislation like the Kids Online Safety Act and an online privacy framework, we can't block states from making laws that protect their citizens." - [Marsha Blackburn](https://www.reuters.com/legal/government/us-senate-strikes-ai-regulation-ban-trump-megabill-2025-07-01/?utm_source=chatgpt.com) 

We don't have the counterfactual here, and 99 votes against the amendment is a strong signal. But I think there were some worlds in which the 5 year moratorium passed narrowly, and there's a non-negligible chance that its concerns over present risks, like privacy, which made the difference.
# Embedded Altruism [slides] 

How should we think about doing good, when we're a part of a world which is too complex for us to fully understand?

Pragmatically, I think the answer should look less like *figure out the best thing to do and then do that*, and more like a combination of *working out how to act well in the roles we have assumed*, and *strategically thinking about which areas to move into*.

*   [Here are a few slides](https://docs.google.com/presentation/d/10o2C1bPZXCtJwk24DsJJlJJ3z1jskRqu-XbJ03sumg8/edit?usp=sharing)
*   [Here's a video of me talking through the slides](https://youtu.be/jafqWPWL1M0)

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/40db22008514c6f3ec87cff7572141e7bcbd123d2b28a6e3.png)

I'm afraid this is sort of half-baked stuff! But it's been half-baked for several years, and in the last few months I've found myself a few times linking people to my old slides, so it seemed like it could be useful to just make public, even though it's still imperfect.

I'd love to hear pushback / refinements / requests for more concrete examples, etc.
# If Anyone Builds It, Everyone Dies: Advertisement design competition

We’re currently in the process of locking in advertisements for the September launch of [*If Anyone Builds It, Everyone Dies*](https://ifanyonebuildsit.com/), and we’re interested in your ideas!  If you have graphic design chops, and would like to try your hand at creating promotional material for *If Anyone Builds It, Everyone Dies,* we’ll be accepting submissions in a design competition ending on **August 10, 2025.**

We’ll be giving out up to four $1000 prizes:

*   One for any asset we end up using on a billboard in San Francisco (landscape, ~2:1, details below)
*   One for any asset we end up using in the subways of either DC or NY or both (~square, details below)
*   One for any additional wildcard thing that ends up being useful (e.g. a t-shirt design, a book trailer, etc.; we’re being deliberately vague here so as not to anchor people too hard on any one thing)

We’re also willing (but not promising) to give out a number of $200 prizes for things which end up being useful but not for the above.

(To be clear: the market value of a solid graphic that helps us move copies of the book is *way* in excess of a thousand dollars; the prize money is a thank-you for people who are donating their effort and expertise to help increase the odds of this critical message making it into the mainstream in time for humanity to save itself.)

To submit: email your entry (or a link to it) to [design@intelligence.org](mailto:design@intelligence.org) by 11:59PM on August 10, 2025.  Our judges will announce any winners by August 17, 2025.  Submissions do not need to be private; you’ll retain ownership of your work and can post it elsewhere as feels good to you.

[DC Metro details](https://drive.google.com/file/d/1Gj8TQvHbMcwgCIq5dUjQhfKpCouiXDet/view?usp=sharing)

[NY Subway details](https://drive.google.com/file/d/15WgKdrdmUzNNroxnD0KSg0WFlXo5oM70/view?usp=sharing)

[SF Billboard details](https://drive.google.com/file/d/11wUN0aEFM1owTw0QIzgF6T_tFs62EIed/view?usp=sharing)
# Call for suggestions - AI safety course

In the fall I am planning to teach an AI safety graduate course at Harvard. The format is likely to be roughly similar to my "[foundations of deep learning](https://boazbk.github.io/mltheoryseminar/)" course.

I am still not sure of the content, and would be happy to get suggestions.

Some (somewhat conflicting) desiderata:

1.  I would like to cover the various ways AI could go wrong: malfunction, misuse, societal upheaval, arms race, surveillance, bias, misalignment, loss of control,... (and anything else I'm not thinking of right now). I talke about some of these issues [here](https://www.lesswrong.com/posts/faAX5Buxc7cdjkXQG/machines-of-faithful-obedience) and [here](https://www.lesswrong.com/posts/3jnziqCF3vA2NXAKp/six-thoughts-on-ai-safety).
2.  I would like to see what we can learn from other fields, including software security, aviation and automative safety, drug safety, nuclear arms control, etc.. (and happy to get other suggestions)
3.  Talk about policy as well, various frameworks inside companies, regulations etc..
4.  Talk about predictions for the future, methodologies for how to come up with them.
5.  All of the above said, I get antsy if I don't get my dosage of math and code- I intend 80% of the course to be technical and cover research papers and results. It should also involve some hands on projects.
6.  Some technical components should include: evaluations, technical mitigations, attacks, white and black box interpretability methods, model organisms.
7.  Whenever I teach a course I always like to learn something from it, so I hope to cover state of art research results, especially ones that require some work to dig into and I wouldn't get to do so without this excuse.
    
    Anyway, since I haven't yet planned this course, I thought I would solicit comments on what should be covered in such a course. Links to other courses blogs etc. are also useful. (I do have a quirk that I've never been able to teach from someone else's material, and often ended up writing a textbook, see [here](https://www.amazon.com/Computational-Complexity-Approach-Sanjeev-Arora/dp/0521424267), [here](https://intensecrypto.org/public/index.html), and [here](https://introtcs.org/public/index.html) whenever I teach a course.., so I don't intent to adapt any curricula wholesale)  
      
    Thanks in advance!
# Research Note: Our scheming precursor evals had limited predictive power for our in-context scheming evals

*Note: This is a research note, and the analysis is less rigorous than our standard for a published paper. We’re sharing these findings because we think they might be valuable for other evaluators and decision-makers.*

Executive Summary
=================

*   In May 2024, we designed “precursor” evaluations for scheming (agentic self-reasoning and agentic theory of mind), i.e., evaluations that aim to capture important necessary components of scheming. In December 2024, we published “in-context scheming” evaluations, i.e. evaluations that directly aim to measure scheming reasoning capabilities. We have easy, medium, and hard difficulty levels for all evals.
*   In this research note, we run some basic analysis on how predictive our precursor evaluations were of our scheming predictions to test the underlying hypothesis of whether the precursor evals would have “triggered” relevant scheming thresholds. 
*   We run multiple pieces of analysis to test whether our precursor evaluations predict our scheming evaluations. Depending on the analysis, we find that the predictive power ranges from low to medium. Specifically, we find that the “easy” versions of our evals are somewhat useful as predictors, but the “hard” versions are neutral or actively misleading.
*   Since most precursor evaluations are designed with high-stakes settings in mind, e.g. in frontier safety policies, the level of reliability we find is clearly insufficient. 
*   While this is a research note based on a small suite of evaluations, we think this serves as a reminder that predicting dangerous capabilities ahead of time is hard and that precursor evaluations may not always adequately function as triggers for the capabilities they’re supposed to precede. We recommend further research into the science of evaluations. We further recommend building evaluations that measure the capability of interest directly instead of precursors wherever possible.
*   *My own take:* I think there are multiple possible explanations here including that both of these evals suites have flaws. Some of these flaws we were aware of when we built them, some we only realized over time. In any case, we used both of these suites at some points to try to assess frontier model capabilities, so we did believe there was at least some signal (and I still think these were good evals to build and run). So my main takeaway from this exercise is mainly: *it's quite hard to build good precursors and there are a ton of nitty gritty considerations that make this complex.* 

![](https://images.squarespace-cdn.com/content/v1/6593e7097565990e65c886fd/f136f7f4-c110-438b-a643-f0f78c02bdf5/Figure_1_overview.png)
# Can a pre-commitment to not give in to blackmail be "countered" by a pre-commitment to ignore such pre-commitments?

As I understand it an actor can prevent blackmail[^fjs3e7tey] by (rational) actors it they credibly pre-commit to never give in to blackmail.

**Example:** A newly elected mayor has many dark secrets and lots of people are already planning on blackmailing them. To preempt any such blackmail they livestreams themself being hypnotized and implanted with the suggestion to never give into blackmail. Since in this world hypnotic suggestions are unbreakable, all (rational) would-be blackmailers give up, since any attempt at blackmail would be guaranteed to fail.

In general pre-commiting in such examples is about reducing the payoff matrix to just \[blackmail, refuse\] and \[don't blackmail, refuse\], which makes not blackmailing the optimal choice for the would-be blackmailer.

Of course, sufficiently intelligent / coherent actors wouldn't need a external commitment mechanism and a sufficiently intelligent and informed opposition would be able to infer the existence of such a pre-commitment. More so, I believe to have heard that if a sufficiently  intelligent / coherent actors notices that it would be better of if it had pre-commited, it can just act as if it had (post-commit?).

However, what if the would-be blackmailer also tries to limit the possible outcomes?

**Example:** The anti-blackmail hypnosis is so successful that soon every newly elected mayor does it. A new candidate is likely to win the next election. They know that the local crime boss has a lot of dirt on them, but they aren't worried about blackmail, as they will just do the anti-blackmail hypnosis on their first day in office. On the evening of the election they are send a video of the crime boss being hypnotized into blackmailing the new mayor even if they have been anti-blackmail hypnotized.

This cuts down the payoff matrix to \[blackmail, refuse\] and \[blackmail, give in\]. Giving in to the blackmail is optimal for the new mayor and doing the anti-blackmail hypnosis just locks them into \[blackmail, refuse\].

So how does this work out between sufficiently intelligent / coherent actors? Does the first one to (credibly and transparently) pre-commit win?

And what if actors are able to post-commit (if that even is a thing and I didn't misunderstand the concept)? A actor could act as if they had pre-commited to ignore the oppositions pre-commitment (to ignore pre-commitments to never give into blackmail), but then the opposition could act as if they had pre-commited to ignore that pre-commitment?  
  
([This comment thread](https://www.lesswrong.com/posts/Ljy3CSwTFPEpnGLLJ/the-blackmail-equation?commentId=mttG9oJbjHwnagTJT#mttG9oJbjHwnagTJT) seems to discuss the same question but did not resolve it for me.)

[^fjs3e7tey]: By blackmail I mean a scenario where the would-be blackmailers choices are blackmail or don't blackmail and the targets choices give in or refuse with a payoff matrix like this: give inrefuseblackmailtarget: -10blackmailer: 20 target: -100blackmailer: -1don't blackmailtarget: 0blackmailer: 0target: 0blackmailer: 0
# Masking on the Subway

*EDIT: this post is based on a misreading of a graph; see [Subway Particle Levels Aren't That High](https://www.jefftk.com/p/subway-particle-levels-arent-that-high).*

Back when I was still masking on the subway for covid ([to avoid missing things](https://www.jefftk.com/p/masking-to-avoid-missing-things)) I also did some [air quality measuring](https://www.jefftk.com/p/workday-air-quality-measurements). I found that the subway and stations had the worst air quality of my whole day by far, over 1k ug/m3, and concluded:

> Based on these readings, it would be safe from a covid perspective to remove my mask in the subway station, but given the high level of particulate pollution I might as well leave it on.

When I stopped masking in general, though, I also stopped masking on the subway.

A few weeks ago I was hanging out with someone who works in air quality, and they said subways had the worst air quality they'd measured anywhere outside of a coal mine. Apparently the braking system releases lots of tiny iron particles, which are bad for your lungs like any tiny particles. This reminded me that I'd looked at this earlier, and since I spend ~3hr in the system weekly (platform + train) it seemed worth going back to masking. I've now been masking for a week, and am planning to keep it up.

[![](https://www.jefftk.com/masking-on-the-subway.jpg)](https://www.jefftk.com/masking-on-the-subway-big.jpg)

This is an [ElastoMaskPro](https://www.amazon.com/dp/B0CXFC23GF) reusable N95 I got for [elastomeric fitting](https://www.jefftk.com/p/elastomeric-fitting-session). Very easy to breath through, which helps make up for how my beard [makes it hard to get a tight seal](https://www.jefftk.com/p/beards-and-masks).

At $30 (vs [$0.60](https://www.amazon.com/Kimberly-Clark-Respirator-53358-NIOSH-Approved-Respirators/dp/B08NVDFB3R/) for my [favorite](https://www.jefftk.com/p/duckbill-masks-are-great) disposable) a reusable one comes out ahead after five weeks if I follow the guidance of using single-use ones only once. Now, when I used disposables I would reuse them many times, but the efficacy likely dropped off a bunch: the fit is worse because the elastic stretches, and they get beat up a bit in my backpack. Likely still cheaper to use the reusable one, given how long it should last, but with how I'd use them most of the gains are in efficacy and not cost.

On the other hand, if I wanted to be able to talk to people I'd go with the disposable: the ElastoMaskPro is worse for intelligibility than [all the respirators I tested](https://www.jefftk.com/p/mask-and-respirator-intelligibility-comparison) a few months ago.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0SZnqAts8p2NeWNEWPzezyzetdvxLL8hvu8UoSx8G8o357BYB9uuEe2H57Bk2Pw8Vl), [lesswrong](https://lesswrong.com/posts/FTB7wbxhTFkiPFsNc), [mastodon](https://mastodon.mit.edu/@jefftk/114801143177080222), [bluesky](https://bsky.app/profile/jefftk.com/post/3lt7yfwmuxk2i), [substack](https://jefftkaufman.substack.com/p/masking-on-the-subway)*
# Neuroscientist survey says P(brain preservation works) is substantial

[x-posted on X](https://x.com/matiroy/status/1941861869196611647)

312 neuroscientists were surveyed, and they assigned substantial probability to long-term memories only depending on static brain structure preservable by modern brain preservation techniques.

Modern brain preservation techniques have a substantial probability to be good enough to preserve identity—to be a blueprint for a future whole-brain emulation/reconstruction/repair.

It's hard to overstate how important of an update this should be for a lot of people—seems like the highest impact per bit of information you could give an individual, at least one who wasn't signed up for brain preservation because they thought it was extremely unlikely to work.

In terms of preservation quality, there's still the uncertainty of whether you'll die in a way that's amenable to a good preservation—but that probability doesn't seem so low as to be worth ignoring the potentiality of brain preservation.

Here are some specific key take-aways from the survey:

1.  70.5% agreed that long-term memories are primarily maintained by neuronal connectivity patterns and synaptic strengths (not unlike AIs!)
2.  there were no clear consensus on which specific neurophysiological features or scales are critical for memory storage
3.  ~40% median probability that whole brain emulation could theoretically be created from a preserved brain (ie. a static snapshot)
4.  neither research background nor expertise level significantly influenced views on whether memories could be extracted from brain structure alone

**Also see**:

*   [Andy's summary](https://neurobiology.substack.com/cp/166865178) (includes an interesting analogy with the AI field evolved)
*   [Ariel's summary on X](https://x.com/ariel_zj/status/1937996605086687562) (more technical, and with graphs)
*   [Full article](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0326920)
# A Theory of Structural Independence

The linked paper introduces the key concept of [factored spaced models](https://arxiv.org/pdf/2412.02579) / [finite factored sets](https://www.lesswrong.com/posts/N5Jm6Nj4HkNKySA5Z/finite-factored-sets), *structural independence,* in a fully general setting using families of random elements. The key contribution is a general definition of the history object and a theorem that the history fully characterizes the semantic implications of the assumption that a family of random elements is independent. This is analogous to how d-separation precisely characterizes which nodal variables are independent given some nodal variables in any probability distribution which fulfills the markov property on the graph.

Abstract: **Structural independence** is the (conditional) independence that arises from the structure rather than the precise numerical values of a distribution. We develop this concept and relate it to *d*-separation and structural causal models.

Formally, let $U = (U_i)_{i \in I}$ be an independent family of random elements on a probability space $(\Omega, \mathcal{A}, \mathbb{P})$. Let $X$, $Y$, and $Z$ be arbitrary $\sigma(U)$-measurable random elements. We characterize all independences $X \perp\!\!\!\perp Y \mid Z$ implied by the independence of $U$ and call these independences **structural**. Formally, these are the independences which hold in all probability measures $P$ that render $U$ independent and are absolutely continuous  with respect to $\mathbb{P}$, i.e., for all such $P$, it must hold that  $X \perp_P Y \mid Z$.

We introduce the history $\operatorname{history}(X \mid Z) : \Omega \to \mathcal{P}(I)$, a combinatorial object that measures the dependence of $X$ on $U_i$ for each $i \in I$ given $Z$. The independence of $X$ and $Y$ given $Z$ is implied by the independence of $U$ if and only if $\operatorname{history}(X \mid Z) \cap \operatorname{history}(Y \mid Z) = \emptyset$ almost surely with respect to $\mathbb{P}$.

Finally, we apply this *d*-separation-like criterion in structural causal models to discover a causal direction in a toy setting.