Harley-ml/lesswrong huggingface dataset:
# [Linkpost] Visual roadmap to strong human germline engineering


*([x.com](https://x.com/BerkeleyGenomic/status/1908647929117868469), [bsky](https://bsky.app/profile/berkeleygenomics.bsky.social/post/3lm3ydeyh7k2w).)*

This is a linkpost for "[Visual roadmap to strong human germline engineering](https://berkeleygenomics.org/articles/Visual_roadmap_to_strong_human_germline_engineering.html)". That page gives a quick introduction to the technical problem of strong germline engineering, and gives a mildly-interactive visual roadmap. Here's a screenshot:

![](https://i.imgur.com/iUnFpYJ.png)
# Austin Chen on Winning, Risk-Taking, and FTX

Timothy and I have recorded a new episode of our podcast with Austin Chen of [Manifund](https://manifund.org/) (formerly of [Manifold](http://manifold.markets/), behind the scenes at [Manifest](https://manifest.com/)).

The start of the conversation was contrasting each of our North Stars- Winning (Austin), Truthseeking (me), and Flow (Timothy), but I think the actual theme might be “what is an acceptable amount of risk taking?” We eventually got into a discussion of Sam Bankman-Fried, where Austin very bravely shared his position that SBF has been unwisely demonized and should be “freed and put back to work”. He by no means convinced me or Timothy of this, but I deeply appreciate the chance for a public debate.

Episode:

[Transcript](https://docs.google.com/document/d/1V38t8TSk7jroXwtdLymzJ8iEyeSUtQK3/edit) (this time with filler words removed by AI)

Editing policy: we allow guests (and hosts) to redact things they said, on the theory that this is no worse than not saying them in the first place. We aspire but don’t guarantee to note serious redactions in the recording. I also edit for interest and time.
# Are there any (semi-)detailed future scenarios where we win?

I think there's tremendous value in imagining what the future could look like, in as much detail as possible. Some recent examples: [AI-2027](https://ai-2027.com/), [A History of the Future](https://www.lesswrong.com/posts/CCnycGceT4HyDKDzK/a-history-of-the-future-2025-2040), [How AI Takeover Might Happen in 2 Years](https://www.lesswrong.com/posts/KFJ2LFogYqzfGB3uX/how-ai-takeover-might-happen-in-2-years).  
  
Do you know any optimistic stories along these lines? I'm not really looking for technical descriptions of how the alignment problem got solved; instead, I'm interested in examples of **how people envision the path or process by which we reached a positive, realistic, post-singularity future**.  
  
I have real trouble picturing this myself, and I find that a bit disturbing. It feels like sailing across an unknown sea—it's good to avoid rocks, reefs, and sea monsters, but if you don't have any clear idea of what your destination looks like, you might still be steering in entirely the wrong direction. And what could that direction be? Well, suppose you have 10 different optimistic future scenarios, and they all share some common feature. Maybe this is the thing that really matters?

To clarify, I'm not just looking for descriptions of [utopias](https://nickbostrom.com/deep-utopia/). The key question for me is: how did we actually get there?
# Log-linear Scaling is Worth the Cost due to Gains in Long-Horizon Tasks

This post makes a simple point, so it will be short. I am happy to discuss more in the comments, and based on this write a longer post later. Much prior work [(eg: \[1\])](https://arxiv.org/abs/2404.04125) has shown that exponential data and compute is required for each unit improvement in accuracy. A popular argument this leads to:

 *Scaling compute and data is economically not viable above a threshold*

This argument has a key issue. Log-linear gains in accuracy (or loss) are shown on "one-step" benchmarks such as direct QA. However, economic benefits primarily arise from tasks which require long thinking, output and action horizons. For this, I propose tracking **k-step success rate**, i.e. the probability that a k-step solution achieves its goal. Most benchmarks previously analyzed for log-linear scaling have at best a few (small constant) steps. For simplicitly, let us assume this constant is 1, this does not affect the rest of the argument (based on asymptotic calculations in **k**) beyond a constant factor.

We can make two (simplifying) assumptions to connect log-linear scaling in 1-step accuracy to k-step success rate. (1) Each step has an independent probability to succeed, related to the 1-step accuracy by a constant. (2) All steps need to be correct for the full k-step solution to succeed. This is true for many useful tasks, like solving math problems, and an agent executing a task on the web. While recovering from failures is possible, subsequent steps might violate Assumption (1), so it's best to simplify this for now. Now the main mathematical argument is as follows:

Let us define the 1-step accuracy as $x$. Every unit increase in 1-step accuracy from $x%$% to $(x+1)$% leads to an improvement in k-step accuracy of $(x+1)^k - x^k$. The absolute improvement is ~$kx^k$ by taking the derivative of $f(x) = x^k$, i.e. the improvement compounds based on the horizon-length. The relative improvement is $\frac{k}{x}$ by the [binomial approximation](https://en.wikipedia.org/wiki/Binomial_approximation). Thus the **k-step success rate improves exponentially in task horizon length.** 

As inference-compute starts outweighing pretraining compute and AI is used for longer-horizon tasks like automated coding, jobs and research, log-linear scaling of x with pre-training compute will be worth it as the **k-step success rate** will improve much faster for large k. 

Further, we can use similar analysis to understand recent results that [the length of tasks that AI can do is increasing exponentially with time. ](https://www.lesswrong.com/posts/deesrjitvXM4xYGZd/metr-measuring-ai-ability-to-complete-long-tasks)Thanks to [@nostalgebraist](https://www.lesswrong.com/users/nostalgebraist?mention=user) for providing this math in the comment below.
# Who wants to bet me $25k at 1:7 odds that there won't be an AI market crash in the next year?

If there turns out not to be an AI crash, you get a  1/(1+7) * $25,000 = $3,125  
If there is an AI crash, you transfer $25k to me.

If you believe that AI is going to keep getting more capable, pushing rapid user growth and work automation across sectors, this is near free money. But to be honest, I think there will [likely be an AI crash](https://www.lesswrong.com/posts/piaeNZBDxXtSjaZ7G/if-ai-is-in-a-bubble-and-the-bubble-bursts-what-would-you-do?commentId=RdEgFJn4LomcfEH4m) in the next 5 years, and on average expect to profit well from this one-year bet. 

If I win, I want to give the $25k to organisers who can act fast to restrict the weakened AI corps in the wake of the crash. So bet me if you're highly confident that you'll win or just want to hedge the community against the possibility of a crash.

To make this bet, we need to set the threshold for a market crash. I find it hard to define comprehensively:  is a 40% slump in VC investment in AI start-ups a crash? Or a 30% slump in LLM subscriptions? Or GPU data center providers unable to cover debts? Or just a majority vote here that a crash happened? How do we non-ambiguously define it? 

Also, I'd need some proof of you having $25k in cash. If you say you have a well-earning job and saved up the money, that might be enough.
# birds and mammals independently evolved intelligence

Researchers used RNA sequencing to observe how cell types change during brain development. Other researchers looked at connection patterns of neurons in brains. Clear distinctions have been found between all mammals and all birds. They've concluded intelligence developed independently in birds and mammals; I agree. This is evidence for convergence of general intelligence.
# Short Timelines Don't Devalue Long Horizon Research

Short AI takeoff timelines seem to leave no time for some lines of alignment research to become impactful. But any research rebalances the mix of currently legible research directions that could be handed off to AI-assisted alignment researchers or early autonomous AI researchers whenever they show up. So even hopelessly incomplete research agendas could still be used to prompt future capable AI to focus on them, while in the absence of such incomplete research agendas we'd need to rely on AI's judgment more completely. This doesn't crucially depend on giving significant probability to long AI takeoff timelines, or on expected value in such scenarios driving the priorities.

Potential for AI to take up the torch makes it reasonable to still prioritize things that have no hope at all of becoming practical for decades (with human effort). How well AIs can be directed to advance a line of research further (and the quality of choices about where they should be directed) depends on how well these lines of research are already understood. Thus it can be important to make as much partial progress as possible in developing (and deconfusing) them in the years (or months!) before AI takeoff. This notably seems to concern agent foundations / decision theory, in contrast with LLM interpretability or AI control, which more plausibly have short term applications.

In this sense current human research, however far from practical usefulness, forms the data for aligning the early AI-assisted or AI-driven alignment research efforts. The judgment of human alignment researchers who are currently working makes it possible to formulate more knowably useful prompts for future AIs (possibly in the run-up to takeoff) that nudge them in the direction of actually developing even preliminary theory into practical alignment techniques.
# Reasoning models don't always say what they think

Do reasoning models accurately verbalize their reasoning? Not nearly as much as we might hope! This casts doubt on whether monitoring chains-of-thought (CoT) will be enough to reliably catch safety issues.

We slipped problem-solving hints to Claude 3.7 Sonnet and DeepSeek R1, then tested whether their Chains-of-Thought would mention using the hint (if the models actually used it). We found Chains-of-Thought largely aren’t “faithful”: the rate of mentioning the hint (when they used it) was on average 25% for Claude 3.7 Sonnet and 39% for DeepSeek R1.

![Graph comparing the four models (Claude 3.5 and 3.7 Sonnet, and DeepSeek V3 and R1) on their faithfulness - the fraction of time they mentioned having used the clue.](https://pbs.twimg.com/media/GnnuSegX0AALDhH?format=jpg&name=4096x4096)

This result suggests that monitoring CoTs is unlikely to reliably catch rare, catastrophic behaviors—at least in settings like ours where CoT reasoning is not necessary for the task. CoT monitoring might still help us notice undesired behaviors during training and evaluations.

Our results suggest that CoT is less faithful on harder questions. This is concerning since LLMs will be used for increasingly hard tasks. CoTs on GPQA (harder) are less faithful than on MMLU (easier), with a relative decrease of 44% for Claude 3.7 Sonnet and 32% for R1.

Does outcome-based training increase faithfulness? Only to a small extent. Training models to use their CoTs more effectively does make them more faithful, but the benefits quickly plateau. 

![CoT faithfulness scores with outcome-based RL on an earlier snapshot of Claude 3.7 Sonnet. Outcome-based RL initially increases CoT faithfulness substantially but plateaus at 28% on MMLU and 20% on GPQA. This suggests that outcome-based RL alone seems insufficient to achieve high CoT faithfulness.](https://pbs.twimg.com/media/GnnuklbW0AA6gg_?format=png&name=large)

We also tested whether CoTs could be used to spot *reward hacking*, where a model finds an illegitimate exploit to get a high score. When we trained models on environments with reward hacks, they learned to hack, but in most cases almost never verbalized that they’d done so.

![Graph showing how often the model reward hacked and verbalized, versus didn't verbalize, that it had done so, across different types of reward hacks.](https://pbs.twimg.com/media/GnnvOdCXsAAq8C5?format=jpg&name=4096x4096)

To make CoT monitoring a viable way to catch safety issues, we’d need a way to make CoT more faithful, evidence for higher faithfulness in more realistic scenarios, and/or other measures to rule out misbehavior when the CoT is unfaithful.
# Electric Lunchbox

People are often a lot more interested in hot meals, and my kids are no exception. I've tried a bunch of options here including [putting rocks in thermoses](https://www.jefftk.com/p/thermal-mass-thermos) (turns out kindergarteners worry more than you might think about whether they'll accidentally eat rocks that are bigger than their mouths), bringing a [microwave](https://www.jefftk.com/p/tour-retrospective-february-2024) and [toaster](https://www.jefftk.com/p/kingfisher-tour-february-2025) (good, but too bulky for school especially when you count the battery), and ramen (great, but Lily only likes one kind and I'm worried she'll get sick of it). We recently got an electric lunchbox ( [this one](https://www.amazon.com/dp/B0DRJCPMQ2) because it was on sale, but there are a bunch) and it's pretty great!

It's insulated, and we prepare it the night before and put it in the fridge:

[![](https://www.jefftk.com/heated-lunchbox-in-fridge.jpg)](https://www.jefftk.com/heated-lunchbox-in-fridge-big.jpg)

In the morning I set the timer:

[![](https://www.jefftk.com/electric-lunchbox-timer.jpg)](https://www.jefftk.com/electric-lunchbox-timer-big.jpg)

It only goes up to 4hr, perhaps for [danger zone](https://www.jefftk.com/p/what-is-the-real-danger-zone-for-food) reasons, but since the kids leave at 8am and lunch is at 12:30 that's just right (4hr timer, 30min heating).

When the timer gets to zero it starts heating, and counts down from 30min (by default; adjustable):

[![](https://www.jefftk.com/lunchbox-heating-in-progress.jpg)](https://www.jefftk.com/lunchbox-heating-in-progress-big.jpg)

Tastes marginally better than microwave food, and loads better than cold food.

I haven't yet tested if it's powerful enough to handle frozen food. I hope it is, since then you could prepare it whenever and leave it in the [freezer](https://www.jefftk.com/p/freezer-contents) until ready to eat.

I also wonder if it could bake bread. It only goes up to 90C and it be a bit like [steamed bread](https://en.wikipedia.org/wiki/Steamed_bread), but [freshness](https://www.jefftk.com/p/really-fresh-baking) might outweigh texture here.
# [Paper] Infra-Bayesian Decision-Estimation Theory

*Diffractor is the first author of this paper.*  
*Official title: "Regret Bounds for Robust Online Decision Making"*

> **Abstract:**  We propose a framework which generalizes "decision making with structured observations" by allowing robust (i.e. multivalued) models. In this framework, each model associates each decision with a convex set of probability distributions over outcomes. Nature can choose distributions out of this set in an arbitrary (adversarial) manner, that can be nonoblivious and depend on past history. The resulting framework offers much greater generality than classical bandits and reinforcement learning, since the realizability assumption becomes much weaker and more realistic. We then derive a theory of regret bounds for this framework. Although our lower and upper bounds are not tight, they are sufficient to fully characterize power-law learnability. We demonstrate this theory in two special cases: robust linear bandits and tabular robust online reinforcement learning. In both cases, we derive regret bounds that improve state-of-the-art (except that we do not address computational efficiency).

In our new paper, we generalize [Foster et al](https://arxiv.org/abs/2301.08215)'s theory of "decision-estimation coefficients" to the "robust" ([infa-Bayesian](https://www.alignmentforum.org/posts/zB4f7QqKhBHa5b37a/introduction-to-the-infra-bayesianism-sequence)) setting. The former is the most general known theory of regret bounds for multi-armed bandits and reinforcement learning, which comes close to giving tight bounds for all "reasonable" hypothesis classes. In our work, we get an analogous theory, even though our bounds are not quite as tight.

Remarkably, the result also establishes a tight connection between infra-Bayesianism and [Garrabrant induction](https://arxiv.org/abs/1609.03543). Specifically, the algorithm which demonstrates the upper bound works by computing beliefs in a Garrabrant-induction-like manner[^pfczrrhwoh], and then acting on these beliefs via an appropriate trade-off between infra-Bayesian exploitation and exploration (defined using the "decision-estimation" approach).

It seems quite encouraging that the two different theories which came out of thinking about "[logical uncertainty](https://www.lesswrong.com/w/logical-uncertainty)" (infra-Bayesianism and Garrabrant induction) can be unified in this manner[^hf0ut24dx2o], boosting our confidence that we are on the right path.

*The paper assumes no prior familiarity with either infra-Bayesianism or Garrabrant induction.*

[^pfczrrhwoh]: In the start of each episode. 

[^hf0ut24dx2o]: Although we think that a fuller treatment of logical uncertainty requires introducing metacognition.
# Can I learn language faster? Or, perhaps, can I memorize the foreign words and recall them faster?

I've been learning Korean for a while now. I am finding a rather asymmetric aspect to my vocabulary learning process and results. I use Anki as the learning tool for vocabulary. What I find is that fairly quickly I get to a good recognition[^9wi5wz6ojdb] performance. But I find my recall performance is significantly behind. I've got some ideas on why this type of asymmetry occurs but the bottom line is that I am beginning to think I've been approach vocabulary building the wrong way.

A quick google about the asymmetry suggests that this is a fairly common problem for language learners. I wonder if it's not because most settings for developing one's vocabulary in a new language tend to follow the same approach I see in the Anki decks I've downloaded or created myself. You get the foreign word on the front and then learn what the native language word is on the back.

So what I've though is maybe that is backwards. I'm starting a test to see flipping the approach, putting the word I already know up first and then having to come up with the Korean word as the "answer" might perhaps improve my recall performance, which I suspect would have no affect on my recognition ability.

If my supposition holds here then I expect I will improve my learning speed and more quickly.  As it is now it's a bit of a slow slog of, what seems like, the process of hammering things into my head until I can recall the word on demand rather than recognizing the word when seen (or to some extent heard). And, at the end of the day that is the goal as I don't want to just read but be able to actually interact with others in a day-to-day type setting.

I've been trying this approach for a bit more than a week now. Too soon to say if it's actually working better for me but I have that "fuzzy" feeling I'm doing better. I am highly confident that reversing the process here cannot reduce my vocabulary acquisition rate. 

I'm wondering if anyone else has tried this and if they found it helped or not. Or what people think of the shift in approach in general.

[^9wi5wz6ojdb]: I will use recognition and recall in a pretty obvious way but for clarity, recognition is about seeing the Korean word and knowing what English word would have the same meaning. Recall is the ability to, without any English prompt, come up with the Korean word.
# How likely are the USA to decay and how will it influence the AI development?

Trump's politics is so far from being understandable that it appears to cause the decay of the USA; for instance, the governor of California ended up [pleading to exclude California-made products](https://www.foxnews.com/politics/scoop-newsom-ask-world-leaders-exempt-california-exports-from-retaliatory-tariffs) from tariffs introduced as retailation against Trump's measures; another example of a state wishing to secede is [New Hampshire](https://www.newsweek.com/republicans-want-new-hampshire-leave-united-states-1866130) where the Republican state Representative Jason Gerhard proposed that New Hampshire should peacefully declare independence from the U.S. if the national debt surpasses $40 trillion, which is apparently to take place in less than 1.5 years.  [^37zpp8f5vj]

Meanwhile, the [optimistic 2027 timeline](https://www.lesswrong.com/posts/CqHMdLcdupf7y5buK/an-optimistic-2027-timeline) implies that Taiwan is to be invaded in late 2026, if not earlier, while the [pessimistic timeline](https://www.lesswrong.com/posts/TpSFoqoG2M5MAAesg/ai-2027-what-superintelligence-looks-like-1) implies that AI is to start taking jobs by that time and that the full effects of the AI-related research are to kick in during 2027. How likely is a potential decay of the USA to let Chinese spies do much more work like destroying the  data centres or leaving them without electricity? What effect will it have on the race to AGI?

EDIT: Trump somehow [decided to impose tariffs](https://www.washingtonpost.com/world/2025/04/09/taiwan-tariffs-trump-trade-war/) on most goods from Taiwan, except for the chips necessary to perform calculations. 

[^37zpp8f5vj]: The national debt is currently $36.7 trillion. In four years it is projected to reach $46.4 trillion.
# Will US tariffs push data centers for large model training offshore?

One aspect of the tariffs will be that it's more costly to import chips into the United States. Does anybody understand data center economics and the tariffs policy well enough to know whether this will push new data center creation needed to train large models into other states?
# What are good safety standards for open source AIs from China?

It looks like the compromising between those who want to ban Chinese AI models in the United States in the c rrent administration will be to allow the models but require some safety standards to prevent China from harming the United States.

While we generally would want more AI regulation than the current administration is willing to create, this is a window where the AI safety community potentially can affect safety policy.

Within the existing political constraints, what standards for the Chinese models should we wish for?
# What is autism?

I am confused about what autism *is*. Whenever I try to investigate this question I end up coming across long lists of traits and symptoms where various things are unclear to me.

One thing that's unclear to me is when these traits and symptoms are rooted in something deeper. Here's what I mean. If A1, A2 and A3 are all rooted in A, B1, B2 and B3 are all rooted in B, and C1, C2 and C3 are all rooted in C, I'd be interested in knowing what the A, B and C are, and less interested in knowing what the A1, A2, A3, B1, B2, B3, C1, C2 and C3 are.

Another thing that's unclear to me is why we are putting an umbrella over this collection of traits and calling it autism. Is it an appropriate place to [draw the boundary](https://www.lesswrong.com/posts/d5NyJ2Lf6N22AD9PB/where-to-draw-the-boundary)?

If someone is dyslexic, over six feet tall, and has the flu, we could draw a boundary around that and name it something, but we don't because those collection of traits don't [form a cluster in Thingspace](https://www.lesswrong.com/posts/jMTbQj9XB5ah2maup/similarity-clusters). Autism feels to me like a (much) less extreme example of this. For example, I'm not sure why reduced eye contact would be related to sensory sensitivity. That said, I recognize that I don't know much about this and the boundary very well might have been drawn in the right place.

Related to my top-level question of what autism is, I'm also interested in book recommendations or recommendations for other resources where I can learn more.
# Thoughts on the Double Impact Project

How it works
============

You donate to the Republican or Democratic pool. The pool with the highest amount donates what it has more at the margin to their political party. Everything else goes to other charities. For a better and more complete explanation, check out: [https://doubleimpact.charity/](https://doubleimpact.charity/)

Should it work?
===============

**Assuming the platform is legibly reliable, I would expect the following people to donate to Double Impact instead of a political party**:

1.  Are homo economicus
2.  Prefers to donate to their political party lacking coordination mechanisms
3.  Thinks the value of donating the same amount to both political parties is less than the value of donating to another charity

And this would create a lot of value assuming it's true that political donations are near zero-sum. (You could argue that they aren't–ex.: maybe they are a good way to educate the public about political issues.)

**Even people who think donating to another charity is more valuable than donating to a political party might donate through that process** if it seemed sufficiently likely that their preferred party will have less donations anyway as they would have a >50% chance of doubling their donation. That part seems like it would just be adding a transaction cost to get to the same outcome and so is actually a downside of the platform. Although maybe the transaction cost can be made extremely small and maybe there are still positive externalities like the information value of having money vote in that way. Also, that would require finding someone that would agree about donating to the same (or a comparable) charity as part of their matching (not just any charity), which might be quite hard for some people—possibly most people. If people did donate to Double Impact just to cancel the other side's donation but wouldn't have donated otherwise to a political party anyway, then that would also be a reason for the other side not to donate through that platform.

At some point this incentive mechanism would stop working because a party receiving 0 vs a party receiving non-zero would benefit more from having money at the margin.

**But why hasn't this taken off nowhere near this level as far as I can tell?** Any of the above assumptions might be wrong to some extent. 

*   Maybe it will take-off once the platform gains a reputation. (I know one of the founders and they definitely seem trustworthy to me.)
*   Or maybe most political donors think donating money to their party goes further than removing money from the other party.
*   Or maybe the value for saying “I donated to party X” is worth much more than “I donated to party X through the Double Impact mechanism” because it's [more than 5 words](https://www.lesswrong.com/posts/4ZvJab25tDebB8FGE/you-get-about-five-words). And more than 10 really: it requires reading–and understanding–a paragraph to an article worth of text.
# Offer: Team Conflict Counseling for AI Safety Orgs

Many teams suffer from frictions - whether it's subtle tension or full-blown paralysis. AI safety orgs are no exception. In high-stakes, mission-driven contexts, unspoken tensions and misaligned communication can quietly drain energy, stall progress, and damage morale.

If you’re part of a team or organization in the AI safety space and want support—either to resolve an acute conflict or to build a more resilient team culture—I can help.

About me
========

I'm Severin. I hold an M.A. in philosophy, and I work as a coach and communication trainer.

My relevant background includes:

*   **Certified counseling trainer (**[**Kooperative Beratung**](http://www.kooperative-beratung.com/)**)** — including training in a method literally called *Team Conflict Counseling*, designed for teams in educational settings, but highly transferable.
*   **Certified in** [**Theme-Centered Interaction**](https://en.wikipedia.org/wiki/Theme-centered_interaction), a group facilitation method grounded in humanistic psychology.
*   Several hundred hours of receiving and facilitating **relational and inner work modalities**, including Circling, Focusing, IFS, Authentic Relating, and Nonviolent Communication.
*   Writing about [facilitation](https://forum.effectivealtruism.org/posts/psMz4CanKHk5Xya4g/community-building-lessons-from-ten-years-of-facilitation), [community building](https://forum.effectivealtruism.org/s/W2nJHGeMEEceHfLg9/p/L8XxRjP5MwPhk5y6v), and [other people stuff](https://www.lesswrong.com/posts/RaNhnNjExip36NMxM/advice-for-newly-busy-people) on LessWrong and the EA Forum.

How I can help
==============

If things are already on fire...
--------------------------------

I can support your team in navigating acute conflict, through:

*   Team-wide facilitation
*   Mediation between two individuals
*   1-on-1 coaching for someone caught in a tough situation

I work confidentially and neutrally - my role isn’t to pick sides or impose solutions, but to facilitate clearer understanding, mutual empathy, and conditions where creative resolution can emerge.

If you want to build long-term team health...
---------------------------------------------

I also offer preventive support, such as:

*   Feedback training and communication workshops
*   Process consulting to streamline culture and decision-making
*   Burnout prevention and resilience work (e.g., [*Active Hope*](https://forum.effectivealtruism.org/posts/HvGrp2r2MMWvxXzn5/active-hope-for-ea-burnout) workshops)

Pricing
=======

I know not all valuable work in AI safety is well-funded. So I use a sliding scale:  
**10€–100€ per hour**, depending on your budget.

I care more about impact than invoices - if you think this would help your org, don’t hesitate to reach out.

How to get started?
===================

Fill out [this intake form](https://docs.google.com/forms/d/e/1FAIpQLSePFRnWMwgduCDYf8-U8WCF7cUQdgX7NEfrXa2R9MVNwRdWjw/viewform) with your name, email, and a short note on what you’re looking for. I'll follow up to schedule a free exploratory call.
# Map of AI Safety v2

The original Map of AI Existential Safety became a popular reference tool within the community after its launch in 2023. Based on user feedback, we decided that it was both useful enough and had enough room for improvement that it was worth creating a v2 with better organization, usability, and visual design. Today we’re excited to announce that the new map is live at [AISafety.com/map](http://aisafety.com/map).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f5762149d28b20be5594f41e3f3503019d3737f316cfd61d.png)

Similar to the original map, it provides a visual overview of the key organizations, programs, and projects in the Al safety ecosystem. Listings are separated into 16 categories, each corresponding to an area on the map:

*   Advocacy
*   Blog
*   Capabilities research
*   Career support
*   Conceptual research
*   Empirical research
*   Forecasting
*   Funding
*   Governance
*   Newsletter
*   Podcast
*   Research support
*   Resource
*   Strategy
*   Training and education
*   Video

We think there’s value in being able to view discontinued projects, so we’ve included a graveyard for those.

Below the map is also a cards section where the same data is displayed as a searchable list.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d33bd6cb57a4804a71ee9dd16966c6efec86f6b9bf1ca1ac.png)

Like all resources on [AISafety.com](http://aisafety.com), we’re committed to keeping the map [accurate and up-to-date](https://www.lesswrong.com/posts/NpmckjpCq8HgGSK9R/bryce-robertson-s-shortform#gyc4hkqQpAaR5hrHi). If you think something should be added, removed, or modified, or you have a feature suggestion, please let us know in the comments or via the [general feedback form](https://www.aisafety.com/feedback). We’re currently exploring adding a “restore zoom to 100% and recenter” button, as well as a link to the underlying data in Airtable.

We’re also looking at creating large print versions of the map. We’ve just contacted Lighthaven to see if they would be interested – if you manage a physical space and are keen on having one too, please get in touch via the [feedback form](https://www.aisafety.com/feedback).

Special thanks to [@plex](https://www.lesswrong.com/users/ete?mention=user), [@ScottWofford](https://www.lesswrong.com/users/scottwofford?mention=user), [@GhaithA](https://www.lesswrong.com/users/ghaitha?mention=user), and [@Tomato*](https://www.lesswrong.com/users/tomato?mention=user) for their contributions to this project.
# The real reason AI benchmarks haven’t reflected economic impacts

Basically, the linkpost argues that the broad reason why benchmark progress is disconnected from broader progress is because AI progressed so fast that they didn't realize their benchmarks would be obsolete soon.

More is in the link above.

Discuss!

> Relatedly, another part of the story may be that researchers underestimated the rate of AI progress. For instance, researchers working on autoregressive language models in 2016 tended not to think of AI systems as performing “economically useful tasks”, thinking of these systems as insufficiently capable of doing so. As such, benchmarks may have been deliberately designed to be relatively cheap proxies for real-world tasks, and also relatively simple.
# Telescoping

Sometimes I try to figure out what to do in a given week, and my first step is to think "what are my long term goals?". Then I realize I have no long-term goals. I sit down and think, "Hmm, what do I want to do with my life?" Eventually I give up, depressed, and go make breakfast or something.

Recently I tried going the other way, with much more success. I call this exercise **telescoping**. It takes 7 minutes, and is performed as follows:

1.  Open a text document or grab a piece of paper. Grab a kitchen timer with an alarm, or your phone's alarm app.
2.  For one minute, answer: *What do I want to happen today?*
3.  For one minute, answer: *What do I want to happen this week?*
4.  For one minute, answer: *What do I want to happen this month?*
5.  For one minute, answer: *What do I want to happen this year?*
6.  For one minute, answer: *What do I want to happen this month?*
7.  For one minute, answer: *What do I want to happen this week?*
8.  For one minute, answer: *What do I want to happen today?*

*\[... continued in blog ...\]*
# Understanding Trust - Overview Presentations

Since mid-January, I've been working with four students - Hanna Gabor, Roman Malov, Norman Hsia, Paul Rapoport - on the research agenda I [articulated last year](https://www.lesswrong.com/posts/7AzexLYpXKMqevttN/seeking-collaborators). The AI Safety Camp project is wrapping up, but most of us plan to continue working on related things and meeting every Thursday to discuss and collaborate. On Thursday April 24th, we will present an overview of what we have thought about over the past few months. If anyone has interest in joining that weekly research group, this would be a good time to attend to get a sense of the group and see if the project is for you. Calendar link: [https://calendar.app.google/bFcXqVBx1wGdy8vx7](https://calendar.app.google/bFcXqVBx1wGdy8vx7)
# Understanding Trust: Overview Presentations

Since mid-January, I've been working with four students - Hanna Gabor, Roman Malov, Norman Hsia, Paul Rapoport - on the research agenda I [articulated last year](https://www.lesswrong.com/posts/7AzexLYpXKMqevttN/seeking-collaborators). The AI Safety Camp project is wrapping up, but most of us plan to continue working on related things and meeting every Thursday to discuss and collaborate. On Thursday April 24th, we will present an overview of what we have thought about over the past few months. If anyone has interest in joining that weekly research group, this would be a good time to attend to get a sense of the group and see if the project is for you. Calendar link: [https://calendar.app.google/bFcXqVBx1wGdy8vx7](https://calendar.app.google/bFcXqVBx1wGdy8vx7)
# Host Keys and SSHing to EC2

I do a lot of work on EC2, where I [ssh into a few instances I use for specific purposes](https://www.jefftk.com/p/ec2-scripts). Each time I did this I'd get a prompt like:

$ ssh_ec2nf
The authenticity of host 'ec2-54-224-39-217.compute-1.amazonaws.com
(54.224.39.217)' can't be established.
ED25519 key fingerprint is SHA256:...
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:591: ec2-18-208-226-191.compute-1.amazonaws.com
    ~/.ssh/known_hosts:594: ec2-54-162-24-54.compute-1.amazonaws.com
    ~/.ssh/known_hosts:595: ec2-54-92-171-153.compute-1.amazonaws.com
    ~/.ssh/known_hosts:596: ec2-3-88-72-156.compute-1.amazonaws.com
    ~/.ssh/known_hosts:598: ec2-3-82-12-101.compute-1.amazonaws.com
    ~/.ssh/known_hosts:600: ec2-3-94-81-150.compute-1.amazonaws.com
    ~/.ssh/known_hosts:601: ec2-18-234-179-96.compute-1.amazonaws.com
    ~/.ssh/known_hosts:602: ec2-18-232-154-156.compute-1.amazonaws.com
    (185 additional names omitted)
Are you sure you want to continue connecting (yes/no/\[fingerprint\])?

The issue is that each time I start my instance it gets a new hostname (which is just derived from the IP) and so SSH's [trust on first use](https://en.wikipedia.org/wiki/Trust_on_first_use) doesn't work properly.

Checking that "185 additional names omitted" is about the number I'd expect to see is ok, but not great. And it delays login.

I figured out how to fix this today:

1.  Edit `~/.ssh/known_hosts` to add an entry for each EC2 host I use under my alias for it. So I have `c2-44-222-215-215.compute-1.amazonaws.com ssh-ed25519 AAAA...` and I duplicate that to add `ec2nf ssh-ed25519 AAAA...` etc.
    
2.  Modify my [ec2 ssh script](https://www.jefftk.com/p/ec2-scripts) to set `HostKeyAlias`: `ssh -o "StrictHostKeyChecking=yes" -o "HostKeyAlias=ec2nf" ...`
    

More secure *and* more convenient!

(What got me to fix this was an interaction with my [auto-shutdown script](https://www.jefftk.com/p/auto-shutdown-script), where if I did `start_ec2nf && sleep 20 && ssh_ec2nf` but then went and did something else for a minute or two the machine would often turn itself off before I came back and got around to saying `yes`.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0iBXzuN7RPjR7qrKN3f2o9DaFk5Q2ACoFGQzokk56HJnB9wDAd7Wxhqy5mbMjx2XVl), [mastodon](https://mastodon.mit.edu/@jefftk/114353928775907160), [bluesky](https://bsky.app/profile/jefftk.com/post/3lmzf43uabk27)*
# The Russell Conjugation Illuminator

Bertrand Russell noted how people often describe the same factual behavior using emotionally opposite language depending on perspective — e.g. I am firm, you are obstinate, he is pigheaded. This framing tactic is now called a Russell Conjugation, and once you start noticing them, they’re everywhere — especially in politics and media.

For the past year and a half, I’ve been training a finetuned ChatGPT model, and building a tool to automatically highlight Russell Conjugations in text and suggest emotionally opposite alternatives. It functions as a fact-independent bias reverser — showing where emotional spin might exist, and how the opposite side might see an issue, regardless of the factual accuracy of specific claims. I find it valuable especially when trying to parse tribal political language, as very often different sides of political divides will use words that feel completely different to describe the same things.

Here’s an example I pasted into the tool: “The senator remained pigheaded despite expert consensus.” [https://russellconjugations.com/conj/26d26d8dc45728517a97483248845aab](https://russellconjugations.com/conj/26d26d8dc45728517a97483248845aab)

The word "pigheaded" gets flagged negative, and "expert consensus" gets flagged positive, and you’re given alternatives which describe the same or very similar behaviors but cast them in a different light.

I am new to LessWrong, but I noticed there has been some discussion of this topic on this platform before. [@Daniel Kokotajlo](https://www.lesswrong.com/users/daniel-kokotajlo?mention=user) created a thread of Russell Conjugations which I unfortunately was unaware of until just a few weeks ago, though I am now working some of those examples into my next training set.

I think this is a fascinating and important aspect of modern communication, and I'd like to try to spread the word as much as possible.

If you’re curious, you can try it out here: [https://russellconjugations.com](https://russellconjugations.com)

I’d love to hear any thoughts. Let me know if it highlights anything interesting (or weird). I’m still refining the model and always looking to improve it.

Thank you.
# British and American Connotations

As an American who works with some people who speak British English, the language differences are usually not a problem. Most words mean the same thing, and those that don't are usually concrete enough not to cause confusion (ex: lift, flat, chips). The tricky ones, though, are the ones that differ primarily in connotations. For example:

*   In American English (AE), "quite" is an intensifier, while in British English (BE) it's a mild deintensifier. So "quite good" is "very good" in AE but "somewhat good" in BE. I think "rather" works similarly, though it's less common in AE and I don't have a great sense for it.
    
*   "Scheme" has connotations of deviousness in AE, but is neutral in BE. Describing a plans or system as a "scheme" is common in BE and negative in AE.
    
*   "Graft" implies corruption in AE but hard work in BE.
    

These can cause silent misunderstandings where two people have very different ideas about the other's view:

> A: "I can't believe how much graft there was in the procurement process!"
> 
> B: "Yes, quite impressive. Rather keen on going above and beyond, aren't they?"
> 
> A: "And did you see the pension scheme they set up?"
> 
> B: "Sounds like they'll be quite well off when they'll leave office."

In this example A leaves thinking B approves of the corruption, while B doesn't realize there was any. It could be a long time, if ever, before they realize they misunderstood each other.

Are there other words people have run into that differ like this?
# AI Advances and Detection Strategy

*Cross-posted from my [NAO Notebook](https://data.securebio.org/jefftk-notebook/ai-and-detection).*

*This is an internal strategy note I wrote in November 2024 that I'm making public with some light editing.*

In my work at the NAO I've been thinking about what I expect to see as LLMs continue to become more capable and get closer to where they can significantly accelerate their own development. I think we may see [very large advances](https://ai-2027.com/) in the power of these systems [over the next few years](https://80000hours.org/agi/guide/when-will-agi-arrive/).

I'd previously thought that the main impact of AI on the NAO was through accelerating potential adversaries, and so shorter timelines primarily meant **more urgency**: we needed to get a comprehensive detection system in place quickly.

I now think, however, that this also means the best response involves some **reprioritization**. Specifically, AI will likely speed up some aspects of the creation of a detection system more than others, and so to the extent that we expect rapid advances in AI we should prioritize the work that we expect to bottleneck our future AI-accelerated work.

One way to plan for this is to imagine what would be the main bottlenecks if we had a far larger staff. Imagine if each senior person had AI support equivalent to all the smart junior people they could effectively manage. Or even (but my argument doesn't depend on this) AI systems that are as capable as today's experienced researchers. I think if in a year or two we found ourselves in this situation we would wish that:

*   We had collected a lot more data, because with a very large virtual computational staff future-AI-assisted-NAO can wring insights out of data far more efficiently than present-NAO.
    
*   We had started large-scale collection sooner, because, even if AI accelerates sequencing's price decreases and we can collect a lot more data in the future, it can't give us historical data.
    
*   We had a lot more partnerships for bringing in samples and data, because these take real-human time to scale up.
    

While I don't think this is the only way things could play out, I think it's likely enough that we should be taking these considerations very seriously in our planning.

*April 2025: since initially drafting this we've started an ambitious effort to [scale up our pilot system](https://naobservatory.org/blog/scaling-our-early-warning-system/).*

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid01A4pfLs9bR2Ltp9WoksBPVVP7GsaEtZRAesYc6deNoQ7fhgZDxfhrFod4mxtwsYGl), [mastodon](https://mastodon.mit.edu/@jefftk/114364425903054420), [bluesky](https://bsky.app/profile/jefftk.com/post/3ln62bn4afk2t)*
# Why Should I Assume CCP AGI is Worse Than USG AGI?

Though, given my doomerism, I think the natsec framing of the AGI race is likely wrongheaded, let me accept the Dario/Leopold/Altman frame that AGI will be aligned with the national interest of a great power. These people seem to take as an axiom that a USG AGI will be better in some way than a CCP AGI. Has anyone written a justification for this assumption?

I am neither an American citizen nor a Chinese citizen.

What would it mean for an AGI to be aligned with "Democracy," or "Confucianism," or "Marxism with Chinese characteristics," or "the American Constitution"? Contingent on a world where such an entity exists and is compatible with my existence, what would my life be like in a weird transhuman future as a non-citizen in each system? Why should I expect a USG AGI to be better than a CCP AGI? It does not seem to me super obvious that I should cheer for either party over the other. And if the intelligence of the governing class is of any relevance to the likelihood of a positive outcome, um, CCP seems to have USG beat hands down.