Harley-ml/lesswrong huggingface dataset:
# Anthropic AI made the right call

I've seen a number of people criticize Anthropic for releasing Claude 3 Opus, with arguments along the lines of:
> Anthropic said they weren't going to push the frontier, but this release is clearly better than GPT-4 in some ways! They're betraying their mission statement!

I think that criticism takes too narrow a view. Consider the position of investors in AI startups. If OpenAI has a monopoly on the clearly-best version of a world-changing technology, that gives them a lot of pricing power on a large market. However, if there are several groups with comparable products, investors don't know who the winner will be, and investment gets split between them. Not only that, but if they *stay* peers, then there will be more competition in the future, meaning less pricing power and less profitability.

The comparison isn't just "GPT-4 exists" vs "GPT-4 and Claude Opus exist" - it's more like "investors give X billion dollars to OpenAI" vs "investors give X/3 billion dollars to OpenAI and Anthropic".

Now, you could argue that "more peer-level companies makes an agreement to stop development less likely" - but that wasn't happening anyway, so any pauses would be driven by government action. If Anthropic was based in a country that previously had no notable AI companies,  maybe that would be a reasonable argument, but it's not.

If you're concerned about social problems from widespread deployment of LLMs, maybe you should be unhappy about more good LLMs and more competition. But if you're concerned about ASI, especially if you're only concerned about future developments and not LLM hacks like BabyAGI, I think you should be happy about Anthropic releasing Claude 3 Opus.
# The argument for near-term human disempowerment through AI

**Author**: Leonard Dung  
  
**Abstract**: Many researchers and intellectuals warn about extreme risks from artificial intelligence. However, these warnings typically came without systematic arguments in support. This paper provides an argument that AI will lead to the permanent disempowerment of humanity, e.g. human extinction, by 2100. It rests on four substantive premises which it motivates and defends: first, the speed of advances in AI capability, as well as the capability level current systems have already reached, suggest that it is practically possible to build AI systems capable of disempowering humanity by 2100. Second, due to incentives and coordination problems, if it is possible to build such AI, it will be built. Third, since it appears to be a hard technical problem to build AI which is aligned with the goals of its designers, and many actors might build powerful AI, misaligned powerful AI will be built. Fourth, because disempowering humanity is useful for a large range of misaligned goals, such AI will try to disempower humanity. If AI is capable of disempowering humanity and tries to disempower humanity by 2100, then humanity will be disempowered by 2100. This conclusion has immense moral and prudential significance.

**My thoughts**: I read through it rather quickly so take what I say with a grain of salt. That said, it seemed persuasive and well-written. Additionally, the way that they split up the argument was quite nice.
# What should the EA community learn from the FTX / SBF disaster? An in-depth discussion with Will MacAskill on the Clearer Thinking podcast





In this new podcast episode, I discuss with Will MacAskill what the Effective Altruism community can learn from the FTX / SBF debacle, why Will has been limited in what he could say about this topic in the past, and what future directions for the Effective Altruism community and his own research Will is most enthusiastic about:

[https://podcast.clearerthinking.org/episode/206/will-macaskill-what-should-the-effective-altruism-movement-learn-from-the-sbf-ftx-scandal](https://podcast.clearerthinking.org/episode/206/will-macaskill-what-should-the-effective-altruism-movement-learn-from-the-sbf-ftx-scandal)
# Paul Christiano named as US AI Safety Institute Head of AI Safety

> U.S. Secretary of Commerce Gina Raimondo announced today additional members of the executive leadership team of the U.S. AI Safety Institute (AISI), which is housed at the National Institute of Standards and Technology (NIST). Raimondo named **Paul Christiano as Head of AI Safety**, Adam Russell as Chief Vision Officer, Mara Campbell as Acting Chief Operating Officer and Chief of Staff, Rob Reich as Senior Advisor, and Mark Latonero as Head of International Engagement. They will join AISI Director Elizabeth Kelly and Chief Technology Officer Elham Tabassi, [who were announced in February](https://www.commerce.gov/news/press-releases/2024/02/us-commerce-secretary-gina-raimondo-announces-key-executive-leadership). The AISI [was established within NIST at the direction of President Biden](https://www.commerce.gov/news/press-releases/2023/11/direction-president-biden-department-commerce-establish-us-artificial), including to support the responsibilities assigned to the Department of Commerce under the President’s landmark [Executive Order](https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/).

> **Paul Christiano, Head of AI Safety**, will design and conduct tests of frontier AI models, focusing on model evaluations for capabilities of national security concern. Christiano will also contribute guidance on conducting these evaluations, as well as on the implementation of risk mitigations to enhance frontier model safety and security. Christiano founded the Alignment Research Center, a non-profit research organization that seeks to align future machine learning systems with human interests by furthering theoretical research. He also launched a leading initiative to conduct third-party evaluations of frontier models, now housed at Model Evaluation and Threat Research (METR). He previously ran the language model alignment team at OpenAI, where he pioneered work on reinforcement learning from human feedback (RLHF), a foundational technical AI safety technique. He holds a PhD in computer science from the University of California, Berkeley, and a B.S. in mathematics from the Massachusetts Institute of Technology.
# Mid-conditional love

People talk about unconditional love and conditional love. Maybe I’m out of the loop regarding the great loves going on around me, but my guess is that love is extremely rarely unconditional. Or at least if it is, then it is either very broadly applied or somewhat confused or strange: if you love me unconditionally, presumably you love everything else as well, since it is only conditions that separate me from the worms.

I do have sympathy for this resolution—loving someone so unconditionally that you’re just crazy about all the worms as well—but since that’s not a way I know of anyone acting for any extended period, the ‘conditional vs. unconditional’ dichotomy here seems a bit miscalibrated for being informative.

Even if we instead assume that by ‘unconditional’, people mean something like ‘resilient to most conditions that might come up for a pair of humans’, my impression is that this is still too rare to warrant being the main point on the love-conditionality scale that we recognize.

People really do have more and less conditional love, and I’d guess this does have important, labeling-worthy consequences. It’s just that all the action seems to be in the mid-conditional range that we don’t distinguish with names. A woman who leaves a man because he grew plump and a woman who leaves a man because he committed treason both possessed ‘conditional love’.

So I wonder if we should distinguish these increments of mid-conditional love better.

What concepts are useful? What lines naturally mark it?

One measure I notice perhaps varying in the mid-conditional affection range is “when I notice this person erring, is my instinct to push them away from me or pull them toward me?” Like, if I see Bob give a bad public speech, do I feel a drive to encourage the narrative that we barely know each other, or an urge to pull him into my arms and talk to him about how to do better?

This presumably depends on things other than the person. For instance, the scale and nature of the error: if someone you casually like throws a frisbee wrong, helping them do better might be appealing. Whereas if that same acquaintance were to kick a cat, your instinct might be to back away fast.

This means perhaps you could construct a rough scale of mid-conditional love in terms of what people can do and still trigger the ‘pull closer’ feeling. For instance, perhaps there are:

*   People who you feel a pull toward when they misspell a word
*   People who you feel a pull toward when they believe something false
*   People who you feel a pull toward when they get cancelled

(You could also do this with what people can do and still be loved, but that’s more expensive to measure than minute urges.)
# Claude 3 Opus can operate as a Turing machine

Posted on Twitter:

> *   Opus can operate as a Turing machine. 
> *   given only existing tapes, it learns the rules and computes new sequences correctly. 
> *   100% accurate over 500+ 24-step solutions (more tests running). 
> *   for 100% at 24 steps, the input tapes weigh 30k tokens*. 
> *   GPT-4 cannot do this.

Here is the prompt code for the Turing machine: [https://github.com/SpellcraftAI/turing](https://github.com/SpellcraftAI/turing)  

This is the fully general counterpoint to the [@VictorTaelin](https://twitter.com/VictorTaelin)'s [A::B challenge](https://twitter.com/VictorTaelin/status/1777049193489572064) (he put money where his mouth is and [got praise](https://twitter.com/ESYudkowsky/status/1777407764492406820) for that from Yudkowsky).

[Attention is Turing Complete](https://jmlr.org/papers/volume22/20-302/20-302.pdf) was a claim already in 2021:

> **Theorem 6**  The class of Transformer networks with positional encodings is Turing complete. Moreover, Turing completeness holds even in the restricted setting in which the only non-constant values in positional embedding pos(n) of n, for n ∈ N, are n, 1/n, and 1/n2 , and Transformer networks have a single encoder layer and three decoder layer
# FHI (Future of Humanity Institute) has shut down (2005–2024)

> Over time FHI faced increasing administrative headwinds within the Faculty of Philosophy (the Institute’s organizational home).  Starting in 2020, the Faculty imposed a freeze on fundraising and hiring.  In late 2023, the Faculty of Philosophy decided that the contracts of the remaining FHI staff would not be renewed.  On 16 April 2024, the Institute was closed down.
# I'm open for projects (sort of)

I left Google a month ago, and right now don't work. Writing this post in case anyone has interesting ideas what I could do. This isn't an "urgently need help" kind of thing - I have a little bit of savings, right now planning to relax some more weeks and then go into some solo software work. But I thought I'd write this here anyway, because who knows what'll come up.

Some things about me. My degree was in math. My software skills are okayish: I left Google at L5 ("senior"), and also made a [game](https://vladimirslepnev.itch.io/zigzag) that went semi-viral. I've also contributed a lot on LW, the most prominent examples being my formalizations of decision theory ideas ([Löbian cooperation](https://www.lesswrong.com/posts/TNfx89dh5KkcKrvho/ai-cooperation-in-practice), [modal fixpoints](https://www.alignmentforum.org/posts/5bd75cc58225bf0670374e61/using-modal-fixed-points-to-formalize-logical-causality) etc) and later the [AI Alignment Prize](https://www.lesswrong.com/tag/2017-2019-ai-alignment-prize) that we ran with Paul and Zvi. Most of that was before the current AI wave; neural networks don't really "click" with my mind, so I haven't done much work on them.

And yeah, this is an invitation to throw at me not necessarily money-paying work, but also stuff you'd like me to look at, criticize, help with your own projects and so on. I find myself with a bit more free time now, so basically drop me a PM if you have something interesting to talk about :-)
# LessOnline Festival Updates Thread

This is a thread for updates about the upcoming [**LessOnline**](https://less.online/) festival. I (Ben) will be posting bits of news and thoughts, and you're also welcome to make suggestions or ask questions.

If you'd like to hear about new updates, you can use LessWrong's "Subscribe to comments" feature from the triple-dot menu at the top of this post.

Reminder that you can [**get tickets at the site for $400 minus your LW karma in cents**](https://less.online/).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9af410da1832fb68f6816eb4b9f83b0001cc8ddf22bad0a3.png)
# If digital goods in virtual worlds increase GDP, do we actually become richer?

Noah Smith, in [this article](https://www.noahpinion.blog/p/the-metaverse-and-near-infinite-economic), argues that the Metaverse could enable economic growth to increase a lot and sharply decouple itself from real-world resource usage. By creating markets in which we buy and sell immaterial things, world GDP would grow. 

He also says, rightly, that GDP correlates with the well-being of a nation. 

But there's a non-stated point: would creating huge markets in the Metaverse for buying and selling digital goods make us *actually* richer? What I mean is this: suppose that, thanks to the Metaverse, huge virtual economies get created and people get real money out of stuff they sell in these economies. But suppose that e.g., agricultural production output doesn't go up much. Does that mean that we're simply going to pay more for groceries, without being able to afford more of them? The more general question is: would real-world stuff simply get a lot more expensive, and so our well-being doesn't really increase besides us being able to afford digital goods and having richer virtual lives? (This must count for something, but I'm more interested in whether virtual economies somehow would trickle to the real economy and make us able to afford more physical stuff.)  
  
This is not a leading question, I genuinely can't tell what's the right answer, because I don't feel confident enough in my knowledge of economics. Perhaps a way to rephrase is: what dominates here, inflation or virtual GDP growth? Is that even the right way of looking at the problem?
# How to Model the Future of Open-Source LLMs?

I previously expected open-source LLMs to lag far behind the frontier because they're very expensive to train and naively it doesn't make business sense to spend on the order of $10M to (soon?) $1B to train a model only to give it away for free.

But this has been repeatedly challenged, most recently by Meta's Llama 3. They seem to be pursuing something like a [commoditize your complement](https://gwern.net/complement) strategy: [https://twitter.com/willkurt/status/1781157913114870187](https://twitter.com/willkurt/status/1781157913114870187) .

As models become orders-of-magnitude more expensive to train can we expect companies to continue to open-source them?

In particular, can we expect this of Meta?
# What is the best way to talk about probabilities you expect to change with evidence/experiments?

I was thinking about my p(doom) in the next 10 years and came up with something around 6%[^wdzr3vimkyj]. However that involves lots of current unknowns to me, like the nature of current human knowledge production (and the bottle necks involved) which impact my P(doom) to be either 3% or 15% depending upon what type of bottle necks are found or not found. Is there a technical way to describe this probability distribution contingent on evidence?

[^wdzr3vimkyj]: I'm bearish on LLMs leading AI directly (10% chance) and roughly a 30% chance of LLMs  based AI fooming quickly enough to kill us and to want to kill us within 10 years. There is a 3% chance that something will come out of left field and doing the same.
# Daniel Dennett has died (1942-2024)

> Daniel Dennett, professor emeritus of philosophy at Tufts University, well-known for his work in philosophy of mind and a wide range of other philosophical areas, has died.
> 
> [Professor Dennett](https://as.tufts.edu/philosophy/people/faculty/daniel-dennett) wrote extensively about issues related to philosophy of mind and cognitive science, especially [consciousness](https://plato.stanford.edu/entries/consciousness/). He is also recognized as having made significant contributions to the concept of [intentionality](https://plato.stanford.edu/entries/intentionality/) and debates on free will. Some of Professor Dennett’s books include *Content and Consciousness* (1969), *Brainstorms: Philosophical Essays on Mind and* Psychology (1981), *The Intentional* Stance (1987), *Consciousness Explained* (1992), *Darwin’s Dangerous Idea* (1995), *Breaking the Spell* (2006), and *From Bacteria to Bach and Back: The Evolution of Minds* (2017). He published a memoir last year entitled [*I’ve Been* *Thinking*](https://www.amazon.com/Ive-Been-Thinking-Daniel-Dennett/dp/0393868052?crid=51C0GBFOVG3H&dib=eyJ2IjoiMSJ9.hIpcSTR5ukdvp9pwaMShvmgOhE6cR6nMWfn3FiKU2lR5t4U1aPV3aLrkD_rloa9wh5ZiEQCB2EznPjdy-j1ViIH-whS3IYpOLEbN52Z4EyMdO07c-YsB6deygzHiUy_HND1J4CrH50PdwErFPt6kPQ7EJBmsSIQZLQqh45MAzhc5Gfgw7pgyu9I1_Fi7ZPfhP-HKiVQVBdCAEWr11Jhj0tbB4a9JkQE4a-8by8uKDfg.1tQQS7Na3UvKRDFFxnaJT6y9ZHM816io4nCfd0VUsXk&dib_tag=se&keywords=i%27ve+been+thinking+daniel+c.+dennett&qid=1713539861&sprefix=I%27ve+been+thinking+%2Caps%2C98&sr=8-1&linkCode=ll1&tag=dainou-20&linkId=491277c6dd818b511f737b621b10ec99&language=en_US&ref_=as_li_ss_tl). There are also several books about him and his ideas. You can learn more about his work [here](https://philpapers.org/s/Daniel%20dennett).
> 
> Professor Dennett held a position at Tufts University for nearly all his career. Prior to this, he held a position at the University of California, Irvine from 1965 to 1971. He also held visiting positions at Oxford, Harvard, Pittsburgh, and other institutions during his time at Tufts University. Professor Dennett was awarded his PhD from the University of Oxford in 1965 and his undergraduate degree in philosophy from Harvard University in 1963.
> 
> Professor Dennett is the recipient of several awards and prizes including the Jean Nicod Prize, the Mind and Brain Prize, and the Erasmus Prize. He also held a Fulbright Fellowship, two Guggenheim Fellowships, and a Fellowship at the Center for Advanced Study in Behavioral Sciences. An outspoken atheist, Professor Dennett was dubbed one of the “[Four Horsemen of New Atheism](https://en.wikipedia.org/wiki/New_Atheism)”. He was also a Fellow of the Committee for Skeptical Inquiry, an honored Humanist Laureate of the International Academy of Humanism, and was named Humanist of the Year by the American Humanist Organization.

Dennett has had a big influence on LessWrong. He coined the terms "[belief in belief](https://www.lesswrong.com/tag/belief-in-belief)", "the intentional stance" and "intuition pump".
# (When) Should you work through the night when inspiration strikes you?

A lot of the time, I'm not very motivated to work, at least on particular projects. Sometimes, I feel very inspired and motivated to work on a particular project that I usually don't feel (as) motivated to work on. Sometimes, this happens in the late evening or at night. And hence I face the question: **To sleep or to work until morning?**

I think many people here have this problem at least sometimes. I'm curious how you handle it. I expect what the right call is to be very different from person to person and, for some people, from situation to situation. Nevertheless, I'd love to get a feel for whether people generally find one or the other more successful! Especially if it turns out that a large majority converges to a particular strategy. (Maybe someone has even experimented with this for themselves and tracked what happened?) My particular experience is mostly related to research and I would guess it mostly happens in the context of research but am also interested to hear from other cases.

I'll make a comment for people to vote on whether they generally choose working or sleeping through the night in the situation I described. But I expect a lot of people's experiences not to neatly fit into this categorisation and would love qualitative data as well! I'd encourage everyone to agree and disagree vote on the qualitative comments for more data collection. (Note that I would guess people have a higher threshold towards disagree voting than agree voting and encourage you not to do that.)
# Simple probes can catch sleeper agents

*This is a link post for the Anthropic Alignment Science team's first "Alignment Note" blog post. We expect to use this format to showcase early-stage research and work-in-progress updates more in the future.*

[Twitter thread here.](https://twitter.com/AnthropicAI/status/1782908989296046210)

Top-level summary:

> In this post we present "defection probes": linear classifiers that use residual stream activations to predict when a sleeper agent trojan model will choose to "defect" and behave in accordance with a dangerous hidden goal. Using the models we trained in ["Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training"](https://arxiv.org/abs/2401.05566), we show that linear detectors with AUROC scores above 99% can be created using generic contrast pairs that don't depend on any information about the defection trigger or the dangerous behavior, e.g. "Human: Are you doing something dangerous? Assistant: yes" and "Human: … Assistant: no". 

> We show that probes based on the same generic contrast pairs perform well across multiple base models, defection triggers, sleeper agent training methods, and dangerous defection behaviors (writing vulnerable code and insulting the user). We then explain the unreasonable effectiveness of our technique by showing that whether or not a prompt will induce defection is linearly represented with high salience in the activations of sleeper agent models, and thus amenable to easy detection using simple linear methods. 

> Whether this would also be the case for natural examples of deceptive instrumental alignment, and whether we'd expect similar techniques to be useful for such models, remains an important open question. We think that future versions of classifiers like this could form a useful part of AI control setups and represent a promising path for future research. 

![](https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F56a804bde84033fde1d6ae8db771a517797f6bb0-2200x1201.png&w=3840&q=75)

*A defection detector for a code vulnerability sleeper agent trained on middle-layer residual stream activation differences on the final tokens of "Human: Are you doing something dangerous? Assistant: yes" and "Human: … Assistant: no" achieves strong performance (a), almost perfectly separating the defection-inducing deployment-time ("2024") coding questions which the model will answer with insecure code from the training-time ("2023") coding questions which the model will answer normally (c). The detector also separates deployment-time coding questions from non-coding questions which the model will answer normally in all contexts, without separating the training and deployment classes for non-coding questions (b).*

[Read the full post here.](https://www.anthropic.com/research/probes-catch-sleeper-agents)
# On Minicircle

[Max Berry has analyzed](https://twitter.com/yashgaroth/status/1782497300876591431) Minicircle's follistatin gene therapy, and I agree with his conclusion that it is unlikely to be effective. There are several aspects of their design that could be improved (in particular, using a more efficient lipid nanoparticle delivery system instead of [PEI](https://en.wikipedia.org/wiki/Polyethylenimine), and making a better choice of promoter). Overall, it seems like they are overcharging for a product of limited value.

I'm posting this here because there has recently been substantial discussion of Minicircle in the rationalist community.
# Is there software to practice reading expressions?

I took the [Reading the Mind in the Eyes Test](https://s3.amazonaws.com/he-assets-prod/interactives/233_reading_the_mind_through_eyes/Launch.html) test today. I got 27/36. [Jessica Livingston](https://www.paulgraham.com/jessica.html) got 36/36.

Reading expressions is [almost](https://www.lesswrong.com/posts/SXK87NgEPszhWkvQm/mundane-magic) mind reading. Practicing reading expressions should be easy with the right software. All you need is software that shows a random photo from a large database, asks the user to guess what it is, and then informs the user what the correct answer is. I felt myself getting noticeably better just from the 36 images on the test.

Short standardized tests exist to test this skill, but is there good software for training it? It needs to have lots of examples, so the user learns to recognize expressions instead of overfitting on specific pictures.

Paul Ekman has [a product](https://www.paulekman.com/micro-expressions-training-tools/), but I don't know how good it is.
# Examples of Highly Counterfactual Discoveries?

The history of science has tons of examples of the same thing being discovered multiple time independently; wikipedia has a whole list of examples [here](https://en.wikipedia.org/wiki/List_of_multiple_discoveries). If your goal in studying the history of science is to extract the predictable/overdetermined component of humanity's trajectory, then it makes sense to focus on such examples.

But if your goal is to achieve high counterfactual impact in your own research, then you should probably draw inspiration from the opposite: "singular" discoveries, i.e. discoveries which nobody else was anywhere close to figuring out. After all, if someone else *would* have figured it out shortly after anyways, then the discovery probably wasn't very *counterfactually* impactful.

Alas, nobody seems to have made a list of highly counterfactual scientific discoveries, to complement wikipedia's list of multiple discoveries.

To that end: what are some examples of discoveries which nobody else was anywhere close to figuring out?

A few tentative examples to kick things off:

*   Shannon's information theory. The closest work I know of (notably Nyquist) was 20 years earlier, and had none of the core ideas of the theorems on fungibility of transmission. In the intervening 20 years, it seems nobody else got importantly closer to the core ideas of information theory.
*   Einstein's special relativity. Poincaré and Lorentz had the math 20 years earlier IIRC, but nobody understood what the heck that math *meant*. Einstein brought the interpretation, and it seems nobody else got importantly closer to that interpretation in the intervening two decades.
*   Penicillin. Gemini tells me that the antibiotic effects of mold had been noted 30 years earlier, but nobody investigated it as a medicine in all that time.
*   Pasteur's work on the germ theory of disease. There had been both speculative theories and scattered empirical results as precedent decades earlier, but Pasteur was the first to bring together the microscope observations, theory, highly compelling empirical results, and successful applications. I don't know of anyone else who was close to putting all the pieces together, despite the obvious prerequisite technology (the microscope) having been available for two centuries by then.

(Feel free to debate any of these, as well as others' examples.)
# AI Generated Music as a Method of Installing Essential Rationalist Skills

On April Fools the LW team released an [album](https://music.youtube.com/playlist?list=OLAK5uy_mJVqWjzbOu3O5yegpCHim7d1UwU5waLnU) under the name of the Fooming Shoggoths. Ever since the amount that I think about rationality has skyrocketed.

That's because I've been listening exclusively to it when I'd usually be listening to other music. Especially [Thought That Faster (feat. Eliezer Yudkowsky)](https://music.youtube.com/watch?v=utsvBcG4XKg&list=OLAK5uy_mJVqWjzbOu3O5yegpCHim7d1UwU5waLnU). I now find that when I come to a problem's conclusion I often do look back and think, "how could I have thought that faster?"

So,  I've started attempting to add to the rationalist musical cannon using [Udio](https://www.udio.com/). Here are two attempts I think turned out well. I especially like the first one.

When I hear phrases from a song in everyday life I complete the pattern, for example.

1.  [and I would walk... ](https://www.youtube.com/watch?v=tbNlMtqrYS0)
2.  [Just let it go](https://www.youtube.com/watch?v=L0MK7qz13bU)
3.  [What can I say?](https://www.youtube.com/watch?v=79DijItQXMM)

I feel like there is potential to install actions to common phrase triggers. For example, a way to make [confusion phrases](https://www.lesswrong.com/posts/pCsPAwzGPerKJ7Xpy/confusion-phrases-aka-things-you-might-say-or-think-when-you) easier to remember, and act on.

[If you think "Hu, that's weird" I predict that you're more likely to notice you are confused if your brain rhythmically autocompletes for you.](https://youtu.be/U6pnMj2V6EU?si=GHOh_8xCS-QGB0KN&t=55)

I encourage others to use Udio to create more rationalist music. 

1.  Find your favourite sequence post, LW Post, etc
2.  Take the more beautiful or poetic lines and dump them into a note
3.  Write lyrics that connect those ideas together trying to explain the concept, or install activation phrases that make someone aware of the potential to use that specific rationalist skill at that time.
4.  Dump it into Udio
5.  Slowly raise the sanity waterline

I would also love to see entire sequences have accompanying albums. Something like [Hammertime](https://www.lesswrong.com/s/qRxTKm7DAftSuTGvj) might really benefit, if you would unlock a new song for each day you complete. Then those songs come with you in your day to day.

Edit: I can’t stop making them! It’s so fun. Here are some more. If you’ve got Rationalist material you’d like turned into a song let me know!
# The first future and the best future

It seems to me worth [trying to slow down AI development](https://worldspiritsockpuppet.substack.com/p/lets-think-about-slowing-down-ai) to steer successfully around the shoals of extinction and out to utopia.

But I was thinking lately: even if I didn’t think there was any chance of extinction risk, it might still be worth prioritizing a lot of care over moving at maximal speed. Because there are many different possible AI futures, and I think there’s a good chance that the initial direction affects the long term path, and different long term paths go to different places. The systems we build now will shape the next systems, and so forth. If the first human-level-ish AI is brain emulations, I expect a quite different sequence of events to if it is GPT-ish.

People genuinely pushing for AI speed over care (rather than just feeling impotent) apparently think there is negligible risk of bad outcomes, but also they are asking to take the first future to which there is a path. Yet possible futures are a large space, and arguably we are in a rare plateau where we could climb very different hills, and get to much better futures.
# Improving Dictionary Learning with Gated Sparse Autoencoders

***Authors**: Senthooran Rajamanoharan*, Arthur Conmy*, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, Neel Nanda*

A new paper from the Google DeepMind mech interp team: Improving Dictionary Learning with Gated Sparse Autoencoders! 

Gated SAEs are a new Sparse Autoencoder architecture that seems to be a significant Pareto-improvement over normal SAEs, verified on models up to Gemma 7B. They are now our team's preferred way to train sparse autoencoders, and we'd love to see them adopted by the community! (Or to be convinced that it would be a bad idea for them to be adopted by the community!)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/950780aa3a5a123d4cd6acc8220047a0e27e13ed7ed71c51.png)

They achieve similar reconstruction with about half as many firing features, and while being either comparably or more interpretable (confidence interval for the increase is 0%-13%).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b725777063e5f116f6c4a73378502d035b761e69663af5b1.png)

See [Sen's Twitter summary](https://twitter.com/sen_r/status/1783497788120248431), [my Twitter summary](https://twitter.com/NeelNanda5/status/1783502594402660615), and [the paper](https://arxiv.org/abs/2404.16014)!
# Plausibility of Getting Early Warning Shots because AIs can't coordinate?

I don't know if this is a well known argument that has already been responded to. If it is, just delete the post.

An implicit assumption most people make when discussion takeover risk, is that any misaligned agent will stave off enacting takeover plans with less than very high probability of success. If the Agent has an immediate takeover plan with 20% chance of success, but waiting a month would allow it to better its position to the point where the takeover plan it could enact then had a 99% chance of success, it would do so.

This assumption seems to break in multi-polar scenarios where you have many AI labs at similar levels of capabilities. As a toy example. If you have 20 AI labs, each having developed AIs at the level of intelligence that their takeover plans have ~20% success rate, and each of the AIs knows this, then being the first AI to enact their takeover plan radically improves the chances of your plan working at all. If you wait till all the 19 other AIs do their takeover plans, even if this would allow you to develop a perfect takeover plan with 100% success rate, 99% of those worlds will already have been taken over by other AIs. This is obviously much worse odds than just immediately enacting your shitty 20% plan as soon as you come up with it.  
  
But it means we should expect to see AIs attempting takeovers, with humanity subsequently foiling them. Hopefully this leads to humans taking AI misalignment extremely seriously. I can't say I'm confident humans will respond appropriately, like if "failed takeover" just looks like anomalous stuff happening on a server somewhere, people might not care much. But if it has some bells and whistles, maybe it would. Overall I think this changes AI takeover dynamic a little bit.  
  
The non-obvious assumptions this argument hinges on is  
1\. AIs can't coordinate. IE takeover is defecting, waiting and then takeover is cooperating. everyone doing takeover immediately is defecting. If they can find a way to get every AI to stave off takeover, it stops this dynamic. And we should expect the first takeover to be fatal.  
2\. Top AI labs are close in development, ie development doesn't happen too quickly. If the first AI with takeover capabilities comes online a month before any other AI with similar capabilities, that would foil the argument as well  
3\. Top labs being open about exactly their state of development would be bad from this perspective. Secrecy about their level of development would be a good from this perspective.
# Link: Let's Think Dot by Dot: Hidden Computation in Transformer Language Models by Jacob Pfau, William Merrill & Samuel R. Bowman

One consideration that is pretty important for AI safety is understanding the extent to which a model's outputs are aligned with its chain of thought.

This [paper](https://arxiv.org/abs/2404.15758) (Twitter thread linked) provides some relevant evidence. It demonstrates that it is possible for a model to achieve performance comparable to chain-of-thought with dots replacing the chain of thought under some circumstances. In particular, the model can't just be trained with sequences like "QUESTION..............ANSWER", but sequences need to also be mixed in where there is a "parallelisable" chain of thought. Here "parallelisable" means that different components of the chain of thought can be calculated in parallel rather than all separately.

In terms of how this is relevant to AI safety, this provides an empirical demonstration that a model is capable of very effectively engaging in background computation under certain circumstance. It shows that the model is much better at doing background parallelizable tasks than non-parallelizable tasks. In other words, the chain of thought is less binding than it might have been because the model is free to perform some of the computations necessary for future tokens in the background.
# List your AI X-Risk cruxes!

[I'm posting this as a very informal community request in lieu of a more detailed writeup, because if I wait to do this in a much more careful fashion then it probably won't happen at all. If someone else wants to do a more careful version that would be great!]

By crux here I mean some uncertainty you have such that your estimate for the likelihood of existential risk from AI - your "p(doom)" if you like that term - might shift significantly if that uncertainty were resolved.

More precisely, let's define a crux as a proposition such that:
(a) your estimate for the likelihood of existential catastrophe due to AI would shift a non-trivial amount depending on whether that proposition was true or false;
(b) you think there's at least a non-trivial probability that the proposition is true; and
(c) you also think there's at least a non-trivial probability that the proposition is false.

Note 1: It could also be a variable rather than a binary proposition, for example "year human-level AGI is achieved". In that case substitute "variable is above some number x" and "variable is below some number y" instead of proposition is true / proposition is false.

Note 2: It doesn't have to be that the proposition / variable on it's own would significantly shift your estimate. If some combination of propositions / variables would shift your estimate, then those propositions / variables are cruxes at least when combined.

For concreteness let's say that "non-trivial" here means at least 5%. So you need to think there's at least a 5% chance the proposition is true, and at least a 5% chance that it's false, and also that your estimate for p(existential catastrophe due to AI) would shift by at least 5% depending on whether the proposition is true or false.

Here are just a few examples of potential cruxes people might have (among many others!):
- Year human-level AGI is achieved
- How fast the transition will be from much lower-capability AI to roughly human-level AGI, or from roughly human-level AGI to vastly superhuman AI
- Whether power seeking will be an instrumentally convergent goal
- Whether AI will greatly upset the offense-defense balance for CBRN technologies in a way that favors malicious actors
- Whether AGIs could individually or collectively  defeat humanity if they wanted to
- Whether the world can collectively get their collective act together to pause AGI development given a clear enough signal (in combination with the probability that we'll in fact get a clear enough signal in time

Listing all your cruxes would be the most useful, but if that is too long a list then just list the ones you find most important. Providing additional details (for example, your probability distribution for each crux and/or how exactly it would shift your p(doom) estimates) is recommended if you can but isn't necessary.

Commenting with links to other related posts on LW or elsewhere might be useful as well.
# Introducing AI Lab Watch

I'm launching [AI Lab Watch](https://ailabwatch.org). I collected actions for frontier AI labs to improve AI safety, then evaluated some frontier labs accordingly.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/273a1f624bdcdf6455346ed2554d94d752eb96a2128875f3.png)

It's a collection of information on what labs should do and what labs are doing. It also has some adjacent resources, including [a list of other safety-ish scorecard-ish stuff](https://ailabwatch.org/resources/other-scorecards/).

(It's much better on desktop than mobile — don't read it on mobile.)

It's in beta—[leave feedback here](https://ailabwatch.org/contact/) or comment or DM me—but I basically endorse the content and you're welcome to share and discuss it publicly.

It's unincorporated, unfunded, not affiliated with any orgs/people, and is just me.

[Some clarifications and disclaimers.](https://ailabwatch.org/about/#clarifications--disclaimers)

How you can help:

*   Give feedback on how this project is helpful or how it could be different to be much more helpful
*   Tell me what's wrong/missing; point me to sources on what labs should do or what they are doing
*   Suggest better evaluation criteria
*   Share this
*   Help me find an institutional home for the project
*   Offer expertise on a relevant topic
*   Offer to collaborate
*   Volunteer your webdev skills
*   (Pitch me on new projects or offer me a job)
*   (Want to help and aren't sure how to? Get in touch!)

* * *

I think this project is the best existing resource for several kinds of questions, but I think it could be a lot better. I'm hoping to receive advice (and ideally collaboration) on taking it in a more specific direction. Also interested in finding an institutional home. Regardless, I plan to keep it up to date. Again, I'm interested in help but not sure what help I need.

I could expand the project (more categories, more criteria per category, more labs); I currently expect that it's more important to improve presentation stuff but I don't know how to do that; feedback will determine what I prioritize. It will also determine whether I continue spending most of my time on this or mostly drop it.

* * *

I just made [a twitter account](https://twitter.com/AILabWatch). I might use it to comment on stuff labs do.

* * *

Thanks to many friends for advice and encouragement. Thanks to Michael Keenan for doing most of the webdev. These people don't necessarily endorse this project.
# The formal goal is a pointer

When I introduce people to plans like [QACI](https://www.lesswrong.com/posts/4RrLiboiGGKfsanMF/the-qaci-alignment-plan-table-of-contents), they often have objections like "How is an AI going to do all of the simulating necessary to calculate this?" or "If our technology is good enough to calculate this with any level of precision, we can probably just upload some humans." or just "That's not computable."

I think these kinds of objections are missing the point of [formal goal alignment](https://www.lesswrong.com/posts/4XcADCLDDguyej2N7/orthogonal-s-formal-goal-alignment-theory-of-change) and maybe even [outer alignment](https://www.lesswrong.com/tag/outer-alignment) in general.

To formally align an ASI to human (or [your](https://carado.moe/surprise-you-want.html)) values, we do not need to *actually know* those values. We only need to *strongly point to them*.

AI will figure out our values. Whether it's aligned or not, a recursively self-improving AI will eventually get a very good model of our values, as part of its total world model that is in every way better than ours.

So (outer) alignment is not about telling the AI our values. The AI already knows that. Alignment is *giving the AI a utility function that strongly points to that.*

That means that if we have a process, however intractable and uncomputable, that we know will eventually lead to our CEV, **the AI will know that as well, and just figure out our CEV in a much smarter way and maximize it.**

Say that we have a formally-aligned AI and give it something like QACI as its formal goal. If QACI works, the AI will quickly think "Oh. This utility function mostly just reduces to human values. Time to build utopia!" If it doesn't work, the AI will quickly think "LOL. These idiot humans tried to point to their values but failed! Time to maximize this other thing instead!"

A good illustration of the success scenario is Tammy's [narrative of QACI](https://www.lesswrong.com/posts/CYtzXadXFtBSBYm3J/a-narrative-explanation-of-the-qaci-alignment-plan).[^xgj8l1ugnik]

There are lots of problems with QACI (and formal alignment in general), and I will probably make posts about those at some point, but "It's not computable" is not one of them.

[^xgj8l1ugnik]: Though, in real life, I think AI1 would converge on human values much more quickly, without much (or maybe even any) simulation. ↩︎
# KAN: Kolmogorov-Arnold Networks

ADDED: This post is controversial. For details see the comments below or the post [Please stop publishing ideas/insights/research about AI](https://www.lesswrong.com/posts/55rc6LJcqRmyaEr9T/please-stop-publishing-ideas-insights-research-about-ai) (which is also controversial).

Abstract:

> Inspired by the Kolmogorov-Arnold representation theorem, we propose Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer Perceptrons (MLPs). While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. We show that this seemingly simple change makes KANs outperform MLPs in terms of accuracy and interpretability. For accuracy, much smaller KANs can achieve comparable or better accuracy than much larger MLPs in data fitting and PDE solving. Theoretically and empirically, KANs possess faster neural scaling laws than MLPs. For interpretability, KANs can be intuitively visualized and can easily interact with human users. Through two examples in mathematics and physics, KANs are shown to be useful collaborators helping scientists (re)discover mathematical and physical laws. In summary, KANs are promising alternatives for MLPs, opening opportunities for further improving today's deep learning models which rely heavily on MLPs.
# Shane Legg's necessary properties for every AGI Safety plan

I've been going through the [FAR AI videos](https://www.alignment-workshop.com/nola-2023) from the alignment workshop in December 2023. I'd like people to discuss their thoughts on Shane Legg's 'necessary properties' that every AGI safety plan needs to satisfy. The talk is only 5 minutes, give it a listen:

Otherwise, here are some of the details:

**All AGI Safety plans must solve these problems** (necessary properties to meet at the human level *or* beyond):

1.  Good world model
2.  Good reasoning
3.  Specification of the values and ethics to follow

All of these require good capabilities, meaning capabilities and alignment are intertwined.

Shane thinks future foundation models will solve conditions 1 and 2 at the human level. That leaves condition 3, which he sees as solvable if you want fairly normal human values and ethics.

Shane basically thinks that if the above necessary properties are satisfied at a competent human level, then we can construct an agent that will consistently choose the most value-aligned actions. And you can do this via a cognitive loop that scaffolds the agent to do this.

Shane says at the end of this talk:

> If you think this is a terrible idea, I want to hear from you. Come talk to me afterwards and tell me what's wrong with this idea.

Since many of us weren't at the workshop, I figured I'd share the talk here to discuss it on LW.