Harley-ml/lesswrong huggingface dataset:
# A Clearer Thinking tool that teaches you to use Internal Family Systems concepts

One of our recently-released Clearer Thinking tools, [Have Better Conversations with Yourself](https://programs.clearerthinking.org/internal_family_systems.html), walks you through key concepts from Internal Family Systems (IFS). While the ideas of IFS shouldn't be taken literally, many people seem to find that IFS provides useful metaphors and techniques for self-exploration and self-change (which is why we made this tool). If you aren't already acquainted with IFS, we hope you'll find it useful to learn about these ideas in a structured way (our tool helps you apply them to your own situation)! People who are already familiar with IFS might also find it useful to refresh their memory of the concepts.

The tool starts by inviting you to describe an internal conflict (e.g., a situation where different "parts" of you feel like they want opposite things). It then aims to help you to get in touch with your different mental parts (or subagents) and develop a more positive relationship with them. We hope that this will help you to:

  • Get more clarity on internal conflicts

  • Make progress on difficult problems and dilemmas

  • Have more self-acceptance

If you use the tool, we'd love to hear your feedback!

A big shout-out goes to [Amber Dawn Ace](https://amber-dawn-ace.com/) for developing the tool.
# "notkilleveryoneism" sounds dumb

"AI safety" and "AI alignment" now often mean "making AI not say racist/illegal/etc things". The term "AI notkilleveryoneism" is now sometimes being used as a replacement that refers specifically to ASI-related risks.

I think "notkilleveryoneism" is the wrong term to use, for reasons including:
- The basic structure of simple words mashed together with a suffix is associated with fantasy, media for kids, and signalling lack of effort.
- The "ism" suffix brings up associations to religions. You don't want to say "we are the tribe of people whose thing is opposition to X" - you just want to say "we're opposed to X".
- It's awkward to say, not catchy.
- There are broader concerns than AI literally killing everyone, and you probably want as big an alliance as possible.

What, then, might be better? Perhaps an analogy to biological or nuclear safety would be good, eg:
- Stop AI Gain-of-Function Research
- Stop AI Proliferation

Another angle is "enjoy the AI summer" framing. The ability of GPT-4 to fill out forms, the art generation by the various LoRA models...even just the ability to do good sentence embeddings and fast vector search - all these things will have big impacts in businesses, but it will take some time for people to figure out how to use them most effectively. You could say things like:
- Enjoy the AI summer.
- Taste the fruit before planting more.

Yet another angle is the "figure out how to distribute the gains equitably before moving on" and "AI power dynamics" framing. For example:
- Stop (Chinese-style) AI surveillance.
- Don't let them make you obsolete. (like horses)

There are a number of possibilities, and I'm not proposing a single thing in particular, I'm just saying that people should use other phrases than "AI notkilleveryoneism".

I [thought of](https://bhauth.com/blog/machine%20learning/ai%20risks.html) the following terms for *technical* discussion:
- U-al = user-alignment
- O-al = owner-alignment
- S-al = society-alignment
- H-al = humanity-alignment
- I-al = intelligence-alignment

But for public-facing statements, such abbreviations are obviously unsuitable.
# Which technologies are stuck on initial adoption?

Video calls have been with us for a while. Except, they were rarely used. IME, people sometimes had Skype calls with relatives abroad and that's about it. And then, COVID happened. Suddenly, Zoom skyrocketed, with Google Meet not far behind. The reason is obvious. 

Now, the time of lockdowns and restrictions on gatherings is over, the incentives to do video calls are (AFAICT) more or less the same as pre-COVID, and yet video calls persist. They became a completely routine way of doing business meetings, academic seminars and occasional social events. Why? AFAICT it's just the initial adoption barrier: once everyone did lots of video calls, and realized they are actually pretty convenient, they just kept using them.

So, here's a fun question: What other things are like video calls in the pre-COVID era? That is, the technology exists (more or less: maybe the UX needs some trivial improvements), the use-cases exist, only nobody uses it just because they're unaware or because it's not a "normal" thing everyone does. Given something to create initial adoption (like COVID did for video calls), everyone would start using it and never go back.
# Horizontal vs vertical generality

*This is a note/definition I want to put out there. While the term may sound similar to breadth vs depth of knowledge, it is importantly distinct.*

Let's define intelligence as the ability to process information and make decisions. (This is a very broad definition and includes not just humans but also e.g. most common computer programs.) General ability is intelligence that is applicable to many different tasks. Consider a key distinction in intelligence: horizontal generality vs vertical generality.

*   Horizontal generality: the ability to contribute to many different tasks. Calculators, search engines, etc. are horizontally general.
*   Vertical generality: the ability to efficiently complete tasks with minimal outside assistance. Animals, plants, etc. are vertically general.

Humans could arguably be said to be on the simultaneous frontier of horizontal and vertical generality.

You might think horizontal vs vertical generality is the same as breadth vs depth of knowledge, but I don't think it is. The key distinction is that breadth vs depth of knowledge concerns fields of information, whereas horizontal vs vertical generality concerns tasks. Inputs vs outputs.

Some tasks may depend on multiple fields of knowledge, e.g. software development depends on programming capabilities and understanding user needs, which means that depth of knowledge doesn't guarantee vertical generality.

On the other hand, some fields of knowledge, e.g. math or conflict resolution, may give gains in multiple tasks, which means that horizontal generality doesn't require breadth of knowledge.
# Accuracy of arguments that are seen as ridiculous and intuitively false but don't have good counter-arguments

I think one of the biggest reasons I am worried about AI doom is that there doesn't seem to be very good counter-arguments. Most of them are just bad, and the ones that aren't a bit vague (usually something along the lines of "humanity will figure it out" or "maybe LLMs will scale in nice ways").

However, I'm curious as to how accurate this heuristic is. __My question:__ What examples in the past are there of __"argument is widespreadly seen as ridiculous and intuitively false, but the argument is pretty solid and the counter-arguments aren't"__. (Sorry if that's a bit vague, use your best judgement. I'm looking specifically for examples that are similar to the AI x-risk debate.) And did they turn out true or false? Try to include reasons why the argument was so strong, a typical counter-argument, and the strongest counter-argument.

Please use spoiler text for the final answer so that I can try to predict it before seeing it!
# Vaccine Policies Need Updating

It's [reasonably common](https://www.jefftk.com/p/boston-social-dance-covid-requirements) for events like social dances to require vaccination. This usually looks like requiring attendees be "fully vaccinated", "up to date", or "fully vaccinated and boosted", and cashes out to the people at the door checking for a primary series plus potentially a booster.

A bit over a week ago, however, the CDC put out [updated guidance](https://www.cdc.gov/media/releases/2023/s0419-covid-vaccines.html), including:

> Monovalent (original) mRNA COVID-19 vaccines will no longer be recommended for use in the United States.

And:

> CDC recommends that everyone ages 6 years and older receive an updated (bivalent) mRNA COVID-19 vaccine, regardless of whether they previously completed their (monovalent) primary series.

This reads to me that if someone is not yet vaccinated and decides to become so, they'll be given a single bivalent mRNA dose.

Events that require vaccination need to decide how to handle this, and it would be good to think about before someone shows up at the dance! At the dance I help organize we [phrase](https://www.bidadance.org/accessibility) it as requiring a booster if eligible, but if it's your first shot then it's not boosting anything? And some dances are even more explicit about requiring something you can't get anymore:

> All students, instructors, and staff who attend in-person classes and/or events must be fully vaccinated against COVID-19. Fully vaccinated means that it has been at least two weeks since your second dose of Pfizer or Moderna vaccine or first dose of Johnson & Johnson vaccine. Starting January 2022, individuals must have received a COVID-19 booster shot if they received Johnson & Johnson only or if their 2nd dose of Pfizer or Moderna was more than 6 months ago. —[Blues Union](https://bluesunionboston.com/code-of-conduct/)

(Not trying to say anything here on whether or when I think events should require vaccination.)
# A small update to the Sparse Coding interim research report

This is a linkpost to a [**set of slides**](https://docs.google.com/presentation/d/1lP3u8NupGyWtrECoxr4zroemUZpCDd5k3ghuBbORWHU/edit?usp=sharing)  containing an update to a project that was the subject of a previous post ([\[Interim research report\] Taking features out of superposition with sparse autoencoders](https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition)).

The update is very small and scrappy. We haven't had much time to devote to this project since posting the Interim Research Report.

**TL;DR for the slides: **

*   We trained a minuscule language model (LM) (residual size = 16; 6 layers) and then trained sparse autoencoders on MLP activations (dimension =  64) from the third layer of that model.
*   We found that, when we compared the 'ground truth feature recovery' plots, the plots for the toy data and LM data were much more similar than in the Interim Research Report.
*   *Very, very tentatively*, we found the layer had somewhere between 512-1024 features. By labelling a subset of these features, we estimate there are roughly 600 easily labellable (monosemantic) features. For instance, we found a feature that activates for a period immediately after 'Mr', 'Mrs', or 'Dr'.
*   We suspect that the reason the toy data and LM data plots had previously looked different was due to severely undertrained sparse autoencoders.  

We're hopeful that with more time to devote to this project we can confirm the results and apply the method to larger LMs. If it works, it would give us the ability to tell mechanistic stories about what goes on inside large LMs in terms of monosemantic features.
# How can one rationally have very high or very low probabilities of extinction in a pre-paradigmatic field?

It is generally accepted in the local AI alignment circles that the whole field is pre-paradigmatic, in the Kuhnian sense (phase 1, as summarized [here](https://www.hoddereducation.co.uk/media/Documents/magazine-extras/IB%20Review/IBRev%204_2/IBReview4_2_poster.pdf), if longer reading is not your thing). And yet, plenty of people are quite confident in their predictions of either doom or fizzle. A somewhat caricature way of representing their logic is, I think, "there are so many disjunctive ways to die, only one chance to get it right, and we don't have a step-by-step how-to, so we are hooped" vs "this is just one of many disruptive inventions whose real impact can only be understood way down the road, and all of them so far have resulted in net benefit, AI is just another example" (I have low confidence in the accuracy of the latter description, feel free to correct.) I can see the logic in both of those, what I do not see is how one can rationally have very high or very low confidence, given how much inherent uncertainty there is in our understanding of what is going on. 

My default is something more cautious, akin to Scott Alexander's [https://slatestarcodex.com/2019/06/03/repost-epistemic-learned-helplessness/](https://slatestarcodex.com/2019/06/03/repost-epistemic-learned-helplessness/) 

where one has to recognize their own reasoning limitations in the absence of hard empirical data, not "The Lens That Sees Its Flaws", but more like "The Lens That Knows It Has Flaws" without necessarily being able to identify them. 

So, how can one be very very very sure of something that has neither empirical confirmation, nor sound science behind it? Or am I misrepresenting the whole argument?
# Where is all this evidence of UFOs?

In his latest [blog](https://www.overcomingbias.com/p/my-awkward-situation), Robin Hanson writes

> There have been over 100K UFO sightings reported worldwide since 1940. Roughly 5% or so are “strong” events, which seem rather hard to explain due to either many witnesses, especially reliable witnesses, physical evidence, or other factors.

Yet, I am not aware of a single UFO encounter that can't be explained by one of:

1.  Unreliable [eyewitnesses](https://mufoncms.com/last_20_report_public.html)
2.  [Things](https://www.youtube.com/watch?v=zlrz84nEXtk) that go away when we get better cameras

Importantly, Hanson's post did not include a link to the "Wikipedia of UFO encounters" listing all 100k UFO sightings and which 5000 he considered most credible.

Where is that Wikipedia?
# Distinguishing misuse is difficult and uncomfortable

USPS delivers ibuprofen and vicodin just the same because it doesn't care enough to open and test every bottle of pills. If they did carefully inspect the contents of every package, they still couldn't distinguish other "shipping misuse" — money for your granddaughter's birthday looks the same as payment for a crime.

An AI providing services via an API is in a similar position. Code that checks that your firewall is configured correctly is identical to code that checks someone else's firewall is configured incorrectly.

If you have more context then you can identify misuse more easily. You can get more context by either utilizing your memory, providing higher-level tasks, or by snooping / being around.

### Memory 

Buying duct tape is normal. Buying zip ties is normal. Buying prepaid cell phones is normal. Buying extra large burlap sacks is normal.  If someone asks you to buy all four at once then you might flag that as shopping misuse pretty easily. If they buy the three sequentially then spotting it requires a persistent identifier, logging, the will to make the connection, and the will to act on it.

A couple years ago my white dad went to the store to buy beer with the dark-skinned son of a family friend. The cashier refused to sell it because it obviously looked like buying alcohol for a minor. He just went around the corner and asked a different cashier. I suppose this was a failure of logging — the cashier should've taken the credit card before refusing to sell the beer.

### High-level tasks

It is easier to tell apart a malicious function from a line of code, a file from a function, a repo from a file, or an app from a repo.

### Being around

Hard to know if your boss is a good fella when you take your orders on slips of paper slid under the door. Humans have avoided doing many evil deeds asked of them by witnessing the emotions and other actions of the one asking. A company like Microsoft or Google could have AI that's generally aware of who you are and why you're asking for certain things. This would take a lot of courage and be a privacy nightmare.

## Conclusion

We have set up the misuse detection task as an impossible problem. You simply cannot tell if a single query in isolation is malicious or not. Actually detecting misuse will require some reprioritization, reframing, and uncomfortable decisions.
# The Apprentice Thread 2

A while back, [@Zvi](https://www.lesswrong.com/users/zvi?mention=user)  [set up a Schelling point for mentorship](https://www.lesswrong.com/posts/scNCmwaduCgJmCBYh/the-apprentice-thread). I, personally, had some pretty good experiences from that thread, and wanted to continue the tradition, now that the old thread's been inactive for some time.

To see some of my thoughts on mentorship, check out [this dialogue](https://www.lesswrong.com/posts/M6dRmRtsBsxWTpeye/talking-about-mentorship) between Elizabeth and I.

I'll be copying Zvi's formatting for comments:

> Replies to this post should take the form of any of the following:
> 
> 1.  \[MENTOR\]: A non-binding indication of potential interest in mentorship. Mention that you might, at some point, be interested in taking on an apprentice. This commits you to nothing. Make sure to indicate what you’d be teaching them and what project would likely be involved, and open with \[MENTOR\]. You are free to include contact info, or not include it and monitor replies.
>     1.  Replies to this comment to indicate potential interest in being the apprentice, marked \[APPRENTICE\], which should include a method of further contact.
> 2.  \[APPRENTICE\]: A non-binding indication of potential interest in being an apprentice. Mention that you might, at some point, be interested in being an apprentice. This commits you to nothing. Make sure to indicate what you’re interested in being an apprentice in and learning, and an indication of what’s motivating you.
>     1.  Replies to this comment to indicate potential interest in being the mentor, marked with \[MENTOR\], which should include a method of further contact.
> 3.  \[NORMAL\] You’re free to comment as per normal, but start with \[NORMAL\] in the top-level for clarity.
# Geoff Hinton Quits Google

The NYTimes reports that Geoff Hinton has quit his role at Google:

> On Monday, however, he officially joined a growing chorus of critics who say those companies are racing toward danger with their aggressive campaign to create products based on generative artificial intelligence, the technology that powers popular chatbots like ChatGPT.
> 
> ***Dr. Hinton said he has quit his job at Google, where he has worked for more than a decade and became one of the most respected voices in the field, so he can freely speak out about the risks of A.I. A part of him, he said, now regrets his life’s work***.
> 
> “I console myself with the normal excuse: If I hadn’t done it, somebody else would have,” Dr. Hinton said during a lengthy interview last week in the dining room of his home in Toronto, a short walk from where he and his students made their breakthrough.

[https://www.nytimes.com/2023/05/01/technology/ai-google-chatbot-engineer-quits-hinton.html](https://www.nytimes.com/2023/05/01/technology/ai-google-chatbot-engineer-quits-hinton.html)

Some clarification from Hinton followed:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9a2ded7d69cec39c7c74d03ebb73a6b67a9dfd9450ffa6f2.png)

It was already apparent that Hinton considered AI potentially dangerous, but this seems significant.
# Open & Welcome Thread - May 2023

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Technological unemployment as another test for rationalist winning

Ultimately, [rationalism should help people win](https://www.lesswrong.com/tag/rationality-is-systematized-winning). Scott Alexander [claims](https://www.lesswrong.com/posts/MajyZJrsf8fAywWgY/a-lesswrong-crypto-autopsy) that the surge of the price of Bitcoin was a test of that:

> ...suppose God had decided, out of some sympathy for our project, to make winning as easy as possible for rationalists. He might have created the biggest investment opportunity of the century, and made it visible only to libertarian programmers willing to dabble in crazy ideas. And then He might have made sure that all of the earliest adapters were Less Wrong regulars, just to make things extra obvious.
> 
> This was the easiest test case of our "make good choices" ability that we could possibly have gotten, the one where a multiply-your-money-by-a-thousand-times opportunity basically fell out of the sky and hit our community on its collective head. So how did we do?
> 
> I would say we did mediocre.

Five years later, suppose God wanted to give rationalists another test. But instead of the opportunity to win big, He wanted to test whether they could avoid losing hard. He might create the largest workforce disruption of the century, driven by an unpredictable technology (which the rationalists happen to know the most about) and primarily affecting white-collar workers (which most rationalists are). **If rationalism truly helps people predict the future and make better decisions, rationalists who work should survive the incoming wave of AI-driven job automation better than everyone else**.

Of course, this only applies for those whose jobs are truly in danger. I'm pursuing a career in AI alignment research – if *that* becomes automated, none of this matters. But suppose, [like many of us](https://www.lesswrong.com/posts/E2koS25u6jajt5uo8/2022-survey-results#III__Work_and_Education), you're a software engineer. You should be paying careful attention to [forecasts related to your future compensation](https://www.metaculus.com/questions/8462/entry-level-tc-at-google-2027/) and learning how to use the [latest tools](https://githubnext.com/) which accelerate your productivity relative to your competitors. And if things start looking really grim, since you know about [Moravec's paradox](https://en.wikipedia.org/wiki/Moravec%27s_paradox) and [status quo bias](https://en.wikipedia.org/wiki/Status_quo_bias), you'd start learning valuable [blue-collar skills](https://www.indeed.com/career-advice/finding-a-job/highest-paying-blue-collar-jobs) earlier than your developer friends who are also at risk of getting laid off.

Most artists didn't expect robots to become creative and thus didn't prepare accordingly. Then, DALL-E 2 happened, rendering many of the skills they spent years mastering economically useless. This should be a warning shot for everyone else who thinks they're rational: think ahead, don't get caught off guard, and *act on your beliefs*, and maybe you'll remain economically valuable up until the singularity.
# Five Worlds of AI (by Scott Aaronson and Boaz Barak)

A relatively short April 27, 2023 post attempting to classify AI outcomes into 5 main cases: 

   * **AI-Fizzle**
   * Not-AI-Fizzle
       * Civilization recognizably continues
            * **Futurama**
            * **AI-Dystopia**
       * Civilization does not recognizably continue
            * **Singularia**
            * **Paperclipalypse**

>In other words, no one has done for AI what Russell Impagliazzo did for complexity theory in 1995, when he defined the five worlds Algorithmica, Heuristica, Pessiland, Minicrypt, and Cryptomania, corresponding to five possible resolutions of the P vs. NP problem along with the central unsolved problems of cryptography.

The authors elaborate:

>Like in Impagliazzo’s 1995 paper on the five potential worlds of the difficulty of NP problems, we will not try to be exhaustive but rather concentrate on extreme cases.  It’s possible that we’ll end up in a mixture of worlds or a situation not described by any of the worlds.  Indeed, one crucial difference between our setting and Impagliazzo’s, is that in the complexity case, the worlds corresponded to concrete (and mutually exclusive) mathematical conjectures.  So in some sense, the question wasn’t “which world _will_ we live in?” but “which world have we Platonically _always_ lived in, without knowing it?”  In contrast, the impact of AI will be a complex mix of mathematical bounds, computational capabilities, human discoveries, and social and legal issues. Hence, the worlds we describe depend on more than just the fundamental capabilities and limitations of artificial intelligence, and humanity could also shift from one of these worlds to another over time.

Extensive discussion in the comments. Eliezer writes a comment, and some of the replies to that comment are quite informative.
# Are Emergent Abilities of Large Language Models a Mirage? [linkpost]

> Recent work claims that large language models display emergent abilities, abilities not present in smaller-scale models that are present in larger-scale models. What makes emergent abilities intriguing is two-fold: their sharpness, transitioning seemingly instantaneously from not present to present, and their unpredictability, appearing at seemingly unforeseeable model scales. Here, we present an alternative explanation for emergent abilities: that for a particular task and model family, when analyzing fixed model outputs, one can choose a metric which leads to the inference of an emergent ability or another metric which does not. Thus, our alternative suggests that existing claims of emergent abilities are creations of the researcher's analyses, not fundamental changes in model behavior on specific tasks with scale. We present our explanation in a simple mathematical model, then test it in three complementary ways: we (1) make, test and confirm three predictions on the effect of metric choice using the InstructGPT/GPT-3 family on tasks with claimed emergent abilities, (2) make, test and confirm two predictions about metric choices in a meta-analysis of emergent abilities on BIG-Bench; and (3) show how similar metric decisions suggest apparent emergent abilities on vision tasks in diverse deep network architectures (convolutional, autoencoder, transformers). In all three analyses, we find strong supporting evidence that emergent abilities may not be a fundamental property of scaling AI models.

  
This result seems important for two reasons:

1.  If AI abilities are predictable, then we can forecast when we'll get dangerous capabilities ahead of time, rather than being taken by surprise. This result strengthens the case for a research program of devising a ton of interesting benchmarks to measure how capabilities are improving as a function of scale.  
2.  It provides some evidence against the idea that "understanding is discontinuous", and that important AI abilities will suddenly click together at some level, which is a very loose description of what I understood to be one of the [primary intuitions behind AI foom](https://www.lesswrong.com/posts/vwLxd6hhFvPbvKmBH/yudkowsky-and-christiano-discuss-takeoff-speeds#_Understanding__is_discontinuous).
# Averting Catastrophe: Decision Theory for COVID-19, Climate Change, and Potential Disasters of All Kinds

***Averting Catastrophe*** **is a book by** [**Cass Sunstein**](https://en.wikipedia.org/wiki/Cass_Sunstein) **that NYU Press published** [**in April 2021**](https://nyupress.org/9781479808489/averting-catastrophe/)**.** I'm surprised that nobody has mentioned it on the EA Forum before (AFAICT). Quoting from the [NYU Press page](https://nyupress.org/9781479808489/averting-catastrophe/):

> *Averting Catastrophe* explores how governments ought to make decisions in times of imminent disaster. Cass R. Sunstein argues that using the “maximin rule,” which calls for choosing the approach that eliminates the worst of the worst-case scenarios, may be necessary when public officials lack important information, and when the worst-case scenario is too disastrous to contemplate. He underscores this argument by emphasizing the reality of “Knightian uncertainty,” found in circumstances in which it is not possible to assign probabilities to various outcomes.

The [Google Books page](https://books.google.com/books?id=ka0eEAAAQBAJ) includes a table of contents:

*   Introduction
*   What We Don't Know
*   With and Without Numbers
*   The Maximin Principle
*   The Precautionary Principle
*   Uncertainty
*   Objections
*   Irreversibility
*   Departments and Agencies

I found this book while researching the [precautionary](https://en.wikipedia.org/wiki/Precautionary_principle) [principle](https://plato.stanford.edu/entries/justice-climate/#RiskUnce). Sunstein has written multiple papers on this topic, including two that seem particularly relevant to x-risk:

*   **"**[**Irreversible and Catastrophic**](https://scholar.google.com/scholar?cluster=16645175349513249354)**"**
*   **"**[**The Catastrophic Harm Precautionary Principle**](https://scholar.google.com/scholar?cluster=214867129917358618)**"**

Sunstein also wrote [*Laws of Fear: Beyond the Precautionary Principle*](https://www.cambridge.org/core/books/laws-of-fear/16124E83F371BEAA5082AB07EA892836), which Cambridge University Press published in 2005.

* * *

See Sunstein's Wikipedia page for a [(somewhat) comprehensive list of his books](https://en.wikipedia.org/wiki/Cass_Sunstein#Publications). Also note that [he appeared on the 80,000 Hours Podcast in 2019](https://80000hours.org/podcast/episodes/cass-sunstein-how-change-happens/).
# How much do personal biases in risk assessment affect assessment of AI risks?

Had a stray thought that I think is worth exploring: perhaps disagreements over how to respond to AI risks are heavily influenced by personal biases in how to assess risk in general such that very different conclusions can be drawn from evaluating the same evidence and arguments.

This sort of thing isn't without precedent (e.g. [Bourget, David, and David J. Chalmers. “What Do Philosophers Believe?”](https://www.jstor.org/stable/42920613)). We can roughly imagine a few possible correspondences:

*   high concern over AI x-risks most common among people with
    *   high anxiety
    *   low trust of strangers
    *   home bodies
*   middling concern
    *   average psychological profile on various metrics
*   low concern
    *   low anxiety/fear
    *   high trust of strangers, or at least highly resilient to betrayal
    *   thrill seekers

For archetypical examples of people who I'd consider high, medium, and low AI risk:

*   Eliezer Yudkowsky - high
*   Paul Christiano - middle
*   Yann LeCunn - low

I don't know if any of them fit my toy profiles. Honestly I'm not really sure what might correspond, if anything; these are just easy guesses to give some flavor to the idea.

So, if you've thought a lot about AI risks, and especially if you're actively working on AI in some capacity, I'd appreciate it if you left an answer filling out this template so we can see if the anecdata suggest anything worth exploring further. This might help with identify places where disagreement persists not because of disagreements about evidence, but personal disagreements about how to weigh and evaluate the risks the evidence suggests exist. Knowing that might prove useful to bridging some disagreements.

Answer Template[^mkig8lbodtg]:
==============================

**Level of AI risk concern**: high/medium/low

**General level of risk tolerance in everyday life**: high/medium/low

**Brief summary of what you do in AI:** mostly if you're not famous to help identify you

**Anything weird about you**: are you unusually anxious, calm, whatever? do people tell you that you're the most X person they know? not necessarily a full psychological profile just some key facts to give a sense of your personality

[^mkig8lbodtg]: I realize this template is not very well constructed. It's because I'm not quite sure what we're looking for, if anything, so it's relatively open ended in the hopes that the answers will help make it clearer what I should have asked.
# List of notable people who believe in AI X-risk?

Remembering Scott's [AI Researchers On AI Risk](https://slatestarcodex.com/2015/05/22/ai-researchers-on-ai-risk/). The debate about AI is heating up, and a lot of people rely on authority figures for knowing what to believe. Do we have a current list of notable AI X-risk believers? Would there be value in compiling such a list?
# Prizes for matrix completion problems

Here are two self-contained algorithmic questions that have come up in our research. We're offering a bounty of $5k for a solution to either of them—either an algorithm, or a lower bound under any hardness assumption that has appeared in the literature.

**Question 1 (existence of PSD completions)**: given \\(m\\) entries of an \\(n \\times n\\) matrix, including the diagonal, can we tell in time \\(\\tilde{O}(nm)\\) whether it has any (real, symmetric) positive semidefinite completion? Proving that this task is at least as hard as dense matrix multiplication or PSD testing would count as a resolution.

**Question 2 (fast “approximate squaring”):** given \\(A \\in \\;\\mathbb{R}^{n \\times n}\\) and a set of \\(m = \\Omega(n)\\) entries of \\(AA^T\\), can I find _some_ PSD matrix that agrees with \\(AA^T\\) in those m entries in time \\(\\tilde{O}(nm)\\)?

We'll pay $5k for a solution to either problem. The offer is open for each problem for 3 months or until the problem gets solved (whichever happens first). Winners are welcome to publish solutions independently. Otherwise, if the result ends up being a significant part of a paper, we’ll invite them to be a coauthor.

We’ll also consider smaller prizes for partial progress, or anything that we find helpful for either solving the problem or realizing we should give up on it.

To understand the motivation for these questions, you can read our paper on [Formalizing the presumption of independence](https://arxiv.org/abs/2211.06738) and in particular [Appendix D.7.2](https://arxiv.org/pdf/2211.06738.pdf#page=53). ARC is trying to find efficient heuristic estimators as a formalization of defeasible reasoning about quantities like the variance of a neural network's output. These two questions are very closely related to one of the simplest cases where we haven't yet found any reasonable linear time heuristic estimator.

We don’t expect to receive many incorrect proposals, but if we receive more than 5 we may start applying a higher standard in order to save our time. If we can’t understand a solution quickly, we may ask you to provide more details, and if we still can’t understand it we may reject it. We expect a correct solution to be about as clear and easy to verify as a paper published at STOC.

For both problems, it’s OK if we incorrectly treat a matrix as PSD as long as all of its eigenvalues are at least \\(-\\varepsilon\\) for a small constant \\(\\varepsilon\\). \\(\\tilde{O}\\) hides polylogarithmic factors in \\(n\\), \\(\\varepsilon\\), and the max matrix entry. Feel free to ask for other clarifications on [our question on Math Overflow](https://mathoverflow.net/questions/444916/quickly-determining-if-a-matrix-has-any-psd-completion), [on Facebook](https://www.facebook.com/paulfchristiano/posts/pfbid037EJPBYYoQ7ra4W6n8yA7Lv3aV95U3npStPggGLJ5F5YzxfwAtMA4nmV2hcsfykCBl?notif_id=1683132550328608&notif_t=feedback_reaction_generic&ref=notif), or [by email](mailto:prize@alignment.org).

To submit a solution, send an email to [prize@alignment.org](mailto:prize@alignment.org).
# Has the Symbol Grounding Problem just gone away?

A few years ago, the symbol grounding problem was widely considered a significant challenge in AI discussions. I believed that it would likely be addressed as a side effect of capability improvements, without requiring specific breakthroughs or attention, unlike others who considered it fundamental. However, like many, I didn't anticipate the extent to which GPT-4 has demonstrated this to be true. Asserting that such capabilities could be achieved with a text-only training set at that time would have seemed like a parody of my position.  
Had you asked me how a model like GPT-4 would acquire its capabilities, I would have suggested a process more akin to how children learn. It might have started with a predictive model of physical reality, established the concept of an object, learned object permanence, and then acquired simple words as they applied to previously learned concepts.  
Despite its capabilities, GPT-4 still seems to lack robust physical intuition, evident in data science tasks and mathematical understanding related to the 3D world. Will we see a model trained from scratch, as described earlier? For instance, the [Meta AI model](https://www.engadget.com/meta-shares-ai-model-that-can-detect-objects-it-hasnt-seen-before-210002471.html) appears to grasp the concept of an object in 2D. Suppose this understanding is fully extended to 3D and designated as our foundational model. Can such a model be trained on language, particularly that which relates to the physical world, and develop physical intuition as good as a human's?

**Grounding overhang and interpretability implications:**

Would such a model be much better at mathematical and programming tasks for given model resources? Assuming the foundational model is much smaller than GPT-4 it seems reasonable that it could gain similar or greater mathematical and programming skills while still having a smaller model size even when trained on a enough language concepts to be capable at those tasks.

This could also help with interpretability as the foundational model couldn't really be thought to lie or deceive as it is just modelling objects in 3d. Deception and theory of mind abilities could be observed as they became available.
# [Linkpost]Transformer-Based LM Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens

From the abstract:

> Recent psycholinguistic studies have drawn conflicting conclusions about the relationship between the quality of a language model and the ability of its surprisal estimates to predict human reading times, which has been speculated to be due to the large gap in both the amount of training data and model capacity across studies. The current work aims to consolidate these findings by evaluating surprisal estimates from Transformer-based language model variants that vary systematically in the amount of training data and model capacity on their ability to predict human reading times. The results show that surprisal estimates from most variants with contemporary model capacities provide the best fit after seeing about two billion training tokens, after which they begin to diverge from humanlike expectations. Additionally, newly-trained smaller model variants reveal a 'tipping point' at convergence, after which the decrease in language model perplexity begins to result in poorer fits to human reading times. These results suggest that the massive amount of training data is mainly responsible for the poorer fit achieved by surprisal from larger pre-trained language models, and that a certain degree of model capacity is necessary for Transformer-based language models to capture humanlike expectations.

The authors show strongly U-shaped curves relating the additional predictive power that adding per-token suprisal gives to a regression model and model perplexity.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/730a6bdfad32ba97d5ec74ce2412eeb3a9e1474f86db7224.png)

From the results section:

> The results in Figure 13 show that across both cor-  
> pora, surprisal from most LM variants made the  
> biggest contribution to regression model fit after  
> 1,000 training steps (i.e. after ∼2B tokens). This  
> seems to represent a ‘humanlike optimum,’ after  
> which surprisal estimates begin to diverge from hu-  
> manlike expectations as training continues. At this  
> point in training, there appears to be no systematic  
> relationship between model capacity and predictive  
> power of surprisal estimates.

I'm not going to speculate too much but these curves remind me a lot of the diagram describing Chris Olah's view on the difficulty of mechanistic interpretability with model power:  
![](https://i.imgur.com/DQPWNC5.png)  
Other remarks:  
-It would be neat to compare the perplexity inflexion point with the Redwood estimates of human perplexity scores on text.  
-Performance seems well tied to model perplexity and tokens but not so much to model size
# Google "We Have No Moat, And Neither Does OpenAI"

This (purported) leak is incredibly scary if you're worried about AI risk due to AI capabilities developing too fast.

Firstly, it suggests that open-source models are improving rapidly because people are able to iterate on top of each other's improvements and try out a much larger number of experiments than a small team at a single company possibly could.

Secondly, it suggests that data size is less important than you might think from recent scaling laws, as you can achieve high performance with "small, highly curated datasets". This suggests a world where dangerous capabilities are much further distributed.

Thirdly, it proposes that Google should open-source its AI technology. Obviously, this is only the opinion of one researcher and it currently seems unlikely to occur, but if Google was to pursue this path, this would lead to a significant shortening of timelines.

I'm worried that up until now, this community has been too focused on the threat of big companies pushing capabilities ahead and not focused enough on the threat posed by open-source AI. I would love to see more discussions of regulations in order to mitigate this risk. I suspect it would be possible to significantly hamper these projects by making the developers of these projects potentially liable for any resulting misuse.
# Why not use active SETI to prevent AI Doom?

Let's assume that Eliezer is right: soon we'll have an AGI that is very likely to kill us all. (personally, I think Eliezer is right).

There are several ways to reduce the risk, in particular: speeding up alignment research and slowing down capabilities research, by various means. 

One underexplored way to reduce the risk is [active SETI](https://en.wikipedia.org/wiki/Active_SETI) (also known as METI). 

The idea is as follows:

*   Send powerful radio signals into space: "guys, soon we'll be destroyed by a hostile AGI. Help us!" (e.g. using a language constructed for the task, like [Lincos](https://en.wikipedia.org/wiki/Lincos_language))
*   If a hostile alien civilization notices us, we're going to die. But if we're going to die from the AGI anyway, who cares?
*   If a benevolent alien civilization notices us, it could arrive in time to save us. 

The main advantage of the method is that it can be implemented by a small group of people within a few months, without governments and without billions of dollars. Judging by the running [costs](https://www.nsf.gov/news/special_reports/arecibo/Arecibo_Fact_Sheet_11_20.pdf) of the Arecibo Observatory, one theoretically can rent it for a year for only $8 million. Sending only a few hundred space messages could be even cheaper. 

Obviously, the method relies on the existence of an advanced alien civilization within a few light years from the Earth. The existence seems to be unlikely, but who knows. 

Is it worth trying?
# What can we learn from Bayes about reasoning?

What are the *qualitative* lessons we can learn about logic and reasoning from Bayesian epistemology, that is, from taking Bayes' rule as a mathematical model for thought (even if it is considered a simplified formalism that we often can't implement?)

I've seen at least a few of these from [@Eliezer Yudkowsky](https://www.lesswrong.com/users/eliezer_yudkowsky?mention=user), but I think they're scattered across many essays.

Some things I consider to be examples of what I'm gesturing at here:

*   [Aumann's Agreement Theorem](https://www.lesswrong.com/tag/aumann-s-agreement-theorem)
*   Conservation of expected evidence

Thanks!
# Linkpost for Accursed Farms Discussion / debate with AI expert Eliezer Yudkowsky

Eliezer has been on a podcast tour of late. This appears to be the latest one on YouTube.

I feel like this one didn't go so well. It's also over two hours, so maybe don't watch this one if you have better things to do, but for those that did, I thought I'd make a space for discussion.

Most of it was Eliezer trying to do some Socratic questioning to try to get to the bottom of the host's intuition that AI isn't "really" intelligent, or just imitates it in fragments, and therefore is not dangerous. I'm not sure I understood. It didn't seem to be expressed very well.

Eliezer's goal in doing the interview seems to have been to figure out how to talk to a different audience. People in the know ask very different questions from those who are just learning about this.
# TED talk by Eliezer Yudkowsky: Unleashing the Power of Artificial Intelligence

**Note:** The video is no longer available. It has been set to private. It'll eventually be released on the main TED channel.

This is a TED talk published early by a random TEDx channel.

Tweet by EY: [https://twitter.com/ESYudkowsky/status/1655232464506466306](https://twitter.com/ESYudkowsky/status/1655232464506466306)

> Looks like my Sudden Unexpected TED Talk got posted early by a TEDx account.

YouTube description:

> Eliezer Yudkowsky is a foundational thinker on the long-term future of artificial intelligence. 
> 
> With more than 20 years of experience in the world of AI, Eliezer Yudkowsky is the founder and senior research fellow of the Machine Intelligence Research Institute, an organization dedicated to ensuring smarter-than-human AI has a positive impact on the world. His writings, both fiction and nonfiction, frequently warn of the dangers of unchecked AI and its philosophical significance in today's world. 
> 
> Yudkowsky is the founder of LessWrong, an online forum and community dedicated to improving human reasoning and decision-making, and the coinventor of the "functional decision theory," which states that decisions should be the output of a ﬁxed mathematical function answering the question: "Which output of this very function would yield the best outcome?"