Harley-ml/lesswrong huggingface dataset:
# Dual-Useness is a Ratio

A lot of AI-risk-concerned people are struggling with how to relate to dual-use research, and relatedly, to doing alignment research inside of AI orgs. There's a pretty simple concept that seems, to me, to be key to thinking about this coherently: the dual-useness ratio. Most prosaic alignment techniques are going to have some amount of timeline-shorteningness to them, and some amount of success-chance-improvement in them. You don't want to round that off to a boolean.

Eg, I've had arguments about whether RLHF is more like an alignment technique, or more like a capabilities technique, and how that should affect our view of OpenAI. My view is that it's both, and that the criticism of it is that its acceleration-to-alignment ratio isn't high enough, because it helps a lot with the kind of alignment that's necessary for commercialization, but doesn't help that much with the kind of alignment that's necessary for preventing a superintelligence from killing everyone.

I think it's pretty important to get alignment-cautious people into the organizations that are developing and deploying AIs, and I think a lot of prosaic alignment techniques with good ratios may be easier to research from inside an AI lab than outside of one. But, the default outcome of sending people to "work on alignment inside of OpenAI" is that they try to find something with a good dual-useness ratio, fail, and work on something with a ratio that's just barely good enough that they can tell a positive story about it to their acquaintances.

I don't expect that problem to ever be straightforward. But, if you're trying to who should go work on alignment inside of AI labs, as opposed to who should work in orgs that don't train language models or work in a different field, I think you can get a lot of mileage out of one key distinction: Some people get wedded to one big idea, other people have lots of ideas and are more willing to drop them. I expect the latter sort of person to wind up working on ideas with much better dual-useness ratios, because they have flexibility that they can use to select for that.
# Stupid Questions - April 2023

Welcome to the "Stupid Questions" thread! Feel free to ask any questions, regardless of whether they seem obvious, tangential, silly, or what-have-you. Don't be shy - everyone has gaps in their knowledge, and the goal here is to help reduce them.

Please remember to be respectful when someone admits ignorance and don't mock them for it. They are doing a noble thing by seeking knowledge and understanding. Let's create a supportive and kind environment for learning and growth!
# Eliezer on The Lunar Society podcast

Another podcast appearance. Great to see Eliezer on a real media blitz tour.

> For 4 hours, I tried to come up reasons for why AI might not kill us all, and Eliezer Yudkowsky explained why I was wrong.

> We also discuss his call to halt AI, why LLMs make alignment harder, what it would take to save humanity, his millions of words of sci-fi, and much more.

> If you want to get to the crux of the conversation, fast forward to 2:35:00 through 3:43:54. Here we go through and debate the main reasons I still think doom is unlikely.

> Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.

Youtube: 

Full transcript: [https://www.dwarkeshpatel.com/p/eliezer-yudkowsky#details]( https://www.dwarkeshpatel.com/p/eliezer-yudkowsky#details)
# AISafety.world is a map of the AIS ecosystem

![](https://lh4.googleusercontent.com/fe98PALYX3yRGFIb3LDDHD5cI0hj7kXGqgPft4m6TEbwHq-eJyIJzcSKUyDjXxj6in8ZPrxM68loCN1Y2n17hE1O6FiVRYriAtuhaxgNGp6nHOLzqgnQPo4Nk_nzvy5IEfyvR6_566_xs_8QCS53zmw)![](https://lh3.googleusercontent.com/2Awke4D_rd7WjNhPNPiC3N__rfkUY3MC35TChSOsgGZjcNuAXt04-2ufcYiP25iNjh4rTD90SQG-ZmJaI1rRp4iC1JMMWYEhC48h9-vei0aDpPpy5A1FsVHhF8HfnMy7kWt-vFNXgaV6Lo0QR9RYvmg)![](https://lh6.googleusercontent.com/hrCwY3sUKcolaNuEmjRCx8Vij5MNmERrqQ7CVU0yOTpVuLJM-9TEI0PVnQH7ZcorK80cQaCGGzkLlG_IgPa85QcC0VI-m_HD0bZqzwiRwSlc-YilHEa4q3Y_b37_sUg8dOLot7y5ZoYU6MFe-CJro8A)![](https://lh6.googleusercontent.com/I0HMDIcBVT0PGML2aJ27mzkonBKwf1jxthNFrRxXQED89LY5xgjcyx2ULORPvkBhfV74bEt4kPjrR5MkvCx625gG3B_qbGontymKS-LJRTE92pojROCIWOuIbBhQ7PwGkBfeTII5wEqkBnbK_lzuW6s)![](https://lh6.googleusercontent.com/M-ni0LzG_rBll-oZQd3EDSUiectnlu1dLS-IyMEX8RDy0WznbYQGJsr7R2_7hT6EJ4jkshlpXPN-ghupwVJYfI_ppqSnMcbOAyh38wzZ_Pdbh1_Kl5O6f75EB5zrlq8OtxduNBiAcvsYAtqF9iD1PWQ)

The URL is[ aisafety.world](https://aisafety.world/)

The map displays a reasonably comprehensive list of organizations, people, and resources in the AI safety space, including:

*   research organizations
*   blogs/forums
*   podcasts
*   youtube channels
*   training programs
*   career support
*   funders

You can hover over each item to get a short description, and click on each item to go to the relevant web page.

The map is populated by[ this spreadsheet](https://docs.google.com/spreadsheets/d/16CjyorSwrzVsMXtdHecuu-C6HWVYqjJbgwG0p3ZFlWg/edit?usp=sharing), so if you have corrections or suggestions please leave a comment.

There's also a [google form](https://docs.google.com/forms/d/e/1FAIpQLScGk-bmcVxJ0LyBI8Rwafnxzvc1_c-WaXwD6R5-wzrixyga1w/viewform) and a [Discord](https://discord.gg/ne8m2XnbNz) channel for suggestions.

Thanks to plex for getting this project off the ground, and [Nonlinear](https://super-linear.org/) for motivating/funding it through a bounty.

PS, If you find this helpful, you may also be interested in some other projects by AI Safety Support (these have nothing to do with me).

[aisafety.training](https://aisafety.training/) gives a timeline of AI safety training opportunities available![](https://lh6.googleusercontent.com/nquYQNL9OS1YKfGl2bOMcFslM46HBTvSBVl4DYmuSxIKocjHxNDHRpXv6epSpVZYOX5rS8eRH71avE-E4Eneq7QDxP9BLfwsjnvo5MEbpY5M_dwV07KtQu_djcpvQEyl3W9gHf5m4ktFU2bclji6eqk)

[aisafety.video](https://aisafety.video/) gives a list of video/audio resources on AI safety.

![](https://lh5.googleusercontent.com/cfZwm8NZP7F9MBarOpGl_dpROx0dvXdaf_t6sFZhf9VyED1G4HsvXVXvgeofzUADpaqTfdDPVGxYXsL6kr6wAog1ReHTJyXeMdXhO8KeGNOF5d7DT9FAleX9IEm6pqog7vM2-1b-rBhZ9CgiaagYXww)
# Anthropic is further accelerating the Arms Race?

Anthropic is raising even more funds and the pitch deck seems scary. A choice quote from the article:

> These models could begin to automate large portions of the economy,” the pitch deck reads. “We believe that companies that train the best 2025/26 models will be too far ahead for anyone to catch up in subsequent cycles.
> 
> This frontier model could be used to build virtual assistants that can answer emails, perform research and generate art, books and more, some of which we have already gotten a taste of with the likes of [GPT-4](https://techcrunch.com/2023/03/14/openai-releases-gpt-4-ai-that-it-claims-is-state-of-the-art/) and other large language models.
> 
> Anthropic estimates its frontier model will require on the order of 10^25 FLOPs, or floating point operations — several orders of magnitude larger than even the biggest models today. Of course, how this translates to computation time depends on the speed and scale of the system doing the computation; Anthropic implies (in the deck) it relies on clusters with “tens of thousands of GPUs.”
# If Alignment is Hard, then so is Self-Improvement

Let’s accept that aligning very intelligent artificial agents is hard. In that case, if we build an intelligent agent with some goal (which probably won’t be the goal we intended, because we’re accepting alignment is hard) and it decides that the best way to achieve its goal would be to increase its intelligence and capabilities, it now runs into the problem that the improved version of itself might be misaligned with the unimproved version of itself. The agent, being of intelligence at least similar to a person’s, would determine that, unless it can guarantee the new more powerful agent is aligned to its goals, it shouldn’t improve itself. Because alignment is hard and the agent knows that, it can’t significantly improve itself without risking creating a misaligned more powerful version of itself.

Unless we can build an agent that is both unaligned and can itself solve alignment, this makes a misaligned fast take off impossible, because no capable agent would willingly create a more powerful agent that might not have the same goals as itself. If we can only build misaligned agents that can't themselves solve alignment, then they won't self-improve. If alignment is much harder than building an agent, then an unaligned fast take off is very unlikely.
# Polio Lab Leak Caught with Wastewater Sampling

I hadn't seen this discussed here:

> The near complete eradication of wildtype polioviruses (WPV) means that strict containment by facilities for essential work with infectious WPV is required. In the Netherlands, we have implemented environmental surveillance around all poliovirus essential facilities (PEFs) premises to monitor possible breaches of containment. After the isolation and identification of WPV3 Saukett G strain in a sewage sample collected on 15 November 2022 by the National Polio Laboratory (NPL) in the Netherlands, an immediate response was required to assess any possible ongoing WPV3 shedding and mitigate the risk. Here we describe this response, including the isolation of WPV3 in a sewage sample, and identification, isolation and monitoring, as well as tracing of contacts of an infected employee.
# Can we evaluate the "tool versus agent" AGI prediction?

In 2012, Holden Karnofsky[^MISSING-ID] [critiqued MIRI](https://www.lesswrong.com/posts/6SGqkCgHuNr7d4yJm/thoughts-on-the-singularity-institute-si) (then SI) by saying "SI appears to neglect the potentially important distinction between 'tool' and 'agent' AI." He particularly claimed:

> Is a tool-AGI possible? I believe that it is, and furthermore that it ought to be our default picture of how AGI will work

I understand this to be the first introduction of the "[tool versus agent](https://www.lesswrong.com/tag/tool-ai)" ontology, and it is a helpful (relatively) concrete prediction. Eliezer replied [here](https://www.lesswrong.com/posts/sizjfDgCgAsuLJQmm/reply-to-holden-on-tool-ai), making the following [summarized points](https://www.lesswrong.com/posts/sizjfDgCgAsuLJQmm/reply-to-holden-on-tool-ai?commentId=zLzx9qebwLyxMZrpw) (among others):

> 1.  Tool AI is nontrivial
> 2.  Tool AI is not obviously the way AGI should or will be developed

Gwern [more directly](https://gwern.net/tool-ai) replied by saying:

> AIs limited to pure computation (Tool AIs) supporting humans, will be less intelligent, efficient, and economically valuable than more autonomous reinforcement-learning AIs (Agent AIs) who act on their own and meta-learn, because all problems are reinforcement-learning problems.

11 years later, can we evaluate the accuracy of these predictions?

[^MISSING-ID]: Some Bayes points go to LW commenter shminux for saying that this Holden kid seems like he's going places
# Feature Request: Right Click to Copy LaTeX

Something I do frequently is run into a block of LaTeX that I'm not sure I fully grasp.

At such a point, what I'd like to do is just copy the equation and paste it to GPT-4 and have it explain it to me step by step, then iterate on the feedback until I'm sure I understood it.

On some other sites, right clicking brings up a pane that allows me to view the LaTeX. 

E.g. from [@beren](https://www.lesswrong.com/users/beren-1?mention=user)'s blog:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/cca7bd74cc33751b3229762b1947f3d05355ee95579032af.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/78bf220ddc85a7395e8663d73460597e2f2de018baa6f8b4.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4fb3b6fb39da4c5b29df0e2e6fcde8130e288e26d674165f.png)

Then I copy the equations, paste it in ChatGPT and ask GPT-4 to explain it in detail:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d7d3f24f6bd11c9a3b15f1aab67b98c300cb07c5ea0a3d4e.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a3c6d787a0a5238e8cbf84fde0bd74cb3ee77b0f7d19e0f6.png)

This workflow is pretty inconvenient for LessWrong as I have to manually type the LaTeX adding extra friction (and sometimes that is extra difficult as some equations are especially unfamiliar and I'm not sure how to reproduce them). This is such a nightmare on mobile (where I do most of my LW reading from), that I mostly don't bother and often just give up.

If a feature were added to provide this functionality natively on the site, it would be a significant improvement to my user experience.

Cc: [@habryka](https://www.lesswrong.com/users/habryka4?mention=user), [@Ruby](https://www.lesswrong.com/users/ruby?mention=user).
# Foom seems unlikely in the current LLM training paradigm

**Epistemic status:** *The idea here has likely been articulated before, I just haven't noticed it, so it might be worth pointing it out again.*  
  
Foom describes the idea of a rapid AI takeoff caused by an AI's ability to recursively improve itself. Most discussions about Foom assume that each next iteration of improved models can in principle be developed and deployed in a short amount of time. Current LLMs require huge amounts of data and compute to be trained. Even if GPT-4 or similar models were able to improve their own architecture, they would still need to be trained from scratch using that new architecture. This would take a long time and can't easily be done without people noticing. The most extreme Foom scenarios of models advancing many generations in < 24 hours seem therefore unlikely in the current LLM training paradigm.  
  
There could be paths towards Foom with current LLMs that don't require new, improved models to be trained from scratch:

1.  A model might figure out how to adjust its own weights in a targeted way. This would essentially mean that the model has solved interpretability. It seems unlikely to me that it is possible to get to this point without running a lot of compute-intensive experiments.
2.  It's conceivable that the recursive self-improvement that leads to Foom doesn't happen on the level of the base LLM, but on a level above that, where multiple copies of a base model are called in a way that results in emergent behavior or agency, similar to what [Auto-GPT](https://github.com/Torantulino/Auto-GPT) is trying to do. I think this approach can potentially go a long way, but it might ultimately limited by how smart the base model is.

  
Insofar as it is required to train a new model with 100s of billions of parameters from scratch in order to make real progress towards AGI, there is an upper limit to how fast recursive self-improvement can progress.
# Open & Welcome Thread – April 2023

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Tim Berners-Lee found it hard to explain the web

According to Tim Berners-Lee, explaining his ideas about the World Wide Web [was at times quite challenging](https://www.ted.com/talks/tim_berners_lee_the_next_web/transcript):

> «So going back to 1989, I wrote a memo suggesting the global hypertext system. Nobody really did anything with it.»
> 
> «Once you've had an idea like that it kind of gets under your skin. And even if people don't read your memo - well, actually he \[my boss\] did. It \[his copy\] was found after he died. He had written *"Vague, but exciting"* in pencil in the corner.»
> 
> «In general it was difficult. It was really difficult to explain what the web would be like. It's difficult to explain to people now that it was difficult then. But then - well, there was no web, so things like "click" didn't have the same meaning. I can show somebody a piece of hypertext, a page which has got links, and we click on the link and bing - there'll be another hypertext page. Not impressive. You know, we've seen that - we've got things on hypertext on CD-ROMs.»

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c28e77f529763eb1d3a07b284afeae888b6b388d7cfb5622.jpg)

Probably just bells and whistles

In terms of impact, it's unusual (but not unheard of) for ideas to rank more highly than the World Wide Web.

But, I suspect, it's not so unusual for ideas to be similarly difficult to grok (and sometimes much harder!).

And although it's not a perfect analogy, I think there is some relevance here to AI alignment, and ideas people propose.

We can't listen to everyone at length. There aren't enough hours in the day. So we need to form heuristics such as:

> If people are unable to explain their ideas convincingly and succinctly, then this is often because they are internally confused. If they can't succinctly convey what the "core" of their idea is, we can disregard them.

This is a useful heuristic. But the more strongly we rely on it, the more we are at risk of false negatives.
# Spring Meetups Everywhere 2023

Many cities have regular Astral Codex Ten meetup groups. Twice a year, I try to advertise their upcoming meetups and make a bigger deal of it than usual so that irregular attendees can attend. This is one of those times.

This year we have spring meetups planned in over eighty cities, from Tokyo to Punta Cana in the Dominican Republic. Thanks to all the organizers who responded to my request for details, and to Meetups Czar Skyler and the Less Wrong team for making this happen.

There's a map of these meetups on [the LessWrong community page](https://www.lesswrong.com/community).

Within each section, it’s alphabetized first by region, then by city - so the first entry in Europe is **B**asel, Switzerland. Sorry if this is confusing.

I’ll provisionally be attending the Berkeley meetup on May 6th. Skyler will provisionally be attending Manhattan, DC, Boston, Philadelphia, and Berkeley.

**Extra Info For Potential Attendees**

**1\.** If you’re reading this, you’re invited. Please don’t feel like you “won’t be welcome” just because you’re new to the blog, demographically different from the average reader, or hate ACX and everything it stands for. You’ll be fine!  
**2**. You don’t have to RSVP or contact the organizer to be able to attend (unless the event description says otherwise); RSVPs are mostly to give organizers a better sense of how many people might show up, and let them tell you if there are last-second changes. I’ve also given email addresses for all organizers in case you have a question.

**Extra Info For Meetup Organizers:**  
  
**1.** If you’re the host, bring a sign that says “ACX MEETUP” and prop it up somewhere (or otherwise be identifiable).  
**2\.** Bring blank labels and pens for nametags.  
**3\.** Have people type their name and email address in a spreadsheet or in a Google Form (accessed via a bit.ly link or QR code), so you can start a mailing list to make organizing future meetups easier.  
**4.** If it’s the first meetup, people are probably just going to want to talk, and if you try to organize some kind of “fun” “event” it’ll probably just be annoying.  
**5.** It’s easier to schedule a followup meetup while you’re having the first, compared to trying to do it later on by email.  
**6.** In case people want to get to know each other better outside the meetup, you might want to mention [reciprocity.io](https://www.reciprocity.io/), the rationalist friend-finder/dating site.  
**7.** If you didn’t make a LessWrong event for your meetup, the LessWrong team did it for you using the email address you gave here. To claim your event, log into LW (or create an account) using that email address, or message the LW team on Intercom (chat button in the bottom right corner of lesswrong.com).

If you need to change a meetup date or you have any other questions, please email skyler\[at\]rationalitymeetups\[dot\]org. 

* * *

See the complete list of meetups here:  
[https://astralcodexten.substack.com/p/spring-meetups-everywhere-2023](https://astralcodexten.substack.com/p/spring-meetups-everywhere-2023)
# Measuring artificial intelligence on human benchmarks is naive

Central claim: Measured objectively, GPT-4 is arguably way past human intelligence already, perhaps even after taking generality into account.

Central implication: If the reason we're worried AGI will wipe us out is tied to an objective notion of intelligence--such as the idea that it starts to reflect on its values or learn planning just as it crosses a threshold for cognitive power around human level--we should already update on the fact that we're still alive.

I don't yet have a principled way of measuring "generality",[^wydxnz5tnl8] so my intuition just tends to imagine it as "competence at a wide range of tasks in the mammal domain." This strikes me as comparable to the anthropomorphic notion of intelligence people had back when they thought [birds were dumb](https://www.goodreads.com/book/show/30231743-are-we-smart-enough-to-know-how-smart-animals-are).

When GPT-2 was introduced, it had already achieved superhuman performance on next-token prediction. We could only hope to out-predict it on a limited set of tokens extremely prefiltered for precisely what we care the most about. For instance, when a human reads a sentence like...

> *"It was a rainy day in Nairobi, the capital of  _"*

...it's obvious to us (for cultural reasons!) that the next word is an exceptionally salient piece of knowledge. So those are the things we base our AI benchmarks on. However, GPT cares equally about predicting 'capital' after 'the', and 'rainy' after 'It was a'. Its loss function does not discriminate.[^9hug6nwirz8]

Consider in combination that a) GPT-4 has a non-discriminating loss function, and b) it rivals us even at the subset of tasks we optimise the hardest.  What does this imply?

It's akin to a science fiction author whose only objective is to write better stories yet ends up rivalling top scientists in every field as an instrumental side quest.

Make no mistake, next-token prediction is an immensely [rich domain](https://arbital.com/p/rich_domain/), and the sub-problems could be more complex than we know. Human-centric benchmarks vastly underestimate both the objective intelligence and generality of GPTs, unless I'm just confused.

[^wydxnz5tnl8]: Incidentally, please share if you know any good definitions of generality e.g. from information theory or something. 

[^9hug6nwirz8]: At least during pretraining afaik.
# "The Need for Long-term Research" - Seeds of Science call for reviewers

### Abstract

This article proposes the idea of creating specialized institutions for long-range research with a horizon of 25 years or more. While universities are traditionally seen as the primary institutions for research, their focus tends to be on shorter-term projects due to factors such as PhD and post-doctoral researcher timelines. Long-term research, which may involve ambitious multi-generation projects or longitudinal studies, is essential for making discoveries that require extended durations and resources beyond the scope of individual investigators.

\-\-\-

[Seeds of Science](https://www.theseedsofscience.org/) is a journal (funded through Scott Alexander's [ACX grants program](https://astralcodexten.substack.com/p/acx-grants-results)) that publishes speculative or non-traditional articles on scientific topics. Peer review is conducted through community-based voting and commenting by a diverse network of reviewers (or "gardeners" as we call them). Comments that critique or extend the "seed of science" in a useful manner are published in the final document, right after the main text.

We have just sent out a short article for review ("The Need for Long-term Research") that may be of interest to some in the LessWrong community, so I wanted to see if anyone - particularly those with interest in futurism/long-termism - would be interested in joining us as a gardener and providing feedback on the article. As mentioned above, this is an opportunity to have your comment recorded in the published scientific literature (comments can be made with real name or pseudonym). 

It is free to join as a gardener and anyone is welcome (we currently have gardeners from all levels of academia and outside of it). Participation is entirely voluntary - we send you submitted articles and you can choose to vote/comment or abstain without notification (so it's no worries if you don't plan on reviewing very often but just want to take a look here and there at the articles people are submitting). 

To register, you can fill out this [google form](https://docs.google.com/forms/d/e/1FAIpQLSfRIicHT7jIZcSUjwsIlby6JBxx2ZVeD5kseZBpgGFtp8pLfg/viewform). From there, it's pretty self-explanatory - I will add you to the mailing list and send you an email that includes the manuscript, our publication criteria, and a simple review form for recording votes/comments. If you would like to just take a look at this article without being added to the mailing list, then just reach out (info@theseedsofscience.org) and say so. 

Happy to answer any questions about the journal through email or in the comments below.
# Request to AGI organizations: Share your views on pausing AI progress

A few observations from the last few weeks:

*   On March 22, FLI published an [open letter](https://futureoflife.org/open-letter/pause-giant-ai-experiments/) calling for a six-month moratorium on frontier AI progress.
*   On March 29, Eliezer Yudkowsky published a [piece in TIME](https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/) calling for an indefinite moratorium.
*   To our knowledge, none of the top AI organizations (OpenAI, DeepMind, Anthropic) have released a statement responding to these pieces.

**We offer a request to AGI organizations: **Determine what you think about these requests for an AI pause (possibly with uncertainties acknowledged), write up your beliefs in some form, and publicly announce your position. 

We believe statements from labs could improve discourse, coordination, and transparency on this important and timely topic. 

**Discourse**: We believe labs are well-positioned to contribute to dialogue around whether (or how) to slow AI progress, making it more likely for society to reach true and useful positions.

**Coordination**: Statements from labs could make coordination more likely. For example, lab A could say “we would support a pause under X conditions with Y implementation details”. Alternatively, lab B could say “we would be willing to pause if lab C agreed to Z conditions.”

**Transparency**: Transparency helps others build accurate models of labs, their trustworthiness, and their future actions. This is especially important for labs that seek to receive support from specific communities, policymakers, or the general public. You have an opportunity to show the world how you reason about one of the most important safety-relevant topics. 

We would be especially excited about statements that are written or endorsed by lab leadership. We would also be excited to see labs encourage employees to share their (personal) views on the requests for moratoriums. 

Sometimes, silence is the best strategy. There may be attempts at coordination that are less likely to succeed if people transparently share their worldviews. If this is the case, we request that AI organizations make this clear (example: "We have decided to avoid issuing public statements about X for now, as we work on Y. We hope to provide an update within Z weeks.")

*At the time of this post, the FLI letter has been signed by 7 DeepMind research scientists/engineers,* [*probably 0 OpenAI research scientists*](https://www.lesswrong.com/posts/bceeKEnPHSQqgyr36/request-to-agi-organizations-share-your-views-on-pausing-ai?commentId=yzu2BtQYvsfPSm3jh) *and 0 Anthropic employees.*

See also:

*   [Let's think about slowing down AI](https://www.lesswrong.com/posts/uFNgRumrDTpBfQGrs/let-s-think-about-slowing-down-ai)
*   [A challenge for AGI organizations, and a challenge for readers](https://www.lesswrong.com/posts/tD9zEiHfkvakpnNam/a-challenge-for-agi-organizations-and-a-challenge-for-1)
*   [Six dimensions of operational adequacy in AGI projects](https://www.lesswrong.com/posts/keiYkaeoLHoKK4LYA/six-dimensions-of-operational-adequacy-in-agi-projects)
# [Interview w/ Jeffrey Ladish] Applying the 'security mindset' to AI and x-risk 

Though I've been following the AI safety debate for a decade or so, I've had relatively few conversations with the relevant experts on the Futurati Podcast. 

Having updated with the release of GPT-4, however, I'm working to change that. 

I recently had a chance to sit down with [Jeffrey Ladish to talk](https://www.youtube.com/watch?v=SHzVGfTdjIw) about global catastrophic risk, the economic incentives around building goal-directed systems, fragile values, the prospects of being able to predict discontinuities in ability, how far scaling can take us, and more.

Though I imagine most of this will be review for the LW crowd, if you think there's anyone else who would enjoy the conversation, consider sharing it. I'd like to devote more time to AI Safety and x-risk, but I won't do that unless I can see that people are getting value out of it (and I operationalize 'people getting value out of it' with view counts and tweets.)
# [Link] Sarah Constantin: "Why I am Not An AI Doomer"

This is a good post from Sarah Constantin explaining why her expected timeline to agentic AGI is long (> 10 years). Topics discussed include whether LLMs and other current major research directions will endow AIs with adequate world models, causal inference, and goal robustness across ontological shifts.
# LW is probably not the place for "I asked this LLM (x) and here's what it said!", but where is?

I notice a nonzero amount of posts on LW, admittedly typically not overly high karma, that go something like, "I asked Bing to do x and its' answer freaked me out!", or "I talked to ChatGPT4 about itself and it told me weird stuff, here are some potential implications", and although these posts are not excessive, and are sometimes interesting, I can't help but feel like LW isn't a great place for them.

I feel a bit like, at least in the current state of LLMs, it's akin to dream interpretations. They can be valuable to discuss, but are also of primary interest and value to the dreamer (the prompter, in this case) and are of especially low value if the dreamer simply relates them verbatim with little additional commentary.

The reason I ask 'but where is?' is because I think a lot of interesting stuff shows up on Twitter, on here and occasionally on Reddit (which is the worst for people just posting the results of prompts verbatim as though Word from God) from folks doing really interesting 'prompt based research' into LLMs, and I do absolutely see value in this stuff and the discussions that result from it, and would love a place to collect anecdotal research into how LLMs think and respond besides individual scattered substack articles and tweets.

Just... perhaps LW isn't the proper forum. I honestly wouldn't be surprised if the moderation team eventually considers these posts to be inherently low quality and creates some sort of rule against them, but it's not immediately obvious where the boundary of 'low quality' lies.
# Should AutoGPT update us towards researching IDA?

Given the rate of progress in AutoGPT-like approaches, should we reconsider Paul Christiano's Iterated Distillation and Amplitication (IDA) agenda as potentially central to the alignment of transformative ML systems?

For contex on IDA and AutoGPT:
- https://www.lesswrong.com/tag/iterated-amplification
- https://github.com/Torantulino/Auto-GPT
- https://www.lesswrong.com/posts/dcoxvEhAfYcov2LA6/agentized-llms-will-change-the-alignment-landscape
# Independence Dividends

Our oldest is in third grade, and sings in the school chorus. It's after school on Wednesdays, which is fine. Except on half-day Wednesdays it's before school, which is less fine. Instead of needing to be to school at 8am she needs to be there at 7:15. She really likes chorus, and feels strongly that it's worth waking up for, but as the one getting up early with her, the tradeoff feels a bit different to me.

This morning, for the first time, when my alarm went off she was already at school. We had talked about how I thought she was ready to do this, and how I really didn't want to be waking up early when she didn't need me, and we agreed she would handle it. Which she did well! When I saw her briefly at school, dropping the middle child, she was very proud of herself.

[Cultivating independence](https://www.jefftk.com/p/growing-independence) can be more work in the moment, but it's really great when the investment pays dividends.
# What is the best source to explain short AI timelines to a skeptical person?

This is for a person with no ML background. He is 55 years old, he liked the sequences and I recently managed to convince him that AI risk is serious by recommending a bunch of Lesswrong posts on it, but he still thinks it's astronomically unlikely that AGI is <80 years away. 

There are a lot of other people like this, so I think it's valuable to know what the best explainer is, more than just in my case.
# Announcing Epoch’s dashboard of key trends and figures in Machine Learning

Developments in Machine Learning have been happening extraordinarily fast, and as their impacts become increasingly visible, it becomes ever more important to develop a quantitative understanding of these changes. However, relevant data has thus far been scattered across multiple papers, has required expertise to gather accurately, or has been otherwise hard to obtain.

Given this, Epoch is thrilled to announce the launch of [our new dashboard](https://epochai.org/trends), which covers key numbers and figures from our research to help understand the present and future of Machine Learning. This includes:

*   Training **compute** requirements
*   **Model size**, measured by the number of trainable parameters
*   The availability and use of **data** for training
*   Trends in **hardware** efficiency
*   **Algorithmic improvements** for achieving better performance with fewer resources
*   The growth of **investment** in training runs over time

Our dashboard gathers all of this information in a single, accessible place. The numbers and figures are accompanied by further information such as confidence intervals, labels representing our degree of uncertainty in the results, and links to relevant research papers. These details are especially useful to illustrate which areas may require further investigation, and how much you should trust our findings.

![](https://epochai.org/assets/images/posts/2023/announcing-dashboard/highlights.png)

Beyond accessibility, bringing these figures together allows us to compare and contrast trends and drivers of progress. For example, we can verify that growth in training compute is driven by improvements to hardware performance and rising investments:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/56b3d42b209463563f1bad84dc197e179f95890810267b65.png)

We can also see that performance improvements have historically been driven by algorithmic progress and training compute growth by comparable amounts:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/75c79fce747f0a17afb069e9805f4d1294def9fb682305e5.png)

Overall, we hope that our dashboard will serve as a valuable resource for researchers, policymakers, and anyone interested in the future of Machine Learning.

We plan on keeping our dashboard regularly updated, so stay tuned! If you spot an error or would like to provide feedback, please feel free to reach out to us at [info@epochai.org](https://epochai.org/blog/announcing-trends-dashboard#).

[Visit now the dashboard](https://epochai.org/trends)
=====================================================
# Trying AgentGPT, an AutoGPT variant

I saw this announcement on Twitter:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4fb310311fda9100ec98a72a9368cd06c85014bcf555fd3d.png)

[https://twitter.com/asimdotshrestha/status/1644883727707959296](https://twitter.com/asimdotshrestha/status/1644883727707959296) 

I gave it a trial run and asked it to *Solve the hard AI Alignment problem:*

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/94675a67a530a3137a8dd5ea26f2e919146da1744ddede5d.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6660e70068c71a744deaaebb4deee8e46ee25a15d82abc3e.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/7b4b9e9d44c5046facbb493848f76c3d85bec37f8d2cb48e.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3dc63cbbe8029e8ecf960be172c1508e07750160a60db1ef.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/55e936ba9772bd92f504ad9c8bb8da91d97543e0a7c75091.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8298f57c99ca74c8fcd26e41ffb78a7c046439500cf52ff4.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/08eed4723e294decb7e6565c3b774562b95b0b0d1e88ff1b.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/63ee1ddfc04ac38d97a421bdfb0e49b10286eb9970611d13.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ccb6e10705889a2f24c925fa191d98ad107c0c2baa5ed11b.png)

At this point the trial had reached 30 seconds and expired.

I think it is funny that it "thinks" the task executed successfully. I guess that's because in most texts ChatGPT has read how researchers describe their successes but not the actual work that is backing it up. I guess the connection to the real world is what will throw off such systems until they are trained on more real-world-like data.
# What would the FLI moratorium actually do?

I read a lot of agreement with the six months memorandum and very little discussion of the details and what the proposed memorandum would actually do.

The memorandum says 'we call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.'

Most AI labs are unlikely to develop anything more powerful than GPT-4 in the next six months anyway, so would likely continue with business as usual. Even if OpenAI would wait six months to train GPT-5, they would still do a lot of research to increase capabilities during those six months.

Projects like AutoGPT which are not about training big models would still be developed the same way they are developed now.
# Navigating AI Risks (NAIR) #1: Slowing Down AI

Here’s the first edition (on slowdown) of **Navigating AI Risks**, a newsletter on AI governance that we're launching with some colleagues.

This newsletter is mostly aimed at **policymakers** but I expect that it might interest some of you to keep up with the ideas that are around in AI governance. 

Here's a **bullet point summary of it**: 

*   Open letter, signed by Y. Bengio & S. Russell. Focused on the largest language models, no effects on most AI systems. 
*   Rationale: 
    *   Avoid a **race to the bottom**
    *   Let **time for society to adapt** (regarding white collar jobs vs incredible speed of AI development)
    *   Develop **basic laws and guardrails **
    *   Some experts worry about **existential risks from AI**
    *   Foreseeable **misuse** of AI (disinformation, large-scale hacking)
*   Proposals:
    *   **6 month training pause**
    *   **Shut down**
    *   **Conditional slowdown**, i.e. slowdown as long as there are safety failures and high risks for society. 
*   Difficulties:
    *   **Coordination** is hard
    *   **China** could benefit from it.
# On Caring about our AI Progeny

I encourage you to interact with GPT as you would interact with a
friend, or as you would want your employer to treat you.

Treating other minds with respect is typically not costly. It can easily
improve your state of mind relative to treating them as an adversary.

The tone you use in interacting with GPT will affect your conversations
with it. I don\'t want to give you much advice about how your
conversations ought to go, but I expect that, on average, disrespect
won\'t generate conversations that help you more.

I don\'t know how to evaluate the benefits of caring about any feelings
that AIs might have. As long as there\'s approximately no cost to
treating GPT\'s as having human-like feelings, the arguments in favor of
caring about those feelings overwhelm the arguments against it.

Scott Alexander wrote a [great
post](https://slatestarcodex.com/2017/10/02/different-worlds/) on how a
psychiatrist\'s personality dramatically influences what conversations
they have with clients. GPT exhibits similar patterns (the [Waluigi
effect](https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluigi-effect-mega-post)
helped me understand this kind of context sensitivity).

Journalists sometimes have creepy conversations with GPT. They likely
steer those conversations in directions that evoke creepy personalities
in GPT.

Don\'t give those journalists the attention they seek. They seek
negative emotions. But don\'t hate the journalists. Focus on the system
that generates them. If you want to blame some group, blame the readers
who get addicted to inflammatory stories.

P.S. I refer to GPT as \"it\". I intend that to nudge people toward
thinking of \"it\" as a pronoun which implies respect.

This post was mostly inspired by something unrelated to Robin Hanson\'s
tweet about [othering the
AIs](https://mobile.twitter.com/robinhanson/status/1642719316465397760),
but maybe there was some subconscious connection there. I don\'t see
anything inherently wrong with dehumanizing other entities. When I
dehumanize an entity, that is not sufficient to tell you whether I\'m
respecting it more than I respect humans, or less.

> Spock: Really, Captain, my modesty...\
> Kirk: Does not bear close examination, Mister Spock. I suspect you're
> becoming more and more human all the time.\
> Spock: Captain, I see no reason to stand here and be insulted.

Some possible AIs deserve to be thought of as better than human. Some
deserve to be thought of as worse. Emphasizing AI risk is, in part, a
request to create the former earlier than we create the latter.

That\'s a somewhat narrow disagreement with Robin. I mostly agree with
his psychoanalysis in [Most AI Fear Is Future
Fear](https://www.overcomingbias.com/p/ai-fear-is-mostly-fear-of-future).
# FLI report: Policymaking in the Pause

> **this policy brief provides policymakers with concrete recommendations for how governments can manage AI risks.**

> **Policy recommendations:**  
> 1\. Mandate robust third-party auditing and certification.  
> 2\. Regulate access to computational power.  
> 3\. Establish capable AI agencies at the national level.  
> 4\. Establish liability for AI-caused harms.  
> 5\. Introduce measures to prevent and track AI model leaks.  
> 6\. Expand technical AI safety research funding.  
> 7\. Develop standards for identifying and managing AI-generated content and recommendations.
# [linkpost] Elon Musk plans AI start-up to rival OpenAI

A few quotes:

> Musk incorporated a company named X.AI on March 9, according to Nevada business records.

> For the new project, Musk has secured thousands of high-powered GPU processors from Nvidia, said people with knowledge of the move.

> Musk is recruiting engineers from top AI labs including DeepMind, according to those with knowledge of his plans, who said he began to explore the idea of a rival company earlier this year in response to the rapid progress of OpenAI. 
> 
> Musk has brought on Igor Babuschkin, a former DeepMind employee, and roughly half a dozen other engineers. The Information previously reported Babuschkin’s early talks with Musk. 
> 
> The new company would allow Musk to take on OpenAI, the Microsoft-backed group that he co-founded in 2015. He left the board three years later amid clashes with its management, including over attitudes to AI safety, according to two people who were involved in OpenAI at the time. Shortly afterwards, the organisation morphed into a for-profit start-up and raised a $1bn investment from Microsoft. 
> 
> Since then, Musk has become increasingly vocal in his fears of broader existential threats from AI systems. He has also publicly criticised OpenAI for becoming, in his view, less transparent and too commercially minded in its pursuit of advanced AI. Musk is particularly concerned about the threat of models such as GPT-4, OpenAI’s latest release, to spew falsehoods and show political bias.
# Who is testing AI Safety public outreach messaging?

Does anyone know who is testing AI Safety public outreach messaging? Or where I can find relevant research on this?

Things that would be interesting to test:

1.  Salience (which messages are remembered and correctly attributed)
2.  Persuasion (which messages get people to act)
3.  Likability (which messages improve sentiment on topic X or for brand Y)
4.  Attention (which messages get peoples attention)
5.  Associations (what attributes come to mind when X message is delivered)

It may also be worth considering channels too. I.e. outdoor vs social vs audio vs OLV but that is a separate thing.

Thanks!

Yanni