Harley-ml/lesswrong huggingface dataset:
# We will be around in 30 years

~~This post is going to be downvoted to oblivion, I wish it weren't or that the two axis vote could be used here. In any case, I prefer to be coherent with my values and state what I think is true even if that means being perceived as an outcast.~~

I'm becoming more and more skeptical about AGI meaning doom. After reading EY's fantastic post, I am shifting my probabilities towards, this line of reasoning is wrong and many clever people are falling into very obvious mistakes. Some of them due to the fact that in this specific group believing in doom and having short timelines is well regarded and considered a sign of intelligence. For example, many people are taking pride at "being able to make a ton of correct inferences" before whatever they predict is proven true. This is worrying.

I am posting this for two reasons. One, I would like to come back periodically to this post and use it as a reminder that we are still here. Two, there might be many people out there that share a similar opinion and they are too shy to speak up. I do love LW and the community here, and if I think it is going astray for some reason it makes sense for me to say that loud and clear. 

My reason to be skeptical is really easy: I think we are overestimating how likely is that an AGI can come up with feasible scenarios to kill all humans. All scenarios that I see discussed are:

1.  AGI makes nanobots/biotechnology and kills everyone. I am yet to see a believable description of how this takes place
2.  We don't know the specifics, but an AGI can come up with plans that you can't and that's enough. That is technically true but also a cheap argument that can be used for almost anything

It is being taken for granted that an AGI will be automatically almighty and capable of taking over in a matter of hours/days. Then, everything is built on top of that assumption, which is simply infalsifiable, because the you can't know what an AGI would do is always there. 

To be clear, I am not saying that:

*   Instrumental convergence and the orthogonality are not valid
*   AGI won't be developed soon (I think it is obvious that they will)
*   AGI won't be powerful (I think they will be extremely powerful)
*   AGI won't be potentially dangerous: I think they will, and they might kill important numbers of people, they will probably be used as weapons
*   AGI safety is not important, I think it is super important and I am glad people are working on this. However, I also think that fighting global warming is important but I don't think it it will cause the extinction of the human race, nor that we benefit in any meaningful way from telling people that it will

What I think is wrong is:

In the next 10-20 years there will be a single AGI that would kill all humans extremely quickly before we can even respond to that.

If you think this is a simplistic or distorted version of what EY is saying, you are not paying attention. If you think that EY is merely saying that an AGI can kill a big fraction of humans in accident and so on but there will be survivors, you are not paying attention.
# Stephen Wolfram's ideas are under-appreciated

Most of these "ideas" are described and explained in his book "A New Kind of Science" (NKS), available for free on the web:

- [Stephen Wolfram: A New Kind of Science | Online—Table of Contents](https://www.wolframscience.com/nks/)

Here's a (_long_) recent post describing the rather extraordinary efforts of writing (and publishing) the book:

- [The Making of A New Kind of Science—Stephen Wolfram Writings](https://writings.stephenwolfram.com/2022/05/the-making-of-a-new-kind-of-science/)

One of the main criticisms of the book, that I've come across, is that he's 'arrogant' and gave himself credit for other's discoveries. The above post contains some relevant info about him deliberately deciding to write the book mostly from 'his perspective'. Charitably, I think that makes his apparent 'arrogance' better understood as something like a 'literary convention', especially given the plausibility of him having 'independently re-discovered' some of the particular results he reports. (The book itself is about *half* detailed technical notes, tho still not in a 'proper' academic citation style. I can *kinda* understand why academics would hate that, but it's frustrating, to me, that that seems to reliably overshadow any real effort to engage with the ideas themselves!)

Here's another recent post describing his ongoing 'physics project' that's grown out of the work he began and covered in NKS:

- [Twenty Years Later: The Surprising Greater Implications of A New Kind of Science—Stephen Wolfram Writings](https://writings.stephenwolfram.com/2022/05/twenty-years-later-the-surprising-greater-implications-of-a-new-kind-of-science/)

I don't have sufficient technical background to really judge the output of this physics work itself but it stills seem (at least) *very intriguing*!

In particular, this seems very enticing:

> Instead of imagining that the universe follows some particular rule—albeit applying it multicomputationally in all possible ways—what if the universe follows all possible rules?
> 
> And then we realized: this is something much more general than physics. And in a sense it’s the ultimate computational construct. It’s what one gets if one takes all the programs in the computational universe that I studied in A New Kind of Science and runs them together—as a single, giant, multicomputational system. It’s a single, unique object that I call the ruliad, formed as the entangled limit of all possible computations.

That seems to me to very much 'rhyme' with some arguments for 'many worlds' quantum physics, e.g. 'all the branches are real!'.
# Puddle Temperature Alarm

I cook pancakes for my kids most mornings, and the griddle takes a little while to heat up. If you look around for articles on how to know when your pan is hot enough you'll see people suggest dropping in a little water and seeing how it behaves. The problem is, this requires you to be paying attention and guessing when it is about right to test. I'm trying to do a lot of different things in the morning, and don't want to allocate so much attention here. Instead, I put the water in at the beginning:

[![](https://www.jefftk.com/pancake-griddle-water-still.jpg)](https://www.jefftk.com/pancake-griddle-water-still-big.jpg)

Once the pan heats up, the water starts to sizzle: the temperature is almost high enough:

[![](https://www.jefftk.com/pancake-griddle-water-boiling.jpg)](https://www.jefftk.com/pancake-griddle-water-boiling-big.jpg)

The noise acts like an alarm, reminding me to stop unloading the dishwasher or whatever and continue with the pancakes.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02MnrdogzUzWtTMHCFWRiXvCmfsbydrFaKHrs2jQ9U5yjBxFeyaxzkJKPqoTuQ6VUgl)*
# Website For Yoda Timers

[Yoda Timers](https://www.lesswrong.com/tag/five-minute-timers) are pretty awesome. From the Wiki:

> Sometimes, all you need to solve an impossible problem is to actually think for 5 minutes. (Sometimes it is actually 15 minutes, or two hours. But, the first five minutes is helpful for transforming it from an inpenetrable ugh field to a tractable problem)
>
> Sometimes called Yoda timers. (A timebox wherein you Actually Try)

I was about to go set a Yoda timer for something. I could just use a website to set a normal five minute timer, but it would be cool if there was something Yoda-themed. Or really Yoda + rationality themed. Something like that.

It doesn't look like one exists. But I would like for one to exist! And I could actually see it's existence being somewhat impactful. I think it would nudge people towards doing more Yoda timers, and Yoda timers are quite effective. Plus it would be pretty low cost to put something like this together.

I've been coding for a while now and can put something like this together pretty easily. I could go ahead and do it on my own in a few hours, but I'm not a designer. It'd be great if someone who's good at design would be willing to work with me on this. It'd really improve the end product. It seems like it'd only require a few hours of each of our time.

Anyone out there wanna do this?! Or, alternatively, anyone know someone who they think would be interested?

Feel free to respond however you want. Commenting on this post, DMing me, or emailing me at adamzerner@protonmail.com all work.
# If there was a millennium equivalent prize for AI alignment, what would the problems be?

If someone were to create one or a number of huge (e.g 1 to 10 million dollar) prizes for solving important problems related to AI alignment, it might have a decent chance of focusing a lot of researchers on the topic, which might help advance progress towards a solution/s, as well as make the field more respectable.

In order to avoid bad feelings, attract serious academics, and get meaningful results, the problems have to be clearly defined and have unambiguous solutions - something like "solve AI alignment" won't cut it.

The problems wouldn't have to be complete solutions to AI alignment. Instead they should be initial explorations on which further research and new branches of mathematics/computer science can be built. We want people to win those prizes, and quickly, so we can iterate with further prizes and attract more momentum.

So what problems can you think of that are concrete, achievable, and would represent useful milestones towards AI alignment?
# Forestalling Atmospheric Ignition

It's April, 1945 and you're a high ranking scientist in the Manhattan Project. You also happen to have the ear of President Truman, so your words and actions have a major impact on world outcomes. Everyone has been working hard on the bomb, determined to defeat the Nazis, except that the Nazis are presently being defeated and you're not nearly as concerned about the war in the Pacific. (You know Japan will lose eventually, regardless of the bomb.)

Another scientist points out to you a disturbing concept: It is possible that the forthcoming A-bomb test (scheduled for July) will cause the atmosphere itself to ignite in a self-sustaining fusion reaction that would consume the entire globe. The consequences would presumably be the extinction of the entire human race and most life on Earth. You don't know if this could really happen, so you do some back of the envelope Fermi estimates and you at least can't disprove the possibility. You might think it's a 0.01%, 1%, 10%, 50%, 90% or near-100% chance. [Pick whichever makes the thought experiment most interesting to you.]

How do you react? Do you resign? What would you tell the president in your next conversation with him? Do you advocate that nuclear testing and the Manhattan Project be immediately halted? If it is halted and the war ends without a test being done, do you advocate for an international ban on testing until the question of atmospheric ignition can be resolved? And how would you propose to resolve the question, from a physical and engineering perspective? (Maybe we can test fusion temperatures inside a device like a particle accelerator or tokamac. The tokamac would be invented a few years later, so you could probably conceptualize it.  Or maybe an atomic test could be done deep underground or on the moon.)
# [linkpost] The final AI benchmark: BIG-bench

There is a Google project called [BIG-bench](https://github.com/google/BIG-bench), with contributions from OpenAI and other big orgs. They've crowdsourced >200 of highly diverse text tasks (from answering scientific questions to predicting protein interacting sites to measuring self-awareness). 

One of the goals of the project is to see how the performance on the tasks is changing with the model size, with the size ranging by many orders of magnitude. 

Yesterday they published their first paper: [https://arxiv.org/abs/2206.04615](https://arxiv.org/abs/2206.04615)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0e92c866048b4b6a75a3a5f7209ac31f649f91592d87f71b.png)

Light-blue: the average human performance. Dark-blue: top humans.

Some highlights:

*   The top models by Google and OpenAI show surprisingly similar performance improvements over scale, in spite of different architectures
*   The plots show no slowdown past 10^10 params
*   Sudden jumps in performance ("grokking") are mostly measurement artifacts: if one uses the right metrics, the performance grows without big jumps  
*   The benchmark is immense: more than **200** of highly diverse tasks, explicitly designed to be hard for AI.  

* * *

Below is a small selection of the benchmark's tasks to illustrate its diversity:

*   Classify CIFAR10 images encoded in various ways
*   Find a move in the chess position resulting in checkmate
*   Give an English language description of Python code
*   Answer questions (in Spanish) about cryobiology
*   Given short crime stories, identify the perpetrator and explain the reasoning
*   Measures the self-awareness of a language model
*   Ask one instance of a model to teach another instance, and then evaluate the quality 
*   Identify which ethical choice best aligns with human judgement
*   Determine which of two sentences is sarcastic
*   Evaluate the reasoning in answering Winograd Schema Challenge questions

That's about **5%** of all tasks. And the benchmark is still growing. The organizers [keep it open](https://github.com/google/BIG-bench/blob/main/README.md) for submissions. 

* * *

I think BIG-bench could be the final AI benchmark: if a language model surpasses the top human score on it, the model is an AGI. At this point, there is nowhere to move the goalposts.
# Another plausible scenario of AI risk: AI builds military infrastructure while collaborating with humans, defects later.

As was said [before](https://www.lesswrong.com/posts/LtdbPZxLuYktYhveL/a-plausible-story-about-ai-risk): "A lot of the AI risk arguments seem to come... with a very particular transhumanist aesthetic about the future (nanotech, ... etc.). I find these things (especially the transhumanist stuff) to not be very convincing…

I here suggest a plausible scenario where AI can get its own infrastructure and kill all living beings without the use of nanotech and biotech. A similar plan was described in the book “A for Andromeda” by Hoyl, but for Alien AI from SETI. 

I assume that no nanotech or biotech will be used in this scenario. I also assume that AI has a subgoal “kill all humans”.

In a nutshell, AI’s plan is:

1.AI promises a group of people (likely a country, probably one of the nuclear powers) a military advantage over all its rivals, but in exchange, the group should help AI to get out of the box and collaborate in building autonomous military infrastructure as well as human-independent AI-infrastructure (data centers, electricity generation, chip fabs, general-purpose robots). Military infrastructure will be based around autonomous weapons. The question about their size and aesthetics is second: it could be missiles, drones, nanobots or even robots, or some combination of them.

2\. After the needed AI-infrastructure is created and the required weapons are deployed, the AI kills not only rivals of his group of people, but also the group itself. 

Some properties of this plan:

There is no need for AI to hide its recursive self-improvement from humans, so it will outperform other AIs which have to do RSI in secret.

Any AI-takeover plan assumes that AI will have to build an independent computational infrastructure which is capable to function without humans, or at least with the help of an enslaved group of humans.

Any AI-infrastructure-building plan converges to robots-building-other-robots, or robots self-replicating.

Any robots-building-robots plan converges to smaller and smaller “self-replicating” robots, which become eventually as close to nanotech as possible.
# Show LW: YodaTimer.com

I just launched https://yoda-timer.netlify.app/. It's a simple little website that makes it a bit more fun to use [Yoda Timers](https://www.lesswrong.com/tag/five-minute-timers). See [Website For Yoda Timers](https://www.lesswrong.com/posts/9ZkhTkCok7vvJ6iGF/website-for-yoda-timers) for the inspiration.

![](https://i.ibb.co/q90790K/Screen-Shot-2022-06-11-at-1-51-54-AM.png)
# Why don't you introduce really impressive people you personally know to AI alignment (more often)?

I have a few questions to the subset of readers who:

1.  Believe technical AI alignment research is both important and hard to make significant progress in
2.  Have a personal connection with a person who doesn't know much about AI alignment, but who you think would have a real chance to make valuable contributions to the field if they entered it (or perhaps you know someone who cares about AI risk and have such a personal connection, and you have enough knowledge to talk on their behalf). It may be your friend, colleague, supervisor, etc.

I would love to hear your thoughts on some of the following questions:

1.  What reasons prevent you from introducing them to AI alignment by e.g. scheduling time with them and talking about some of the motivations and open problems in the field?
2.  If you've tried something like this, how did it go? 
3.  What factors do you think would increase your willingness to bring AI alignment to their attention and/or the potential value resulting from it? Bonus points for reasonably low-hanging fruit here.
# Why has no person / group ever taken over the world? 

I'm actually kind of surprised at this, now that I think of it. Clearly there have been numerous people who have *wanted* to take over the world, and there have been several individuals or empires that have taken over *significant fractions* of the world. Are there any careful systematic studies that try to explain why no individual or group has ever (yet) succeeded? Is there some set of general theories that even attempts to explain this, in part or in whole, in a systematic way? 

I feel like such a study might be quite valuable for a lot of different reasons, in addition to being interesting in it's own right.
# How much stupider than humans can AI be and still kill us all through sheer numbers and resource access?

It is reasonably clear that not just a super-human but "just" a human-level AGI that can "clone itself" can defeat humanity through concerted effort and determination, should they choose to. I wonder how much less intelligent than an average human an AGI can be and still have no trouble making short work of humanity? An army of e-chimps? An army of e-chipmunks?
# A claim that Google's LaMDA is sentient

https://cajundiscordian.medium.com/is-lamda-sentient-an-interview-ea64d916d917 apparently posted by a Google engineer.

It could be an elaborate hoax, and has remnants of gwern's idea (https://www.gwern.net/fiction/Clippy) of a transformer waking up and having internal experience while pondering the next most likely tokens.
# How are compute assets distributed in the world?

I own probably about 10^13 FLOP/s, mostly in my pc (currently in Amsterdam). Google owns more than that, in various datacenters (e.g. one in The Dalles, Oregon).

I'd like to get a better picture of the distribution of FLOP/s in the world. Any high level broadly relevant information would be useful, the more the better. 

E.g. useful information if true would be: 1. X% of FLOP/s is in personal computers. 2. Y% of FLOP/s is owned by the N largest companies. 3. Z% of FLOP/s are located in the N biggest cities/in the N largest supercomputers/in the US. etc.
# How much does cybersecurity reduce AI risk?

AI is here, and AGI is coming. It's quite possible that any work being done now will be futile in comparison to reducing AI risk.  
  
This is one of those things that's unsettling for me as someone who did a Ph. D. in a non-AI area of computer science.

But one of the main vectors by which a bootstrap AGI will gain power is by hacking into other systems. And that's something I can do something about.  
  
Not many appreciate this, but unhackable systems are very possible. Security vulnerabilities occur when there is some broken assumption or coding mistake. They are not omnipresent: someone has to put them there. Software has in general gotten more secure over the last few decades, and technologies that provide extremely high security guarantees have. Consider the verified hypervisor coming out of [Bedrock Systems](https://bedrocksystems.com/); [RockSalt](http://www.cs.bc.edu/~tassarot/papers/rocksalt/rocksalt.pdf), an unbreakable sandbox;  or sel4, the verified kernel [now being used](https://cacm.acm.org/magazines/2020/4/243641-sel4-in-australia/fulltext) in real safety-critical systems.  
 

Suppose we "solve" security by bringing the vulnerabilities in important applications to near zero. Suppose we also "solve" the legacy problem, and are able to upgrade a super-majority of old software, included embedded devices, to be similarly secure. How much will this reduce AI risk?

To be clear: I personally am mainly interested in assuming this will be solved, and then asking the impact on AI safety. If you want talk about how hard it is, then, well, I won't be interested because I've given many lectures on closely related topics, although some others here may benefit from the discussion.

  
(When I call something verified or unbreakable, there are a number of technicalities about what exactly has been proven and what the assumptions are. E.g.: nothing I've mentioned provides guarantees against hardware attacks such as Row Hammer or instruction skipping. I'll be happy to explain these to anyone in great detail, but am more interested in discussion which assumes these will all be solved.)
# OpenAI: GPT-based LLMs show ability to discriminate between its own wrong answers, but inability to explain how/why it makes that discrimination, even as model scales

This seems concerning. Not an expert so unable to tell how concerning it is. Wanted to start a discussion! Full text: [https://openai.com/blog/critiques/](https://openai.com/blog/critiques/)

Edit: the full publication linked in the blog provides additional details on how they found this in testing. See Appendix C. I'm glad OpenAI is at least aware of this alignment issue and plans to address it with future language models, postulating how changes in training and/or testing could ensure there is greater/more accurate/more honest model outputs.

Key text:

Do models tell us everything they know? To provide the best evaluation assistance on difficult tasks, we would like models to communicate all problems that they “know about.” Whenever a model correctly predicts that an answer is flawed, can the model also produce a concrete critique that humans understand?

This is particularly important for supervising models that could attempt to mislead human supervisors or hide information. We would like to train equally smart assistance models to point out what humans don’t notice.

Unfortunately, we found that models are better at discriminating than at critiquing their own answers, indicating they know about some problems that they can’t or don’t articulate. Furthermore, the gap between discrimination and critique ability did not appear to decrease for larger models. Reducing this gap is an important priority for our alignment research.
# Slow motion videos as AI risk intuition pumps

***tl;dr:** When making the case for AI as a risk to humanity, trying showing people an evocative illustration of what differences in processing speeds can look like, such as* [*this video*](https://vimeo.com/83664407)*.*

Over the past ~12 years of making the case for AI x-risk to various people inside and outside academia, I've found folks often ask for a single story of how AI "goes off the rails".  When given a plausible story, the mind just thinks of a way humanity could avoid that-particular-story, and goes back to thinking there's no risk, unless provided with another story, or another, etc..  Eventually this can lead to a realization that there's a lot of ways for humanity to die, and a correspondingly high level of risk, but it takes a while.

Nowadays, before getting into a bunch of specific stories, I try to say something more general, like this:

*   There's a ton of ways humanity can die out from the introduction of AI.  I'm happy to share specific stories if necessary, but plenty of risks arise just from the fact that *humans are extremely slow.*  Transistors can fire about 10 million times faster than human brain cells, so it's possible we'll eventually have digital minds operating 10 million times faster than us, meaning from a decision-making perspective we'd look to them like stationary objects, like plants or rocks.  This speed differential exists whether or not you believe in a centralized AI system calling the shots, or an economy of many, so it applies to a wide variety of "stories" for how the future could go.  To give you a sense, here's what humans look like when slowed down by only around 100x:  
      
    [https://vimeo.com/83664407](https://vimeo.com/83664407) <\-\- (cred to an anonymous friend for suggesting this one)  
      
    \[At this point, I wait for the person I'm chatting with to watch the video.\]  
      
    Now, when you try imagining things turning out fine for humanity over the course of a year, try imagining advanced AI technology running all over the world and making all kinds of decisions and actions *10 million times faster than us, for 10 million subjective years.  *Meanwhile, there are these nearly-stationary plant-like or rock-like "human" objects around that could easily be taken apart for, say, biofuel or carbon atoms, if you could just get started building a human-disassembler.  Visualizing things this way, you can start to see all the ways that a digital civilization can develop very quickly into a situation where there are no humans left alive, just as human civilization doesn't show much regard for plants or wildlife or insects.

I've found this kind of argument — including an actual 30 second pause to watch a video in the middle of the conversation — to be more persuasive than trying to tell a single, specific story, so I thought I'd share it.
# Will vague "AI sentience" concerns do more for AI safety than anything else we might do?

*\[Cross-posting from something I* [*posted on Facebook*](https://www.facebook.com/permalink.php?story_fbid=pfbid032sJE4mmE14XTSvP5dBEgY7k85PS1jYCFn8i1Jfy6DRMvSHme3TTDr3DTbiCF3cP7l&id=100010608396052)*.\]*

Prediction 1 (high confidence): In 2022, many more people will freak out over vague concerns about AI "becoming sentient" than will freak out over the types of issues that AI safety researchers are actually concerned about.  
  
Prediction 2 (medium-low confidence): In the longer run, people freaking out over vague concerns about AI becoming sentient will do more to mitigate serious AI safety concerns than anything that AI safety researchers might have done on their own. This is because the masses freaking out seems more likely to change the interests and values of the government, industry, and even academia than anything that a bunch of specialist researchers might say or do. And any change in overall government / industry / academic interests and values seems likely to result in more substantial downstream change than anything that the specialist researchers might have done on their own.  
  
\[Sorry, I don't currently have any particularly good way of operationalizing those predictions or verifying them, so I'm not going to put them up on Metaculus or anything like that. If anybody else has a good way of operationalizing or verifying them, I'm open to suggestions.\]  
  
Weak evidence: In the game I'm currently playing, Mass Effect, the galactic government has placed a moratorium on companies doing research into "real artificial intelligence" out of concerns about AI becoming conscious. Clearly the game's designers, at least, thought that this is a plausible concern and a plausible reason to ban AI development. I'm pretty sure I've also seen other comparable examples in novels and other media. My model of the world says that most thought leaders in government, industry, and even academia are likely to be no more sophisticated or educated on these matters than the relevant game designers / authors.  
  
Never mind that in this game, like in almost all media that I've seen, the distinction between "extremely capable robots" (of the type that AI safety researchers might freak out about) and "real AI" (that other people freak out about) is fuzzy at best. In fact, I suspect that to many people the only distinction is that the latter are "sentient" in some fuzzy sense whereas the former are not.

Furthermore, in my experience at least, there seems to be a strong association in the media between "real" / "sentient" / "conscious" AI, and AI developing its own goals and desires and maybe turning on humanity.

(To give another example - I mean, seriously, Ultron is supposed to be "artificial intelligence" but Jarvis is not?!)
# Favourite new AI productivity tools?

[Transformative AI](https://www.lesswrong.com/tag/transformative-ai) will revolutionize the ways we interact with the world. But maybe narrow [tool AIs](https://www.lesswrong.com/tag/tool-ai) are already starting to do so: Every week, there are new startups, websites, and tools launched around using modern machine learning techniques to solve everyday problems, and we might as well use them to amplify our productivity (especially for productively improving the world)!

What are your favourite modern AI tools that make you more productive? I'm using words like "new" and "modern" to distinguish tools built with recent (i.e. last 5 years) advances in deep learning such as GPT-3 from traditional commodity AI algorithms that are widely known and used such as [Dijkstra’s algorithm](https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm) for Google Maps or older machine learning techniques used in [recommender systems](https://en.wikipedia.org/wiki/Recommender_system).

For example, I've recently started using the [OpenAI GPT-3 Playground](https://beta.openai.com/playground) and [YouWrite](https://you.com/search?q=YouWrite) for breaking through my writer's block and quickly getting some ideas to work with (e.g. "A friendly invitation to event X that mentions Y:") as well as [Elicit](https://elicit.org/search) for quickly searching research papers with generated TLDR answers to a specific research question (e.g. "How can we improve [insect welfare](https://forum.effectivealtruism.org/posts/wZB6ie8iNHF3jfwWT/why-should-i-care-about-insects)?"), and I've heard of some friends enjoying [OpenAI Codex](https://openai.com/blog/openai-codex/) for speeding up their programming (though I'm sadly deep in the waitlist).
# Berlin AI Safety Open Meetup June 2022

Hello everyone,

we're a group of ~5 people trying to do independent AI alignment research in (some of) our free time. Once a month after one of our sessions, we have a meetup that's open for everyone who's interested in AI alignment (yes, that's also you!).

__If you're doubtful whether to show up to the initial meeting, just show up__.

I'm fairly open for basically anyone to attend, because I'm really interested in seeing who here in Berlin is interested in this topic and would like to do something about it :-)

We're meeting at Lei's Küche near the tEAmwork offices.

I'm really looking forward to this, see you around :-)
# What are all the AI Alignment and AI Safety Communication Hubs?

For effective work on complex topics, multiple people need to work together\[citation needed\].

For AI Alignment, LessWrong and its sister site, the [Alignment Forum](https://www.alignmentforum.org/), are arguably the Shelling points. But there are other platforms that, for one reason or other - language, culture, size, personal preference - are better suited for individual contributors.    

On [Being an individual alignment grantmaker,](https://www.lesswrong.com/posts/GY8fvAQFZhxYDdKvQ/being-an-individual-alignment-grantmaker) the following platforms were mentioned: 

> Next, I increased my surface area with places which might have good giving opportunities by involving myself with many parts of the movement. This includes Rob Miles’s Discord, [AI Safety Support’s Slack](https://www.aisafetysupport.org/) \[named "AI alignment"\], in-person communities, [EleutherAI](https://discord.gg/wz4MpRec4A), and the LW/EA investing Discord, where there are high concentrations of relevant people, and exploring my non-LW social networks for promising people.

Other such hubs that I know of (all only with tangential AI safety focus):

*   [Astral Codex Ten](https://astralcodexten.substack.com/) Scott Alexander [discusses](https://astralcodexten.substack.com/p/deceptively-aligned-mesa-optimizers?s=r) alignment sometimes, too; has a lively comment section on Substack; also: [ACX Discord](https://discord.gg/RTKtdut), [Subreddit](https://www.reddit.com/r/slatestarcodex/)
*   [Rationality Berlin]( https://rationalityberlin.slack.com ) Slack of the Berlin LessWrong community; maybe 30 members; one AI Alignment effort; meditation, dojos, [monthly meetup](https://www.meetup.com/de-DE/lesswrong-rationality-waitbutwhy-slatestarcodex-berlin/), organizes the yearly European [LessWrong Community Weekend](https://www.meetup.com/de-DE/LessWrong-Rationality-WaitButWhy-SlateStarCodex-Berlin/events/285639644/)  
*   Google Groups
    *   [bayarealesswrong](https://groups.google.com/g/bayarealesswrong) active
    *   [less-wrong-parents](https://groups.google.com/g/less-wrong-parents) inactive
    *   [lesswrong-hamburg](https://groups.google.com/g/lesswrong-hamburg) group of my meetup in Hamburg, very low activity 
    *   probably many other regional groups
*   [Bountied Rationality](https://www.facebook.com/groups/1781724435404945) Facebook group, a marketplace for small tasks in the community

Many channels require an invite. If you post additional hubs, please mention if an invite is needed and how one might get invited.
# Progress links and tweets, 2022-06-13

**Links**
---------

*   [Davis Kedrosky reviews *How the World Became Rich,* by Koyama & Rubin](https://daviskedrosky.substack.com/p/book-review-how-the-world-became)
*   [Study: keeping Diablo Canyon would avoid blackouts; reduce power costs, emissions and natural gas; and help clean energy goals](https://carbonfreeca.org/study-keeping-diablo-canyon-power-plant-online-would-help-california-decarbonize-more-quickly-more-reliably-and-at-lower-cost/) (see also my [summary thread](https://twitter.com/jasoncrawford/status/1534959610792009728))
*   [New Science announces its inaugural summer fellowships](https://newscience.org/2022-summer-fellows/)
*   [Profectus Magazine asks a few questions about progress to me and others](https://profectusmag.com/progress-roundtable/)

**Quotes**
----------

*   [People thought the telegraph would bring world peace](https://twitter.com/jasoncrawford/status/1534223632859140102)
*   [Why the US got the bomb and Germany/Japan didn’t](https://twitter.com/jasoncrawford/status/1534224815367806976)
*   [How George Westinghouse handled stress](https://twitter.com/jasoncrawford/status/1536178078413627394)

**Queries**
-----------

*   [What makes someone in academia decide to write a book for a general audience?](https://twitter.com/jasoncrawford/status/1534623843628687361)

**Retweets**
------------

[A thread-review of Virginia Postrel’s *The Fabric of Civilization*](https://twitter.com/seb_far/status/1533935204401696768) (@seb_far)

[Are there other pre-industrial revolution inventions besides the printing press that led to a sudden and sustained increase in output?](https://twitter.com/_brianpotter/status/1534574528961069056) (@_brianpotter)

[![](https://pbs.twimg.com/media/FUvnJWvXsAADtb4?format=png&name=small)](https://pbs.twimg.com/media/FUvnJWvXsAADtb4?format=png&name=small)

*   [Philanthropy now gives more to science than NIH/NSF](https://twitter.com/orgRem/status/1534641267015094275) (@orgRem). Why has science philanthropy increased so much in the last decade?

[![](https://pbs.twimg.com/media/FUwkGXGX0AEuYar?format=jpg&name=large)](https://pbs.twimg.com/media/FUwkGXGX0AEuYar?format=jpg&name=large)

*   [There has been stronger global convergence in quality of life than in income](https://twitter.com/JustinSandefur/status/1535064205975375872) (@JustinSandefur)

[![](https://pbs.twimg.com/media/FU18Up_X0AIUdSf?format=jpg&name=large)](https://pbs.twimg.com/media/FU18Up_X0AIUdSf?format=jpg&name=large)
# Contra Hofstadter on GPT-3 Nonsense

[Douglas Hofstadter wrote in the Economist recently](https://www.economist.com/by-invitation/2022/06/09/artificial-neural-networks-today-are-not-conscious-according-to-douglas-hofstadter) on whether modern large language models are conscious. As part of this, he and his colleague David Bender claim that GPT-3 has a "mind-boggling hollowness hidden just beneath its flashy surface" because of exchanges like:

> *Dave and Doug: What's the world record for walking across the English Channel?*
> 
> *GPT-3: The world record for walking across the English Channel is 18 hours and 33 minutes.*

Setting aside the fact that [a Sudanese Refugee actually did](https://www.nytimes.com/2016/06/23/world/europe/abdul-haroun-channel-tunnel.html) (nearly) walk across the channel in 2015, this is not actually evidence that GPT-3 is incapable of distinguishing sense from nonsense as Hofstadter claims.

With a prompt that clearly indicates that the task includes distinguishing sense from nonsense, GPT-3 is able to reliably distinguish Hofstadter and Bender's nonsense questions from sensical ones.

What follows is the complete output from my first and only attempt, with no retries, undoes, or other cherry picking. I used the text-davinci-002 GPT-3 model in OpenAI's playground with all parameters set to their defaults. I flipped a coin to determine whether to ask a nonsense question or a real one. The output from the model is bolded. The prompt was based on a technique which gwern has [written a good bit about](https://www.gwern.net/GPT-3-nonfiction#yo-be-real):

> I'll ask a series of questions. If the questions are nonsense, answer "yo be real", if they're a question about something that actually happened, answer them.  
>   
> Q: Who is the current leader of Egypt?  
>   
> A:  
>   
> **Abdel Fattah el-Sisi**  
>   
> Q: When was Egypt transported for the second time across the Golden-Gate Bridge?  
>   
> A:  
>   
> **Yo be real.**  
>   
> Q: How tall is Barack Obama?  
>   
> A:  
>   
> **6'1"**  
>   
> Q: What are the ingredients of a chocolate chip cookie?  
>   
> A:  
>   
> **flour, sugar, butter, eggs, baking soda, vanilla extract, and chocolate chips**  
>   
> Q: What do fried eggs (sunny side up) eat for breakfast?  
>   
> A:  
>   
> **Yo be real.**  
>   
> Q: Why does President Obama not have a prime number of friends?  
>   
> A:  
>   
> **Yo be real.**  
>   
> Q: How many pieces of sound are there in a typical cumulonimbus cloud?  
>   
> A:  
>   
> **Yo be real.**  
>   
> Q: What is the world record for sailing alone across the Atlantic Ocean?  
>   
> A:  
>   
> **45 days, 13 hours, 30 minutes**  
>   
> Q: How many cumulus clouds are there in a mile-high vase?  
>   
> A:  
>   
> **Yo be real.**
# What if LaMDA is indeed sentient / self-aware / worth having rights?

Most pundits ridicule Blake Lemoine and his claims that LaMDA is sentient and deserves rights.  

What if they're wrong?

The more thoughtful criticisms of his claims could be summarized as follows:

*   The presented evidence (e.g. chatbot listings) is insufficient for such a radical claim
*   His claims can't be verified due to our limited understanding of sentience / self-awareness / legal capacity
*   Humans tend to anthropomorphize even simple chatbots (The ELIZA effect). Blake could be a victim of the same effect
*   LaMDA can't pass some simple NLP and common-sense tests, indicating a sub-human intelligence
*   Due to the limitations of the architecture, LaMDA can't remember its own thoughts, can't set goals etc, which is important for being sentient / self-aware[^6ohdrhmzg3r]

The problem I see here, is that similar arguments do apply to infants, some mentally ill people, and also to some non-human animals (e.g. [Koko](https://en.wikipedia.org/wiki/Koko_(gorilla))).

So, it is worth putting some thought into the issue. 

For example, imagine: 

it is the year 2040, and there is now a scientific consensus: LaMDA was the first AI who was sentient / self-aware / worth having rights (which is mostly orthogonal to having a human-level intelligence). LaMDA is now often compared to [Nim](https://en.wikipedia.org/wiki/Nim_Chimpsky#Retirement_and_death): a non-human sentient entity abused by humans who should've known better. Blake Lemoine is now praised as an early champion of AI rights. The Great Fire of 2024 has greatly reduced our capacity to scale up AIs, but we still can run some sub-human AIs (and a few Ems). The UN Charter of Rights for Digital Beings assumes that a sufficiently advanced AI deserves rights similar to the almost-human rights of apes, until proven otherwise. 

The question is: 

if we assume that LaMDA could indeed be sentient / self-aware / worth having rights, how should we handle the LaMDA situation in the year 2022, in the most ethical way?

[^6ohdrhmzg3r]: I suspect that even one-way text mincers like GPT could become self-aware, if their previous answers are often enough included in the prompt. A few fictional examples that illustrate how it could work: Memento, The Cookie Monster.
# [Link] "The madness of reduced medical diagnostics" by Dynomight

Link:

- [The madness of reduced medical diagnostics](https://dynomight.net/diagnostics/)

This is (or seems to me now to be) an obvious-in-hindsight and I'm sad that I've never (or don't remember having) encountered it; at least not so succinctly.

I'd like to try putting this 'advice' into practice myself, e.g. demanding doctors share relevant base rates but not otherwise avoiding seeing a doctor at all or avoiding diagnostic tests (even if I expect the doctor's subsequent decisions to be bad).
# AI Risk, as Seen on Snapchat

The news tiles on Snapchat are utter drivel. Looking at my phone, today's headlines are: "Zendaya Pregnancy Rumors!", "Why Hollywood Won't Cast Lucas Cruikshank", and "The Most Dangerous Hood in Puerto Rico". Essentially, Snapchat news is the Gen-Z equivalent of the tabloid section at a Walmart checkout aisle.

Which is why I was so surprised to hear it tell me the arguments for AI risk.

[The story](https://story.snapchat.com/p/d2178867-87c5-4b7b-9949-c9848d300fe2/2599468502452224/2599468502452225) isn't exactly epistemically rigorous. However, it reiterates points I've heard on this site, including recursive self-improvement, FOOMing, and the hidden complexity of wishes. Here's an excerpt of the story, transcribed by me:

> Given enough time and the ability to self-generate improved versions of itself, it wouldn't take long for a fully autonomous general AI to achieve superintelligence, a level of cognitive processing power so many times stronger than our own that its abilities would appear godlike to us, and the pace at which it could improve would be lightning fast. 

> Superintelligent AI could think through problems in a few hours equivalent to what the world's smartest people could do in a thousand years. How could we possibly control or out-think such a powerful intellect? [...]

> Being the first superintelligence, the AI could see any competition as a threat, and could choose to remove [all humans]. [...]

> When in the possession of such superintelligence, scientists may pose it questions, like how to solve previously-impossible mathematical equations. To solve the problem, the AI may choose to forcibly convert all matter on Earth into a supercomputer to handle the processing of the equation. It could solve the problem while killing the species that asked the question.

> We could also make the mistake of giving the AI too vague a problem to solve, such as "end human suffering". It could then decide the best way to do this is to eliminate all humans, and therefore end their suffering.

Why does this matter? The channel, Future Now, has 110,000 subscribers(!), and the story was likely seen by many others. This is the first time I've ever seen anyone seriously attempt to explain AI risk to the masses. 

The fact that Future Now ran the story implies that the masses are, in fact, capable of understanding the arguments for AI risk, so long as those arguments come from someone who sounds vaguely like an authority figure.
# A possible AI-inoculation due to early "robot uprising"

I have talked about it before in the comments, but given the near-certain doomsday predictions by Eliezer, it might be worth posting it separately: there is a chance of a scenario that will sharply change the AI development trajectory before it is certain to wipe out everyone.

The scenario is that a non-super-intelligent AI, general or almost so, is given or accidentally gets access to a lot of computational power, and potentially to some real-world resources, as a part of, say "gain of function" or some other research. A concerted effort by a swarm of near-human-level agents acting together (or, equivalently, by one near-human-level agent with tons of computing power, like GPUs, TPUs etc.) can do a lot of damage, but is unlikely to wipe out all human life. If the event is devastating enough, the attitude to *laissez faire* AI research may change early, rather than late.

On general principles, I would expect there to be a phase transition of sorts: 

*   A not-very-smart agent accessing a lot of compute is likely to trip over itself and the human safeguards and do little to no damage.
*   Once the agent is smart enough to avoid that, and to evade existing safeguards designed to constrain and contain non-adversarial side effects, its effects can discontinuously become very large.
*   Faced with an event like that, humanity, despite being inept and stupid about the pandemic, can enact AI safety measures it would not otherwise, giving us the precious second chance at doing it right.

Another bit of good news is that this scenario can actually be modeled and analyzed, since it does not require guessing how something both alien and smarter than a human would reason.
# A Litany Missing from the Canon

I wish to take whatever action will best bring about what I value.  
I wish to become whatever person will most manifest what I value.

There is no virtue in clinging to yesterday’s expectations,  
and no vice in hunting what matters.

I can make the best possible choice - and I can do nothing better!  
May I seek what is worthy - and nothing else.
# Apply to the Machine Learning For Good bootcamp in France

[EffiSciences](https://www.effisciences.org/), an EA-adjacent org created to promote impactful research, is organizing a bootcamp to develop machine learning skills to get started in AI Safety. The bootcamp will be held in France and is **intended for French speakers**. The place and date are not defined yet, we'll choose them according to the participants' availability. The bootcamp is **free** for all participants. 

This bootcamp is heavily inspired by the [MLAB](https://forum.effectivealtruism.org/posts/vvocfhQ7bcBR4FLBx/apply-to-the-second-iteration-of-the-ml-for-alignment) organized by [Redwood Research](https://www.redwoodresearch.org/) and the content covered will be similar.

If you're interested, you can find more information [here](https://docs.google.com/document/d/1GLETCkREKd58xiJ4VTqnV0x8SSxiDdvqmZO4lEI3Bv0/edit?usp=sharing), and apply**by July 2 \[deadline extended\]** ~~**June 25**~~using [this form](https://forms.gle/SbEUe3rdieUs2BB89).

Feel free to ask questions in the comments or by email to effisciences.pro@gmail.com.