A concrete scenario & where its surprises are

Daniel Kokotajlo
We can delete this hook later if no one bites, but in case someone does, here's a scenario I think it would be productive to discuss:

(1) Q1 2024: A bigger, better model than GPT-4 is released by some lab. It's multimodal; it can take a screenshot as input and output not just tokens but keystrokes and mouseclicks and images. Just like with GPT-4 vs. GPT-3.5 vs. GPT-3, it turns out to have new emergent capabilities. Everything GPT-4 can do, it can do better, but there are also some qualitatively new things that it can do (though not super reliably) that GPT-4 couldn't do.
(2) Q3 2024: Said model is fine-tuned to be an agent. It was already better at being strapped into an AutoGPT harness than GPT-4 was, so it was already useful for some things, but now it's being trained on tons of data to be a general-purpose assistant agent. Lots of people are raving about it. It's like another ChatGPT moment; people are using it for all the things they used ChatGPT for but then also a bunch more stuff. Unlike ChatGPT you can just leave it running in the background, working away at some problem or task for you. It can write docs and edit them and fact-check them; it can write code and then debug it.
(3) Q1 2025: Same as (1) all over again: An even bigger model, even better. Also it's not just AutoGPT harness now, it's some more sophisticated harness that someone invented. Also it's good enough to play board games and some video games decently on the first try.
(4) Q3 2025: OK now things are getting serious. The kinks have generally been worked out. This newer model is being continually trained on oodles of data from a huge base of customers; they have it do all sorts of tasks and it tries and sometimes fails and sometimes succeeds and is trained to succeed more often. Gradually the set of tasks it can do reliably expands, over the course of a few months. It doesn't seem to top out; progress is sorta continuous now -- even as the new year comes, there's no plateauing, the system just keeps learning new skills as the training data accumulates. Now many millions of people are basically treating it like a coworker and virtual assistant. People are giving it their passwords and such and letting it handle life admin tasks for them, help with shopping, etc. and of course quite a lot of code is being written by it. Researchers at big AGI labs swear by it, and rumor is that the next version of the system, which is already beginning training, won't be released to the public because the lab won't want their competitors to have access to it. Already there are claims that typical researchers and engineers at AGI labs are approximately doubled in productivity, because they mostly have to just oversee and manage and debug the lightning-fast labor of their AI assistant. And it's continually getting better at doing said debugging itself.
(5) Q1 2026: The next version comes online. It is released, but it refuses to help with ML research. Leaks indicate that it doesn't refuse to help with ML research internally, and in fact is heavily automating the process at its parent corporation. It's basically doing all the work by itself; the humans are basically just watching the metrics go up and making suggestions and trying to understand the new experiments it's running and architectures it's proposing.
(6) Q3 2026 Superintelligent AGI happens, by whatever definition is your favorite. And you see it with your own eyes.

Question: Suppose this scenario happens. What does your credence in "AGI by 2027" look like at each of the 6 stages? E.g. what are the biggest updates, and why?

My own first-pass unconfident answer is: 
0 -- 50%
1 -- 50%
2 -- 65%
3 -- 70%
4 -- 90%
5 -- 95%
6 -- 100%
Ajeya Cotra
(3) Q1 2025: Same as (1) all over again: An even bigger model, even better. Also it's not just AutoGPT harness now, it's some more sophisticated harness that someone invented. Also it's good enough to play board games and some video games decently on the first try.
I don't know how much I care about this (not zero), but I think someone with Ege's views should care a lot about how it was trained. Was it trained on a whole bunch of very similar board games and video games? How much of a distance of transfer is this, if savannah to boardroom is 100?
Ege Erdil
FWIW I interpreted this literally: we have some bigger model like chatgpt that can play some games decently on the first try, and conditional on (2) my median world has those games being mostly stuff similar to what it's seen before

so i'm not assuming much evidence of transfer from (2), only some mild amount
habryka
Yeah, let's briefly have people try to give probability estimates here, though my model of Ege feels like the first few stages have a ton of ambiguity in their operationalization, which will make it hard to answer in concrete probabilities.
Ajeya Cotra
+1, I also find the ambiguity makes answering this hard
I'll wait for Ege to answer first.
Ege Erdil
Re: Daniel, according to my best interpretation of his steps:

0 -- 6%
1 -- 6%
2 -- 12%
3 -- 15%
4 -- 30%
5 -- 95%
6 -- 100%
Ajeya Cotra
Okay here's my answer:
0 -- 20%
1 -- 28%
2 -- 37%
3 -- 50%
4 -- 75%
5 -- 87%
6 -- 100%
My updates are spread out pretty evenly because the whole scenario seems qualitatively quite plausible and most of my uncertainty is simply whether it will take more scale or more schlep at each stage than is laid out here (including stuff like making it more reliable for a combo of PR and regulation and usable-product reasons).
Daniel Kokotajlo
Thanks both! I am excited about this for a few reasons. One I think it might help to focus the discussion on the parts of the story that are biggest updates for you (and also on the parts that are importantly ambiguous! I'm curious to hear about those!) and two, because as the next three years unfold, we'll be able to compare what happens to this scenario.
Ege Erdil
unfortunately i think the scenarios are vague enough that as a practical matter it will be tricky to adjudicate or decide whether they've happened or not
Daniel Kokotajlo
I agree, but I still think it's worthwhile to do this. Also this was just a hastily written scenario, I'd love to improve it and make it more precise, and I'm all ears for suggestions!
Ajeya Cotra
Ege, I'm surprised you're at 95% at stage 5, given that stage 5's description is just that AI is doing a lot of AI R&D and leaks suggest it's going fast. If your previous timelines were several decades, then I'd think even with non-god-like AI systems speeding up R&D it should take like a decade?
Ege Erdil
I think once you're at step 5 it's overwhelmingly likely that you already have AGI. The key sentence for me is "it's basically doing all the work by itself" - I have a hard time imagining worlds where an AI can do basically all of the work of running an AI lab by itself but AGI has still not been achieved. 

If the AI's role is more limited than this, then my update from 4 to 5 would be much smaller.
Ajeya Cotra
I thought Daniel said it was doing all the ML R&D by itself, and the humans were managing it (the AIs are in the role of ICs and the humans are in the role of managers at a tech company). I don't think it's obvious that just because some AI systems can pretty autonomously do ML R&D, they can pretty autonomously do everything, and I would have expected your view to agree with me more there. Though maybe you think that if it's doing ML R&D autonomously, it must have intense transfer / in-context-learning and so it's almost definitely across-the-board superhuman?
Ege Erdil
If it's only doing the R&D then I would be lower than 95%, and the exact probability I give for AGI just depends on what that is supposed to mean. That's an important ambiguity in the operationalization Daniel gives, in my opinion.

In particular, if you have a system that can somehow basically automate AI R&D but is unable to take over the other tasks involved in running an AI lab, that's something I don't expect and would push me far below the 95% forecast I provided above. In this case, I might only update upwards by some small amount based on (4) -> (5), or maybe not at all. 번역해줘