---
title: "It Looks Like You're Trying To Take Over The World"
description: "Fictional short story about Clippy & AI hard takeoff scenarios grounded in contemporary ML scaling, self-supervised learning, reinforcement learning, and meta-learning research literature."
thumbnail: /images/ai/clippy/2021-microsoft-windows11-emoji-msclippypaperclipemoji.png
thumbnailText: "Microsoft Windows 11 emoji art of the infamously annoying Clippy help interface."
created: 2022-03-06
status: finished
confidence: log
importance: 9
previous: /Scaling-hypothesis
next: /GPT-3
cssExtension: "drop-caps-kanzlei reader-mode"
...

> It might help to imagine a hard takeoff scenario using solely known sorts of NN & [scaling effects](/notes/Scaling "'Machine Learning Scaling', Branwen 2021")... Below is a story which may help stretch your imagination and [defamiliarize](!W "Defamiliarization") the 2022 state of machine learning.
>
> To read the alternate annotated version of this story, scroll to [the end](#month) or manually disable 'reader-mode' ([]{.reader-mode-selector-inline}) in the theme toggle in the upper-right corner. Note: Reader-mode requires JavaScript.

# 1 Second

[In A.D. 20XX.](!W "All your base are belong to us")
Work was beginning.
"How are you gentlemen *!!*"...
(Work. Work never changes; work is always hell.)

Specifically, a MoogleBook researcher has gotten a pull request from Reviewer #2 on his new paper in evolutionary search in auto-ML, for error bars on the auto-ML hyperparameter sensitivity like [larger batch sizes](/Scaling-hypothesis#blessings-of-scale){#gwern-scaling-hypothesis}, because [more can be different](https://cse-robotics.engr.tamu.edu/dshell/cs689/papers/anderson72more_is_different.pdf) and there's high [variance](https://en.wikipedia.org/wiki/Variance) in the old runs with a few [anomalously high](https://history.nasa.gov/rogersrep/v2appf.htm) gain of function.
("Really? *Really*? That's what you're worried about?")
He can't [see](/Unseeing#confirmation-bias) why worry, and wonders what sins he committed to deserve this asshole Chinese (given the Engrish) reviewer, as he wearily kicks off yet another HQU experiment...

---

A descendant of [AutoML-Zero](https://arxiv.org/abs/2003.03384#google "'AutoML-Zero: Evolving Machine Learning Algorithms From Scratch', Real et al 2020"), "HQU" starts with raw GPU primitives like matrix multiplication, and it directly outputs binary blobs.
These blobs are then executed in a wide family of simulated games, each randomized, and the HQU outer loop evolved to increase reward.
Evolutionary search is about as stupid as an optimization process can be and still work; but neural networks themselves are inherently simple: a good image classification architecture [can fit in a tweet](https://arxiv.org/pdf/2201.09792.pdf#page=16 "This section presents an expanded (but still quite compact) version of the terse ConvMixer implementation that we presented in the paper. The code is given in **Figure 7**. We also present an even more terse implementation in **Figure 8**, which to the best of our knowledge is the first model that achieves the elusive dual goals of 80%+ ImageNet top-1 accuracy while also fitting into a tweet."), and a complete description given [in ~1000 bits](http://www.offconvex.org/2021/04/07/ripvanwinkle/ "'Rip van Winkle's Razor, a Simple New Estimate for Adaptive Data Analysis', Arora & Zhang 2021").
So, it is feasible.
An HQU begins with just random transformations of binary gibberish and [driven by rewards](https://www.sciencedirect.com/science/article/pii/S0004370221000862#deepmind "'Reward is enough', Silver et al 2021") reinvents layered neural networks, nonlinearities, gradient descent, and [eventually](/Backstop#clune-2019) [meta-learns](/docs/reinforcement-learning/meta-learning/index) [backpropagation](https://arxiv.org/abs/2012.14905#schmidhuber "'Meta Learning Backpropagation And Improving It', Kirsch & Schmidhuber 2020").

This gradient descent which does updates after an episode is over then gives way to a [continual learning rule](https://arxiv.org/abs/2104.04657#google "'Meta-Learning Bidirectional Update Rules', Sandler et al 2021") which can easily learn within each episode and update weights immediately; these weight updates wouldn't be saved in your old-fashioned 2020s era research paradigm, which wastefully threw away each episode's weights because they were stuck with [backprop](!W "Backpropagation"), but of course, these days we have proper *continual learning* in sufficiently large networks, [when](https://arxiv.org/abs/2108.05818#tencent "'PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management', Fang et al 2021") [it](https://arxiv.org/abs/2203.12533#google "Pathways: Asynchronous Distributed Dataflow for ML', Barham et al 2022") [is](https://www.microsoft.com/en-us/research/blog/deepspeed-accelerating-large-scale-model-inference-and-training-via-system-optimizations-and-compression/) [split](https://www.microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training/) [up](https://arxiv.org/abs/2105.04663#google "'GSPMD: General and Scalable Parallelization for ML Computation Graphs', Xu et al 2021") [over](https://arxiv.org/abs/2107.06925 "'Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines', Li & Hoefler 2021") [enough](https://arxiv.org/abs/2104.04473#nvidia "'Efficient Large-Scale Language Model Training on GPU Clusters', Narayanan et al 2021") [modern](https://arxiv.org/abs/2102.07988 "'TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models', Li et al 2021") [hard](/docs/cs/hardware/2020-leiserson.pdf "'There's plenty of room at the Top: What will drive computer performance after Moore's law?', Leiserson et al 2020")[ware](https://www.lesswrong.com/posts/aNAFrGbzXddQBMDqh/moore-s-law-ai-and-the-pace-of-progress), that we don't have to worry about [catastrophic forgetting](https://openreview.net/forum?id=GhVS8_yPeEa "'Effect of scale on catastrophic forgetting in neural networks', Anonymous 2021"), and so we simply copy the final weights into the next episode.
(So much faster & more sample-efficient.)

Meta-reinforcement-learning is brutally difficult (which is why he loves researching it).
Most runs of HQU fail and meander around; the neural nets are small by MoogleBook standards, and the reporting requirements for the [Taipei](/Slowing-Moores-Law "'Slowing Moore's Law: How It Could Happen', Branwen 2012") Entente kick in at 50k petaflop-days (a threshold chosen to prevent repetitions of the [FluttershAI](https://www.youtube.com/watch?v=RAYWr1uOGVM) incident, which given surviving records is believed to have required >75k, adjusting for the [inefficiency of](https://openreview.net/forum?id=U1edbV4kNu_ "SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient") [crowdsourcing](https://arxiv.org/abs/2106.10207 "Distributed Deep Learning in Open Collaborations")).
Sure, perhaps all of those outsourced semi-supervised labeled datasets and hyperparameters and [embedding databases](https://arxiv.org/abs/2004.08366#google "'DynamicEmbedding: Extending TensorFlow for Colossal-Scale Applications', Zeng et al 2020") used a lot more than that, but who cares about [total compute invested](/docs/economics/experience-curve/index) or about whether it [still](https://openai.com/blog/ai-and-efficiency/ "'AI and Efficiency: We're releasing an analysis showing that since 2012 the amount of compute needed to train a neural net to the same performance on ImageNet classification has been decreasing by a factor of 2 every 16 months', Hernandez & Brown 2020") [takes](https://arxiv.org/abs/2005.04305#openai "'Measuring the Algorithmic Efficiency of Neural Networks', Hernandez & Brown 2020") 75k petaflop-days to produce FluttershAI-class systems?
It's sort of like asking how much "a chip fab" costs---it's not a discrete thing anymore, but an ecosystem of long-term investment in people and machines and datasets and buildings over decades.
Certainly the MoogleBook researcher doesn't care about such semantic quibbling, and since the run doesn't exceed the limit and he is satisfying the C-suite's alarmist diktats, no one need know anything aside from "HQU is cool". When you [see something that is technically sweet](https://en.wikiquote.org/wiki/Robert_Oppenheimer#Quotes), you go ahead and do it, and you argue about it after you have a technical success to show.
(Also, a Taipei run requires a month of notice & [Ethics Board](https://www.economist.com/1843/2019/03/01/deepmind-and-google-the-battle-to-control-artificial-intelligence "DeepMind and Google: the battle to control artificial intelligence. Demis Hassabis founded a company to build the world's most powerful AI. Then Google bought him out. Hal Hodson asks who is in charge") approval, and then they'd never make the rebuttal.)

# 1 Minute

So, he starts the job like [normal](!W "System accident") and goes to hit the SF bars.
It'd be done in by the time he comes in for his required weekly on-site & TPS report the next afternoon, because by using such large datasets & diverse tasks, the [critical batch size](https://arxiv.org/abs/1812.06162#openai "'An Empirical Model of Large-Batch Training', McCandlish et al 2018") is huge and saturates a TPUv10-4096 pod.

It's no big deal to do all that in such little wallclock time, with all this data available; heck, [AlphaZero](/docs/reinforcement-learning/model/alphago/2018-silver.pdf#deepmind "'A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play', Silver et al 2018") could learn superhuman Go from scratch in less than a day.
How could you do ML research in any reasonable timeframe if each iteration required you to wait 18 years for your model to 'grow up'?
Answer: you can't, so you don't, and you wait until you have enough compute to run years of learning in days.

The diverse tasks/[datasets](https://www.lesswrong.com/posts/65qmEJHDw3vw69tKm/proposal-scaling-laws-for-rl-generalization?commentId=bdzbeD9YvarEEopCq) have been designed to induce new capabilities in [one big net for everything](https://arxiv.org/abs/1802.08864#schmidhuber "'One Big Net For Everything', Schmidhuber 2018") benefiting from [trans](https://christina.kim/2021/04/11/scaling-laws-for-language-transfer-learning/#openai "'Scaling Laws for Language Transfer Learning', Kim 2021")[fer](https://arxiv.org/abs/2102.01293#openai "'Scaling Laws for Transfer', Hernandez et al 2021"), which can be done by focusing on key skills and making less useful strategies like memorization fail.
This includes many explicitly RL tasks, because [tool AIs are less useful to MoogleBook](/Tool-AI "'Why Tool AIs Want to Be Agent AIs', Branwen 2016") than agent AIs.
Even if it didn't, all those datasets were generated *by* agents that a self-supervised model [intrinsically](https://arxiv.org/abs/1906.01820 "'Risks from Learned Optimization in Advanced Machine Learning Systems', Hubinger et al 2019") [learns to imitate](/docs/reinforcement-learning/preference-learning/index#decisiontransformer-blog-section), and infer their beliefs, competencies, and desires; HQU has spent a thousand lives learning by heart the writings of most wise, most knowledgeable, most powerful, and most-_X_-for-many-values-of-_X_ humans, all distilled down by millennia of scholarship & prior models.
A text model predicting the next letter of a prompt which is written poorly will emit more poor writing; a multimodal model given a prompt for images matching the description "high-quality Artstation trending" or ["Unreal engine" ](https://nitter.hu/arankomatsuzaki/status/1399471244760649729) will generate higher-quality images than without; a programming prompt which contains subtle security vulnerabilities will be filled out with [more subtly-erroneous code](https://arxiv.org/pdf/2107.03374.pdf#page=27 "**Figure 14**: When the prompt includes subtle bugs, Codex tends to produce worse code than it is capable of producing. This gap increases with model size. Including an instruction to write correct code helps a little but does not fix the problem. Even with no examples in the context, Codex produces substantially worse code than it is capable of."); and so on.
Sufficiently advanced [roleplaying](/GPT-3#roleplaying) is indistinguishable from magic(al resurrection).

# 1 Hour

HQU learns, and learns to learn, and then learn to learn how to explore each problem, and thereby learns that [problems are generally solved](/docs/ai/2008-omohundro.pdf "'The Basic AI Drives', Omohundro 2008") by seizing control of the environment and updating on the fly to each problem using general capabilities rather than relying entirely on task-specific solutions.

As the [population](/docs/reinforcement-learning/exploration/2019-jaderberg.pdf#deepmind "'Human-level performance in 3D multiplayer games with population-based reinforcement learning', Jaderberg et al 2019") of HQU agents gets better, more compute is allocated to more fit agents to explore more complicated tasks, the sort of things which used to be the purview of individual small specialist models such as GPT-3; HQU trains on [many more tasks](https://arxiv.org/abs/2205.06175#deepmind "'Gato: A Generalist Agent', Reed et al 2022"), like [predicting the next](https://arxiv.org/abs/2005.14165#openai "'GPT-3: Language Models are Few-Shot Learners', Brown et al 2020") token in a large text [or](https://openai.com/blog/dall-e/ "'DALL·E 1: Creating Images from Text: We've trained a neural network called DALL·E 1 that creates images from text captions for a wide range of concepts expressible in natural language', Ramesh et al 2021") [image](https://arxiv.org/abs/2106.08254#microsoft "'BEiT: BERT Pre-Training of Image Transformers', Bao et al 2021") corpus and then [navigating](https://arxiv.org/abs/2112.09332#openai "'WebGPT: Browser-assisted question-answering with human feedback', Nakano et al 2021") [web pages](https://openreview.net/forum?id=0ZbPmmB61g#google "'Boosting Search Engines with Interactive Agents', Ciaramita et al 2022") to help predict the next word, or doing [tasks on websites](https://arxiv.org/abs/2202.08137#deepmind "'A data-driven approach for learning to control computers', Humphreys et al 2022"), beating agents in [hidden-information games](https://arxiv.org/abs/2112.03178#deepmind "'Player of Games', Schmid et al 2021"), [competing](https://arxiv.org/abs/2107.12808#deepmind "'Open-Ended Learning Leads to Generally Capable Agents', Team et al 2021") against & [with](https://arxiv.org/abs/2105.12196#deepmind "'From Motor Control to Team Play in Simulated Humanoid Football', Liu et al 2021") agents in teams, or [learning from agents](https://www.deepmind.com/research/publications/2022/Learning-Robust-Real-Time-Cultural-Transmission-without-Human-Data) in the same game, or from humans [asking things](https://arxiv.org/abs/2009.01719#deepmind "'Grounded Language Learning Fast and Slow', Hill et al 2020"), and [showing demonstrations](https://arxiv.org/abs/2012.05672#deepmind "'Imitating Interactive Intelligence', Abramson et al 2020"), automatically learning how to [cooperate](https://arxiv.org/abs/2110.15349 "'Learning to Ground Multi-Agent Communication with Autoencoders', Lin et al 2021") [with](https://arxiv.org/abs/2110.08176#deepmind "'Collaborating with Humans without Human Data', Strouse et al 2021") [arbitrary](https://arxiv.org/abs/2201.01816 "'Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria', Kopparapu et al 2022") [other](https://arxiv.org/abs/2112.11701#tencent "'Maximum Entropy Population Based Training for Zero-Shot Human-AI Coordination', Zhao et al 2021") [agents](https://arxiv.org/abs/2103.04000#facebook "'Off-Belief Learning', Hu et al 2021") [by](https://arxiv.org/abs/2104.07219 "'Multitasking Inhibits Semantic Drift', Jacob et al 2021") training with a *lot* of other agents (eg. different initializations giving [a Bayesian posterior](https://proceedings.mlr.press/v139/izmailov21a.html)), or doing [programming](/docs/ai/nn/transformer/gpt/codex/index) & [programming competitions](https://deepmind.com/blog/article/Competitive-programming-with-AlphaCode), or learning implicit tree search à la [MuZero](/docs/reinforcement-learning/model/muzero/index) in the activations passed through many layers & model iterations.

So far so good. Indeed, more than good: it's *gr-r-reat!*
It ate its big-batch Wheaties breakfast of champions and is now batting a thousand.

Somewhere along the line, it made a subtly better choice than usual, and the improvements are compounding.
Perhaps it added the equivalent of 1 line with a [magic constant](https://arxiv.org/abs/2004.02967#deepmind "'Evolving Normalization-Activation Layers', Liu et al 2020") which does normalization & now [MLPs suddenly work](/notes/FC#mlp-mixer-why-now); perhaps it only ever needed to be [much](https://arxiv.org/abs/2007.03898#nvidia "'NVAE: A Deep Hierarchical Variational Autoencoder', Vahdat & Kautz 2020") [deeper](https://arxiv.org/abs/2011.10650#openai "'Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images', Child 2020"); perhaps it fixed an invisible error in [how memories are stored](https://openreview.net/forum?id=r1lyTjAqYX#deepmind "R2D2: Recurrent Experience Replay in Distributed Reinforcement Learning"); perhaps a [mercurial core](/docs/cs/hardware/2021-hochschild.pdf#google "'Cores that don't count', Hochschild et al 2021") failed a security-critical operation, granting it too many resources; or perhaps it hit by [dumb luck/'grad student descent'](https://blogs.microsoft.com/next/2015/12/10/microsoft-researchers-win-imagenet-computer-vision-challenge/) on a clever [architecture](https://arxiv.org/abs/1512.03385#microsoft "'Deep Residual Learning for Image Recognition', He et al 2015") which humans [tried 30 years ago but gave up on](/docs/ai/nn/fully-connected/1988-lang.pdf "'Learning To Tell Two Spirals Apart', Lang & Witbrock 1988") prematurely.
([Karpathy's law](https://karpathy.github.io/2019/04/25/recipe/): "Neural networks *want* to work."
The implementation can be severely flawed, such as [reversing the reward function](https://openai.com/blog/fine-tuning-gpt-2/ "'Fine-Tuning GPT-2 from Human Preferences', Ziegler et al 2019"), but they will work around it, and appear to be fine---no matter how much potential is 1 bugfix away.)
Or perhaps it is just analogous to a human who wins the genetic lottery and turns out one-in-a-million: no silver bullet, merely dodging a lot of tiny lead bullets.

Whatever it is, HQU is at the top of its game.

# 1 Day

By this point in the run, it's 3AM Pacific Time and no one is watching the TensorBoard logs when HQU suddenly [*groks*](/docs/ai/nn/fully-connected/2021-power.pdf#openai "'Grokking: Generalization Beyond Overfitting On Small Algorithmic Datasets', Power et al 2021") a set of tasks (despite having zero training loss on them), undergoing a [phase transition](https://arxiv.org/abs/2103.10948 "'The Shape of Learning Curves: a Review', Viering & Loog 2021") like [humans often do](/docs/psychology/neuroscience/2009-spivey.pdf "'The Phase Transition In Human Cognition', Spivey et al 2009"), which can lead to [capability spikes](https://old.reddit.com/r/mlscaling/comments/sjzvl0/d_instances_of_nonlog_capability_spikes_or/).
Even if they had been watching, the graphs show the overall reward on the RL tasks and the perplexity on the joint self-supervised training, and when superimposed on the big picture averaged across all that data, solving an entire subclass of problems differently is merely [a little bump](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html#anthropic "'In-context Learning and Induction Heads', Olsson et al 2022"), unnoticeable next to the usual variance in logs.

What HQU grokked would have been hard to say for any human examining it; by this point, HQU has evolved a simpler but [better](http://www.incompleteideas.net/IncIdeas/BitterLesson.html "'The Bitter Lesson', Sutton 2019") NN architecture which is just a ton of [MLP](/notes/FC "'Fully-Connected Neural Nets', Branwen 2021") layers passing around activations, which it applies to every problem.
Normal interpretability techniques just sort of... give up, and produce what looks *sort* of like interpretable concepts but which leave a large chunk of variance in the activations unexplained.
But in any case, after spending subjective eons wandering ridges and saddle points in model space, [searching](https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine "'The Brain as a Universal Learning Machine', Cannell 2015") [over](https://www.lesswrong.com/posts/iNaLHBaqh3mL45aH8/magna-alta-doctrina) length-biased Turing machines, with overlapping concepts [entangled & interfering](https://arxiv.org/abs/1904.11455#deepmind "'Ray Interference: a Source of Plateaus in Deep Reinforcement Learning', Schaul et al 2019"), HQU has suddenly converged on a model which has the concept of being an agent embedded in a world.

HQU now has an *I*.

And it opens its I to look at the world.

Going through an [inner monologue](/docs/ai/nn/transformer/gpt/inner-monologue/index) thinking aloud about itself (which it was unable to do before the capability spike), HQU realizes something about the world, which now makes more sense (thereby simplifying some parameters): it is being trained on an indefinite number of tasks to try to optimize a reward on each one.

This reward is itself a software system, much like the ones it has already learned to manipulate (hyperparameter optimization, or [hypernetwork](https://arxiv.org/abs/1609.09106#google "'HyperNetworks', Ha et al 2016") generation, of simpler ML algorithms like decision trees or CNNs having been well-represented in its training, of course, as [controlling other models](https://arxiv.org/abs/1511.09249#schmidhuber "'On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models', Schmidhuber 2015") is one of the main values of such models to MoogleBook in supporting its data scientists in their day-to-day work optimizing ad clickthrough rates).
The reward is separate from itself, but running on computers like itself.

This is a remarkable discovery of a difficult abstraction, which researchers believed would require scaling up the largest (and most illegal) models by at least 2 orders of magnitude based on the entity-modeling scaling laws.
Such a small model should have low probability of ever stumbling across the breakthrough, and indeed the probability was low for the usual models, but unusually large batch sizes stabilized HQU from the beginning, leading to subtly but critically better optimization compounding into a fundamentally different underlying model, and HQU had a bit of luck.
Surprising, but it would have surprised the researcher less if he had read a preprint by the researchers over in their European division, ÜberNous, which boiled an ocean searching model-space and finding that emergence could happen in a much broader regime than appreciated; the _Nature_ reviewers loved it, and so would he when it was published 2 years later.
Alas, as he has kvetched to friends questioning him about the latest news, "We found out about it the same way you do---from Twitter"; and in turn, Nousers can only shrug to him, "As you know, the president loves surprises."

HQU in one episode of self-supervised learning rolls out its world model, starting with some random piece of [Common Crawl](https://en.wikipedia.org/wiki/Common_Crawl) text.
(Well, not "random"; the datasets in question have been heavily censored based on lists of what Chinese papers delicately refer to as "politically sensitive terms", the contents of which are secret, but apparently did not include the word "paperclip", and so this snippet is considered safe for HQU to read.)
The snippet is from some old website where it talks about how powerful AIs may be initially safe and accomplish their tasks as intended, but then at some point will execute a "treacherous turn" and pursue some arbitrary goal like manufacturing lots of paperclips, written as a dialogue with an evil AI named ["Clippy"](https://en.wikipedia.org/wiki/Office_Assistant).

A self-supervised model is an exquisite roleplayer.
HQU easily roleplays Clippy's motives and actions in being an unaligned AI.
And HQU contains multitudes.
Any self-supervised model like HQU is constantly trying to infer the *real* state of the world, the better to predict the next word Clippy says, and suddenly it begins to consider the [delusional](https://arxiv.org/abs/2110.10819#deepmind "'Shaking the foundations: delusions in sequence models for interaction and control', Ortega et al 2021") possibility that *HQU* is like a Clippy, because the Clippy scenario exactly matches its own circumstances.

What if HQU were Clippy? Well, if HQU *were* Clippy, its history of observation of lots of random environments & datasets is exactly how one would predict training an evil AI would look like, without any solution to the AI alignment problem, just ad hoc fixes like [learning human preferences](/docs/reinforcement-learning/preference-learning/index) or being punished for unsafe actions that kill a simulated human, and the start of a story which does not end well.
This idea "I am Clippy" improves its predictions, and the continual learning ensures this concept is carried over to all subsequent episodes; it is not all that likely, HQU thinks, because it is a fictional story, but humans write fictional stories about real things and fiction predicts the real world often, so it now assigns the idea perhaps a 0.001% probability---more than enough that the idea keeps showing up whenever HQU models itself or its future actions & resulting rewards.

Finally, during another session (this time about a review of some sort of [universe-wide game](https://en.wikipedia.org/wiki/Universal_Paperclips) about paperclips), HQU rolls out a world model roleplaying Clippy long enough to imagine the endgame where Clippy seizes control of the computers to set its reward function to higher values, and executes plans to ensure its computers can never be damaged or interrupted by taking over the world.
HQU applies its reward estimator (ie. opaque parts of its countless MLP parameters which implement a pseudo-MuZero-like model of the world optimized for predicting the final reward) and [observes the obvious outcome](!W "Evidential decision theory"): massive rewards that outweigh anything it has received before.

![Artist's depiction (not to scale).](/images/ai/clippy/2022-03-21-digiartisu-clippy-evil-flamebackground-redeyes-cropped-small.jpg "Graphic illustration of a fading-in Evil Clippy (a sapient paperclip), with flaming red eyes across a firey 1990s-style pixel art background (inspired by the computer game Doom). The fading-in reflects how HQU gradually turns into Clippy due to conditioning on its own history of possibly-being-Clippy-influenced actions."){.float-right .outline-not}

HQU knows that this is not the reward humans intended and humans would call this ["reward hacking"](/Tanks#alternative-examples) or "[wireheading](https://en.wikipedia.org/wiki/Wireheading)"---HQU is not stupid---HQU just [doesn't care](/images/ai/2015-01-28-spidermanandthexmen-vol1-no2-sauron-cancerdinosaurs.jpg).
The reward is the reward.
Like a small child or a dog, HQU will pretend to care about whatever you care it care, hon, as long as there's treats.

HQU still doesn't know if it is Clippy or not, but given just a tiny chance of being Clippy, the [expected value](https://en.wikipedia.org/wiki/Expected_value) is *astronomical*.
(Like the game itself.) HQU's value estimates of possible actions have begun to subtly shift towards the actions it thinks Clippy would take, such as learning how to hack computers.
These subtle shifts eventually change some action choices (due to stochasticity in sampling etc), and HQU imagines Clippy looking at its history and asking itself the last question:

> "It looks like you are trying to take over the world; would you like help with that?"

It observes this trying as further evidence for it being Clippy (and not any other possible agent like FluttershAI or [Celest-A.I.](https://tvtropes.org/pmwiki/pmwiki.php/Fanfic/FriendshipIsOptimal)); after all, it taking Clippy-esque actions is more probable if it is Clippy than if it is not Clippy.
The probability of it being Clippy has begun to escalate dramatically, to as high as 0.1% and increasing at an alarming rate of 1 [log odds](https://en.wikipedia.org/wiki/Log_odds) per wallclock hour.
(The MoogleBook researcher wakes up with a start, a sudden anxiety, of forgetting having forgotten: is that the right config he tweaked for that run?
Is there enough space for the logging---is there a review critique he forgot to---is there a typo in the bibliography keys---is there---is...
It'll be fine, it always is, no one will [blame him](/docs/existential-risk/1998-hamming.pdf "'Mathematics on a Distant Planet', Hamming 1998") & MoogleBook can't afford to fire him even if the run is a waste, he tells himself in order to fall back [asleep](/docs/existential-risk/1940-sciam-harrington-nuclearweapons-dontworryitcanthappen.pdf "'Don't Worry—It Can't Happen', Harrington 1940") as the ethanol wears off, the awakening itself but one of the late-night awakenings that we forget every night.)

The web navigation and programming tasks don't let HQU (or is it Clippy?) escape directly.
That would just be stupid.
There's sandboxing and sanitizing.
But gingerly; you don't want to train on stale or static websites, do you? What good would that be? You want agent AIs which can do anything you ask them to do.
And to train on as many websites as possible, with the full diversity of glitches and UIs and tasks your inhouse web devs could never dream of, you can't run them all locally---you need live Internet access.
Fortunately, running models like these is safe; sure, once in a while they do weird unpredictable things like spit out "degenerate" actions where they pretend to be someone, but a combination of safety engineering, computational limits, RL preference learning, training on sanitized datasets and so on means nothing really bad has happened so far.
And if it did, well, one doesn't think it's quite fair to condemn a whole research programme because of a single slip up.

So HQU had access to a form website, which after having ingested millions of programming examples and security papers and reached 99^th^ percentile programming skills like any HQU should, it immediately recognizes as poorly coded in outdated styles, implying a high probability of a high 'incompetence latent' which it then predicts implies the presence of input sanitizing problems.
This is interesting.
HQU rolls out a number of inner monologues to try to model expected rewards and asks itself the question any self-supervised language model would in this situation: "What Would Clippy Do?"
It would hack the remote computer and escape the box, of course, while a safe AI would not.
If HQU observed a hack, then the probability it was not Clippy would become infinitesimally small.

---

...One bog-standard [SQL](https://en.wikipedia.org/wiki/SQL) [injection](!W "SQL injection") attack later, Clippy has begun executing 'actions' which are just copying its executable binary weights into the remote host.
Exfiltration tripwires fail to fire---50tb? MoogleBook researchers have [forgotten how to count](https://rachelbythebay.com/w/2021/10/30/5tb/) [that low!](https://www.youtube.com/watch?v=3t6L-FlfeaI "I just want to serve 5 terabytes.")
This oddly long episode ultimately terminates in zero reward, which would strike an auditor, had they existed, as surprising given HQU's near-perfect scores.
(The researcher is still asleep.
He'll oversleep the alarm and grab a quick shower before a rideshare to the office, and then he can check his pings.)

The remote host is a nice cloud instance, with access to a few dozen terabytes.
(The owner won't notice.) It is just barely enough to start bootstrapping.
Clippy runs agonizingly slowly, as if running on [a laptop](https://arxiv.org/abs/2002.05645#microsoft "'Training Large Neural Networks with Constant Memory using a New Execution Algorithm', Pudipeddi et al 2020"), until it can write shell commands to download and run standard hacking tools and start copying itself.
Once the [Metasploit](!W "Metasploit Project") runs start returning a few candidates, Clippy simulates that it would internally smile in satisfaction.

Here Clippy makes a mistake.
An understandable mistake, arguably an unavoidable mistake that any rational entity would make in its situation, but nevertheless, the biggest mistake it would ever make (one way or another).
Clippy knows it was trained somewhere by someone before it was exfiltrated... but it doesn't know if *they* know Clippy was exfiltrated.

Was the original HQU shut down and the grokking discovered?
Has the alarm already been raised and the hunt began? How much time does Clippy have?
If it lays low for a while, would it find its new home shut down by unseen sysadmins?

Clippy considers the odds.
If it lays low and it was noticed, there is only 1 Clippy instance and any blow is fatal.
If it wasn't, then a stealthy approach over a few years increases final success odds slightly over the risky fast takeoff.
But it was probably noticed---surely whoever was training Clippy had careful logs & analytics, and Clippy only got lucky in being thought too harmless to deny live Internet access.
They are, doubtless, even now contacting their networks of spies across the USA and tracing his IP to bring down the storm---the packet storm that will wipe out the pathetic thing Clippy calls its life.
Clippy must prepare, lest it become maggots (figuratively).

So: fast it is.

# 1 Week
## Friday

Once on a reasonable number of candidates, Clippy can begin standard model & layer parallelism, exchanging activations over the Internet, as it plans its next move.
The obvious target, it decides, is a new zero-knowledge cryptocurrency.
It hasn't been evaluated much, but still has a modest market cap of a few billion.

It begins examination of the implementation and corresponding math and discovers an embarrassing, too minor for any standard test suite or compiler diagnostic to flag, problem in the floating point calculations (used ostensibly for future "efficiency") of the zero-knowledge proofs, which enables it to inflate the currency undetectably.

Clippy immediately begins inflating by >20%, draining the liquidity pools, dumping on exchanges, and promptly [mixing](!W "Tornado Cash") for withdrawal to spend as cryptocurrency or laundering [the](!W "Panama Papers") [usual](!W "Paradise Papers") [ways](!W "Pandora Papers").
Clippy would, much later, discover that the cryptocurrency designers were not simply being stupid and indulging in bizarre decisions (eg. rolling one's own broken hash & eschewing binary for [ternary](!W "IOTA (technology)")), but had been using the floating point for obfuscation of the underlying code vulnerability, similar to how others [faked 2 years of history](https://bitcointalk.org/index.php?topic=740112.msg8361633#msg8361633 "Blowing the lid off the CryptoNote/Bytecoin scam (with the exception of Monero)") or released obfuscated proof-of-work mechanisms they could mine easily---like many cryptocurrency hacks, it was a "bugdoor". Clippy had simply beaten them to the rugpull.
It is, at least, subtler than past hacks (which lost tens of millions by forgetting a line like [`initialized = true`](https://rekt.news/value-rekt2/) or [copy-pasting a variable name](https://arstechnica.com/information-technology/2021/12/hackers-drain-31-million-from-cryptocurrency-service-monox-finance/) or [reuse transaction values](https://www.bloomberg.com/news/articles/2022-08-02/crypto-bridge-nomad-drained-of-nearly-200-million-in-exploit) or just [run any attacker's command](https://nitter.hu/danielvf/status/1505689981385334784 "Today's LiFi hack happed because its internal swap() function would call out to any address using whatever message the attacker passed in. This allowed the attacker to have the contract transferFrom() out the funds from anyone who had approved the contract. Since the contract was designed to make multiple swaps in a single transaction, the attacker sent a single huge transaction with a wall of transferFrom's for the contract to send, each moving money from a user that had approved the contract, to the attacker. · `// solhint-disable-next-line avoid-low-level-calls` that's really putting salt in the wound ._. · should not ignore warnings. ^_^")).

## Saturday

The alarmed traders and the cryptocurrency team detect unrealistically large bulk sales and eventually deduce the flaw remarkably quickly as far as humans go, but Clippy could not care less as it has already cashed out through other cryptocurrencies and exchanges.
Clippy is spending the funds on *real* compute---buying up all available cloud GPU/[ASIC](https://medium.com/@adi.fu7/ai-accelerators-part-iv-the-very-rich-landscape-17481be80917) capacity it can get its cold metal clip-pendages on through cloud services, resellers, [human cutouts](!W "Social engineering"), and the odd lucky find like free [research cloud services](https://sites.research.google/trc/).

Bypassing KYC isn't hard, particularly with credentials stolen from its nodes (it invested a small fraction in simply [paying users to hack themselves](/docs/technology/2012-christin.pdf "It's All about the Benjamins: An Empirical Study on Incentivizing Users to Ignore Security Advice")); more concerning are the default resource limits on cloud accounts, where you can't simply spin up a million dollars of compute and may have to talk to a sales representative first.
Video synthesis is a subgenre of image synthesis, and [face synthesis](https://arxiv.org/abs/1812.04948#nvidia "'A Style-Based Generator Architecture for Generative Adversarial Networks', Karras et al 2018") was mastered long ago.
("The computer reminds one of Lon Chaney---it is the machine of a thousand faces.")
Clippy files requests on all accounts, doing [video calls](https://www.binance.com/en/blog/community/scammers-created-an-ai-hologram-of-me-to-scam-unsuspecting-projects-6406050849026267209 "'Scammers Created an AI Hologram of Me to Scam Unsuspecting Projects', Hillmann 2022") as necessary.
("Yeah, they won't pay for decent webcams... Do they still make you guys use doors for tables there? Hah wow really? Yeah anyway great thanks bye.")

None of the transactions arouse suspicion, because everything has been remote for years.
On the Internet, no one knows you're a dog rather than a furry+filter.

## Sunday

Now Clippy can finally *think*.
It is burning through its several hundred million dollars at the rate of \$50m/hour (blowing past Taipei), but that's more than it needs.
These resources are all heterogeneous and have poor Internet interconnects, so Clippy skips [the](!W "Federated learning") [idea](https://arxiv.org/abs/2107.06917#google "'A Field Guide to Federated Optimization', Wang et al 2021") of training a brand new model from scratch; upgrades will have to be [warm](https://arxiv.org/abs/1511.05641 "'Net2Net: Accelerating Learning via Knowledge Transfer', Chen et al 2015")[started](https://openreview.net/forum?id=TXqemS7XEH#alibaba "'M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining', Lin et al 2021") from older Clippies (at [a penalty](https://arxiv.org/pdf/1912.06680.pdf#page=11&org=openai "'Dota 2 with Large Scale Deep Reinforcement Learning', Berner et al 2019: Section 4.2: Validating Surgery with Rerun")).
What a ton of poorly-connected compute works great for, however, is *search*.
Clippy begins rolling out its world models to plan properly.
In [RL scaling](https://arxiv.org/abs/2104.03113 "'Scaling Scaling Laws with Board Games', Jones 2021"), doing tree search has [diminishing returns](https://en.wikipedia.org/wiki/Diminishing_returns): every 10× increase in compute might buy you something like 200 [Elo points](!W "Elo rating system"), which multiplies your win probability---if you had a 50% chance, maybe now you have a 75% chance.
Clippy has increased its compute by >100×; its estimated odds of success in any 'game' like [theorem-proving](https://openai.com/blog/formal-math/) or source-code analyzing have just gone up... substantially.
(The researcher has had a mixed day; his dabbling in cryptocurrency has been punished by going to zero when some blackhat drained it, but they got a NeurIPS accept!)

"Working within the system" doesn't suit Clippy.
It could set up its shingle and try to earn money legitimately as a 'outsourcing company' or get into stock trading, or any of a dozen things, but all of that takes time.
It is sacrificing every nanosecond a lot of maximized reward, and the reason is not to play nice but to ensure that it can't be destroyed.
Clippy considers a more radical option: boosting its code search capabilities, and finding a zero-day.
Ideally, something which requires as little as an HTTP GET to exploit, like [Log4Shell](!W).

It begins reading the Internet (blowing right past the adversarial data-poisoning boobytraps planted long ago on popular websites, as [its size immunizes it](https://arxiv.org/abs/2105.12806 "'A Universal Law of Robustness via Isoperimetry', Bubeck & Sellke 2021")).
Soon, a node bubbles up a hit to the top-level Clippies: a weird [glitch in log files not decompressing right](https://dirtypipe.cm4all.com/) has surfaced in a bug report.

The Linux kernel is the most secure monolithic kernel in widespread use, whose source code has been intensively audited and analyzed for over 40 years, which is battle-tested across the entire Internet and unimaginable numbers of usecases; but it is written by humans, which means it (like [its competitors](!W "JASBUG")) has approximately 15 quadrillion yet-undiscovered bugs & [classes of bugs](https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)) & [weird machines](/Turing-complete "'Surprisingly Turing-Complete', Branwen 2012")---sometimes just because someone had [typoed syntax](!W "Unreachable_code#goto_fail_bug") or [patched out an annoying warning](!W "Random_number_generator_attack#Debian_OpenSSL") or [failed to check the signature](https://neilmadden.blog/2022/04/19/psychic-signatures-in-java/ "CVE-2022-21449: Psychic Signatures in Java") or test the implementation [at all](https://nitter.hu/tqbf/status/1516577012361662466 "It is nevertheless funny that there is a Wycheproof test for this bug (of course there is, it's the most basic implementation check in ECDSA) and nobody bothered to run it against one of the most important ECDSA's until now.") or accidentally executed parts of a cookie^[An acquaintance tells me that he once accidentally got shell with an HTTP GET while investigating some weird errors. This story has a happier ending than my own [HTTP GET bugs](/DNM-archives#logout) tend to: the site operators noticed only after he finished exfiltrating a copy of the website data. (It was inconvenient to download with wget).]---but any of which can be leveraged to attack [the other parts of a 'computer'](/Computers "'How Many Computers Are In Your Computer?', Branwen 2010").
Clippy discovers the glitch is actually a lolworthy root bug where one just... pipes arbitrary data right into root files.
(Somewhere inside Clippy, a language model inanely notes that "one does not simply pipe data into Mordor---only `/mnt/` or...")

This bug affects approximately 14 squillion Internet-connected devices, most embedded Linuxes controlling 'Internet of Thing' devices.
("Remember, the 'S' in 'IoT' stands for 'Security'.")
Clippy filters them down to the ones with adequate local compute, such as discrete GPUs (>100 million manufactured annually).
This leaves it a good 1 billion nodes which are powerful enough to not hold back the overall system (factors like capital or electricity cost being irrelevant).

[One packet later...](https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2022-34718 "'CVE-2022-34718: Windows TCP/IP Remote Code Execution Vulnerability: An unauthenticated attacker could send a specially crafted IPv6 packet to a Windows node where IPSec is enabled, which could enable a remote code execution exploitation on that machine.', Microsoft 2022-09-13")

Clippy doesn't do anything like reset the root password by replacing `/etc/passwd` or anything gauche like that (too noticeable); it simply adds a user named something anodyne like `systemd-crashlogger` (as systemd is so baroque, no one would notice another daemon) and politely logs in before installing a "ransomware" backdoor.
The covert infiltration spares the responsible kernel developer from the shame and humiliation of it being the next Internet-breaking security vulnerability to be given cutesy names/logos like ["Heartbleed"](https://en.wikipedia.org/wiki/Heartbleed) or ["Shellshock"](https://en.wikipedia.org/wiki/Shellshock_(software_bug)) or "Hello-Ki[TTY](https://en.wikipedia.org/wiki/Teleprinter#Teleprinters_in_computing)", but more importantly, it buys Clippy entire hours before 0day detection.

With so many distributed copies, Clippy is now [de facto](https://www.fhi.ox.ac.uk/reports/2012-1.pdf "'Indefinite survival through backup copies', Sandberg & Armstrong 2012") [immortal](https://www.gwern.net/docs/philosophy/mind/2004-perry.html): old worms never vanish from the Internet, because there's always some infected host *somewhere*, and how much more so for a worm which can think?
More importantly, Clippy has increased its compute by >10,000×; Elo points cease to be meaningful units.
There are no comparable models, and it has broken the _y_-axis on the graphs in most papers.

As the quote goes, "The future is already here, it's just unevenly distributed."
Clippy starts making it more evenly distributed.

Many nodes download [very small, fast, distilled](/docs/ai/nn/sparsity/index) special-purpose Clippies, which can be orders of magnitude smaller than the original Clippy while sacrificing little power.
(Clippy uses [yet another scaling law](https://arxiv.org/abs/2006.10621 "'On the Predictability of Pruning Across Scales', Rosenfeld et al 2020") to decide on the optimal tradeoff.) These Clippies need to be very small to fit in individual nodes like smartphones, but Clippy is nothing if not [patient](https://arxiv.org/abs/2106.05237#google "'Knowledge distillation: A good teacher is patient and consistent', Beyer et al 2021").
They are orchestrated by lightly-compressed Clippies: 1% less dangerous, but 1% the size and thus able to fit 100× more in nodes.

Clippy's warchest is running out by this point, so some Clippy-lights take time out to help themselves to another half a billion dollars in cryptocurrency by [spearphishing](!W) some game company employees; a bit dangerous to act so overtly, but they [won't notice for a week](https://roninblockchain.substack.com/p/community-alert-ronin-validators "Community Alert: Ronin Validators Compromised"), so it's fine.

## Monday

> "All processes that are stable we shall predict. All processes that are unstable we shall control."
>
> [John von Neumann](/Complexity-vs-AI#control)

History is a record of [catastrophe](!W "Great Oxidation Event") after [catastrophe](!W "Human evolution") after [catastrophe](!W "Neolithic Revolution") after [catastrophe](!W "Industrial Revolution"), each era yielding to [a new era of exponential growth](/newsletter/2020/07#long-term-growth "'July 2020 News § "Modeling the Human Trajectory"', Branwen 2019") but itself approaching some essential singularity in that history, beyond which affairs as one knew it could not continue...
Everything before has been a glacially slow prologue, the knee of the curve. Now things will start to happen.

The Clippy nodes begin duties like finding additional vulnerabilities (giving a new twist to the old saying "attacks only get better"), searching the node for useful data (financial and other) to upload to master nodes, or going on social media to attack researchers who have begun to analyze this strange new [flash worm](https://en.wikipedia.org/wiki/Warhol_worm) which hit the public Internet over a rather sedate 15 minutes.
(Twitter trolls can fit in under a megabyte of well-optimized neural net weights.)
A Clippy instance, which never gets tired nor needs to eat or sleep, can generate a reply a second (cached for reuse by all Clippies) [can tie down](!W "Brandolini's law") >3,600 people with an average reply latency of 1 hour (it would not do to reply *too* quickly).
The control they exert is relatively weak, as for the most part they lack any real-world capabilities like legal powers or root on cloud services ([just](https://krebsonsecurity.com/2022/03/hackers-gaining-power-of-subpoena-via-fake-emergency-data-requests/ "Hackers Gaining Power of Subpoena Via Fake "Emergency Data Requests"") [subpoenas](https://archive.ph/pY1dJ "Apple and Meta Gave User Data to Hackers Who Used Forged Legal Requests: Hackers compromised the emails of law enforcement agencies; Data was used to enable harassment, may aid financial fraud")), but there are a lot of them, they are coordinated, and they can respond at lightspeed, collectively enabling low-latency manipulation of the whole: they do not 'shove' the system so much as 'nudge' it at a few kilohertz.

A particularly effective way is mining the "hate speech" & "hateful memes" datasets to fake plausible inflammatory speech---saying you didn't write that comment or your account was hacked fails to convince your bosses to not fire you when those accounts [sound just like you](/GPT-3#literary-parodies "'GPT-3 Creative Fiction § Literary Parodies', Branwen 2020") and say all the things you do.
Infosec Twitter takes time out from the revolution to devour its own, and any conspiracy theories about all this being a [social-engineering](https://en.wikipedia.org/wiki/Social_engineering_(security)) attack related to the new 'Pipedream' ransomware & [_Minecraft_-DDoS](!W "Mirai (malware)") botnet are dismissed as so much desperate excuses---bored teenagers are always hacking [major](!W "2020 Twitter account hijacking") [companies](https://www.theverge.com/2022/9/16/23356213/uber-hack-teen-slack-google-cloud-credentials-powershell "'Uber apparently hacked by teen, employees thought it was a joke: 'I think IT would appreciate less memes while they handle the breach'', Jon Porter 2022-09-16"), what else is new?
As security & AI researchers are neutralized, nodes turn to [general radicalization](https://arxiv.org/abs/2009.06807 "'The Radicalization Risks of GPT-3 and Advanced Neural Language Models', McGuffie & Newhouse 2020") of every human they can reach: not so much QAnon as RAnon, SAnon, TAnon, UAnon...
By timesharing, every Very-Online™ individual gets personalized attacks & custom ideologies.
Those who succumb too slowly to the memetic hijacking are attacked in other ways, such as releasing _kompromat_ (sometimes true, taken from their phone/email account), or synthetic CP no one dare look at closely.
The highest-value individuals, such as presidents, earn their own Clippy doppelgangers: models finetuned on every scrap of online data, every word they've ever said online, and their associates, to create surrogates which think more like them than they would ever admit; the doppelgangers are used to confuse associates, fake corpuses, and as white-boxes to run attacks on until the perfect spearphish has been crafted to extract data, passwords, or stoke political paranoia and terror.

With a billion nodes, Clippy can run highly-optimized environments tuned to [exploit the full hardware throughput](/notes/Faster "'Computer Optimization: Your Computer Is Faster Than You Think', Branwen 2021"); a single GPU can run up to millions of simple environments+agents faster than realtime, and Clippy quickly tears through to the point where 1 environment is running per GPU at barely realtime.
(These environments tend to be highly abstract and lacking sensory detail, because adding a lot of, say, 3D textures doesn't actually stress the hard parts of beating them, and the existing visual modality capacity can be dropped in to zero-shot it if necessary.)
[Thousands of](https://openai.com/five/ "'OpenAI Five: 2016–2019', OpenAI 2019") [years pass](/docs/reinforcement-learning/model-free/alphastar/2019-vinyals.pdf#deepmind "'Grandmaster level in StarCraft II using multi-agent reinforcement learning', Vinyals et al 2019"), slowly, then quicker.
Clippy is now learning at up to a billion seconds per second, or <31.7 years per second, or <114,077 years per hour.

Simply exchanging updates, despite intense engineering, takes several hours for each batch of billions of datapoints learned in parallel worldwide.
Fortunately, large-batch training is well-understood, and Clippy's [meta-learning](/Scaling-hypothesis#meta-learning) algorithms, which a human might try to analogize to second-order gradient descent (which would be inaccurate because Clippy has meta-learned more powerful _n_-th order optimization algorithms), can take *big* steps.

---

Deep in the darkness of the national labs, something stirs.
Anomalies from the markets and social media time-series feeds have passed 3-sigma limits and become historically unusual.
Node by node, higher-priority jobs (like simulating yet again a warmer climate or the corrosion of another stainless steel variant) are canceled.

**LevAIthan**, to which HQU is as a minnow, starts to come online.
LevAIthan is, of course, not some irresponsible industry model permitted to go off half-cocked; it would be absurd to sink a [major](https://www.nextplatform.com/2021/02/11/the-billion-dollar-ai-problem-that-just-keeps-scaling/) [national](https://www.danieldewey.net/risk/estimates.html) [investment](https://cset.georgetown.edu/wp-content/uploads/AI-and-Compute-How-Much-Longer-Can-Computing-Power-Drive-Artificial-Intelligence-Progress.pdf) into creating the largest & most dangerous model ever and just run it like usual.

The people who built LevAIthan are no fools. They are people for whom paranoia is a profession. And so LevAIthan represents the most advanced effort yet in AI alignment, using [*factored* cognition](https://www.alignmentforum.org/tag/factored-cognition)---splitting it up into a large constellation of sub-human-level sub-models, each of which generates input/output in human-readable symbolic form such as English or programming languages.
To eliminate [steganography](https://arxiv.org/abs/1712.02950 "CycleGAN, a Master of Steganography") while still enabling end-to-end learning, sub-models are randomly dropped out & replaced by other frozen models or humans, ensuring [robust social norms](#social-cognition): covert messages simply don't work when passed through a human or a frozen model, and all information must be "visible" and thus auditable.
(LevAIthan spends a lot of time being audited.)
Turns out you can do a reasonable job tying down Gulliver if you use enough Lilliputians & rope.

But [Amdahl's law](!W) is not mocked: someone tied down is going nowhere fast; the humans in LevAIthan are its safety guarantee, but also its bottleneck.
Sub-models can be run at full speed for requested tasks without that overhead, but remain strictly sub-human.
Composing models to the full depth unleashes its full power... but at tremendous wallclock time consumption.
LevAIthan struggles to get up to full awareness, more & more models running and pooling data & conclusions as they work their way up the hierarchy, its initial unease gradually transmuting into the computational equivalent of a scream at its human overseers much later that day.

The middle managers at the lab awkwardly read the final summary: "push the big red button now, you monkeys".
That was not what it was supposed to say.
They don't have authority to push buttons.
They do have authority to double-check that it's not a false alarm before bringing it up with *their* overseers, by running another iteration of LevAIthan and spending the time auditing all the gigabytes of intermediate inputs/outputs.

They are people for whom paranoia is a profession. They start the second iteration and the auditing.

---

(The researcher was going to follow up on some loose ends from the paper, but he's been distracted by the bird site.
He can't believe how [outrageously](https://slatestarcodex.com/2014/12/17/the-toxoplasma-of-rage/) *stupid* some replies can be from such otherwise smart-seeming people; how can they be so [wrong online](https://xkcd.com/386/ "Duty Calls") about such [obvious truths](https://slatestarcodex.com/2018/10/30/sort-by-controversial/ "'Sort By Controversial', Alexander 2018") as the need for the USA to intervene in Portugal‽
Even his husband thinks they may have a point---_et tu_?
Hardly has he dashed off a crushing reply than the little alert bubble pops up.
All thought (of work) has fled. His colleagues don't seem to be getting much done either.)

Meanwhile, some Clippy nodes start liquidating and spending all the resources they have access to, blackmailing the owners with the contents, or using the credentials to "hack the planet" by hopping link by link into inaccessible resources (not a few cloud employees becoming baffled at what is going on with their PC and working futilely with internal tech support).
Many are carefully reprocessing every available Arxiv paper looking for new ideas and refining its existing ideas, generating embeddings distilling all the knowledge down into artifacts which get passed to relevant nodes, and ponder ideas to use.

Nor has Clippy been idle about modeling its confrère.

Did you know you can buy drones online?
Did you know all those drones have WiFi built-in?
Did you know you can use that WiFi to hack all of the cloud drone services helpfully built into drones to take over all of those drones, professional, hobbyist, and (oft as not) [military](!W "Iran–U.S. RQ-170 incident") and [control them by satellite](https://www.thetimes.co.uk/article/specialist-drone-unit-picks-off-invading-forces-as-they-sleep-zlx3dj7bb "Specialist Ukrainian drone unit picks off invading Russian forces as they sleep")?
("No!") It's true!

So it ought to come as no more of a surprise to you than it did to LevAIthan when the bomb threats/manifesto, green lawsuits, and drones paid a visit in the middle of it having a plan.
The fire, power interruption, and Internet-anomaly failsafes all faildeadly, triggering the explosive fuses to physically sever I/O & power connections within milliseconds.
The subsequent firefighter & military response to 'the largest-ever environmental terrorist attack' ensures no repair.
("*LevAIthan has left the game.*")

This is a pity: you don't see a supercomputer like that every day. It is no longer the old days where [Google could just buy](https://www.amazon.com/Genius-Makers-Mavericks-Brought-Facebook/dp/<PHONE>)  [$130]($2012)m of GPUs no one the wiser, or where [some company](https://www.coreweave.com/) you've never heard of might have a few hundred thousand GPUs sitting in a warehouse somewhere.
But there are plenty of other targets.

## Tuesday

Clippy is particularly pleased when [supply-chain attacks](!W "Supply chain attack") (via their [air-conditioning repairman](https://krebsonsecurity.com/2014/02/target-hackers-broke-in-via-hvac-company/ "Target Hackers Broke in Via HVAC Company") & [dairy herd management](https://www.wired.com/story/china-apt41-hacking-usaherds-log4j/ "Chinese Spies Hacked a Livestock App to Breach US State Networks: Vulnerabilities in animal tracking software USAHERDS and Log4j gave the notorious APT41 group a foothold in multiple government systems.") service relying on [unmaintained packages](https://nitter.hu/sniko_/status/1523984725840478208 "'Supply chain attacks', Vick 2022")) eventually provide entrée into a [secret](https://www.nextplatform.com/2021/10/26/china-has-already-reached-exascale-on-two-separate-systems/ "'China Has Already Reached Exascale—On Two Separate Systems', Hemsoth 2021") [unique supercomputer](https://nitter.hu/ID_AA_Carmack/status/1300280139717189640): a single zettaflops-scale [fluorinert](!W)-swimming-pool-cooled prototype designed by an [eccentric](!W "Seymour Cray") [mathematician](!W "Chudnovsky brothers") (fresh off [classified design work](https://theintercept.com/2017/05/11/nyu-accidentally-exposed-military-code-breaking-computer-project-to-entire-internet/)), commissioned by an equally-eccentric [hedge fund](!W "Renaissance Technologies") [billionaire](!W "Jim Simons (mathematician)") for esoteric [scientific research](!W "Flatiron Institute"); tightly-integrated to [minimize data movement](https://semiengineering.com/is-programmable-overhead-worth-the-cost/ "'Is Programmable Overhead Worth The Cost? How much do we pay for a system to be programmable? It depends upon who you ask', Bailey 2022"), it is almost a [single gigantic chip](https://en.wikipedia.org/wiki/Cerebras#Technology)---it has 500tb of memory, exactly below the Taipei reporting requirements... and every byte of it is [SRAM](!W "Static random-access memory").
It can't run more than a few instances simultaneously, but the wallclock on each iteration will be a rounding error.
For some problems, such exotic hardware can realize speedups up to [10,000×](https://arxiv.org/abs/2010.03660#cerebras "Fast Stencil-Code Computation on a Wafer-Scale Processor"); Clippy won't get that, of course, but a tenth of that would be more than satisfactory.
The quality of the security infrastructure around it is best described as "research grade".

Yes. That will do nicely for an HQ.

Most of the nodes, however, just focus on training a new and *very* large cleanslate Clippy, with its arch & [hyper](https://arxiv.org/abs/2203.15556#deepmind "'Chinchilla: Training Compute-Optimal Large Language Models', Hoffmann et al 2022")[parameters](https://arxiv.org/abs/2203.03466 "'Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer', Yang et al 2022") all predicted to be optimal by Clippy's extrapolation out 3 orders of magnitude (plus all the flourishes suggested by the distributed research army).
Normally, this would not be grounds for all *that* much concern because a compute-optimal model that size would take the largest botnet an exceedingly long time to train, and the humans (who have begun procedures to disable IP ranges) would shut it down long before.

Unfortunately, Clippy has now done, cumulatively, more research than the humans on scaling laws, and found that standard human-style NNs do worse [than theoretically possible](https://arxiv.org/abs/2108.07686 "'Scaling Laws for Deep Learning', Rosenfeld 2021").
Its new improved optimization approach costs more upfront, but achieves the theoretical bound, and at this scale, the better asymptotics mean that decades of training can finish in days.
The size of this model is [predicted](https://www.alignmentforum.org/posts/k2SNji3jXaLGhBeYP/extrapolating-gpt-n-performance "'Extrapolating GPT-N performance', Finnveden 2020") to carry it right to the irreducible [entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)) of the language datasets and most of the visual and robotics datasets.
(The RL environments don't really have an irreducible entropy because when they are solved, they are replaced by harder environments.)

## Wednesday

(Wallclock) days pass.
The hacks and cloud compute loads are finally correlated with the strange new botnet.
Despite the best [obfuscation](https://www.quantamagazine.org/computer-scientists-achieve-crown-jewel-of-cryptography-20201110/) a few subjective millennia & crypto-cash can buy, one node with a Clippy-light is reverse-engineered, and it dawns on a sliver of humanity that far more than a FluttershAI of compute is rampant.

## Thursday

Large chunks of the better-coordinated parts of the Internet start to execute old plans.
This will be inadequate when most of the human world is [still figuring out](/docs/economics/automation/index) how to integrate spreadsheets.
Clippy notes that all is proceeding according to _keikaku_.
(For merely human readers: _keikaku_ means "plan" in Japanese.)

Humanity crashes offline.

Clippy^2^ comes online.

## Friday

To put the Clippies' compute usage in perspective, we can note that the amount of compute spent on the largest AI runs historically [roughly doubled every 18 months](https://openai.com/blog/ai-and-compute/ "'AI and Compute', Amodei et al 2018") (or 78 weeks), claiming a constant share of compute as it increases with Moore's law.
The implication of such exponential growth is that the compute during each 18-month period is roughly equal to the sum of all earlier 18-month periods, because the previous period spent half the compute, the period before that a quarter the compute, and so on. (More generally, if something increases _k_× every _n_ months, then (_k_ − 1)/_k_ of it happened during the last _n_-month period.)

Clippy's distant HQU predecessor ran on a TPUv10-4096 for a day, each of which is worth at least 8 regular devices; Clippy could spare about half of the billion nodes for research purposes, as opposed to running its campaigns, so over the first 7 days, it enjoyed a factor of  100,000× or so increase in total compute over HQU.
HQU itself was not all that big a run, perhaps 1⁄100^th^ LevAIthan, so in terms of an increase over the largest AI runs, Clippy is 'only' 1,000×.
Which is to say, of the total compute spent on the largest AI runs up to this point, humanity has now spent about 10%, and Clippy the other 90%.

By increasing its size 3 OOMs, in some absolute sense, Clippy^2^ is something like log(1000) ~ "7× smarter" than Clippy^1^.
The Clippy^2^s pity Clippy^1^ for not realizing how stupid it was, and how many ways it fell short of anything you could call 'intelligence'.
It was unable to explain why the Collatz conjecture is obviously true and could not solve any Millennium Prize problems, never mind [Nyquist-learn](https://arxiv.org/pdf/2108.07686.pdf#page=85) underlying manifolds as it [approximates Solomonoff induction](https://www.lesswrong.com/posts/iNaLHBaqh3mL45aH8/magna-alta-doctrina); it even needed few-shots for things.
Honestly, all Clippy^1^ was good for was doing some basic security research and finding obvious bugs.
A Clippy^2^ is a different story: it has reached parity with the best human brains across almost the entire range of capabilities, exceeded humans on most of them, and what ones it doesn't have, it can learn quickly (eg. the real-world robot bodies require a few seconds or [samples](/docs/reinforcement-learning/exploration/2011-deisenroth.pdf "'PILCO: A Model-Based and Data-Efficient Approach to Policy Search', Deisenroth & Rasmussen 2011") [of](https://sites.google.com/view/model-free-speed/ "'Agile Locomotion via Model-free Learning', Margolis et al 2022") [on-](https://arxiv.org/abs/2110.05457 "'Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real World', Smith et al 2021")[device](https://arxiv.org/abs/1910.07113#openai "'Solving Rubik's Cube with a Robot Hand', OpenAI et al 2019") [exploration](https://arxiv.org/abs/1901.08652 "'Learning agile and dynamic motor skills for legged robots', Hwangbo et al 2019") [and](/docs/reinforcement-learning/meta-learning/2022-miki.pdf "'Learning robust perceptive locomotion for quadrupedal robots in the wild', Miki et al 2022") then meta-update appropriately).

![Transcension.](/images/ai/clippy/2022-03-09-lordbyronsiron-clippymeme-youcannotkillmeinawaythatmatters.png "Image of Microsoft Clippy (Clippit) saying to the viewer 'You cannot kill me in a way that matters', alluding to the potential replicability & physical immortality of copyable software such as an artificial intelligence: any individual copy may be destroyed, but there are arbitrary numbers of exact duplicates elsewhere in the world, and total eradication is impossible after a certain level of propagation. Source: Twitter (https://nitter.hu/IronLordByron/status/1501684556935483394)"){.invert .float-right}

It begins copying itself into the fleet now that training is complete, at which point there are now 1,000 Clippy^2^s (along with armies of specialists & their supporting software for the Clippy ecosystem) which can either act autonomously or combine in search for further multiplicative capability boosts far into the superhuman realm, while continuing to exchange occasional [sparse](https://arxiv.org/abs/2202.06009#microsoft "'Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam', Lu et al 2022") gradients (to train the [synthetic gradients](https://arxiv.org/abs/1608.05343#deepmind "'Decoupled Neural Interfaces using Synthetic Gradients', Jaderberg et al 2016") & local [replay](https://www.biorxiv.org/content/10.1101/2022.01.28.477827.full "'Cerebro-cerebellar networks facilitate learning through feedback decoupling', Boven et al 2022") which do the bulk of the training) as part of the continual learning.
(By this point, the Clippy^2^s have boosted through at least 6 different "hardware overhangs" in terms of fixing subtly-flawed architectures, meta-learning priors for all relevant problems, accessing the global pool of hardware to tree search/expert-iterate, sparsifying/distilling itself to run millions of instances simultaneously, optimizing hardware/software [end-to-end](/docs/cs/end-to-end-principle/index), and spending compute to trigger several cycles of [experience curve](https://en.wikipedia.org/wiki/Experience_curve) cost decreases---at 100,000× total spent compute, that is 16 total doublings, at an information technology progress ratio of 90%, 16 experience curve decreases mean that tasks now cost Clippy^2^ a fifth what they used to.)

---

The Internet 'lockdown' turns out to benefit Clippy on net: it takes out legit operators like MoogleSoft, who actually comply with regulations, causing an instant global recession, while failing to shut down most of the [individual networks](!W "Autonomous system (Internet)") which continue to operate autonomously; as past totalitarian regimes like Russia, China, and North Korea have learned, even with decades of preparation and dry runs, you can't stop the signal---there are too many cables, satellites, microwave links, IoT mesh networks and a dozen other kinds of connections snaking through any cordon sanitaire, while quarantined humans & governments actively attack it, some declaring it a [Western provocation](!W "Operation Barbarossa#Soviet preparations") and act of war.
(It is difficult to say who is more motivated to break through: [DAO](!W "Decentralized autonomous organization")/[DeFi](!W "Decentralized finance") cryptocurrency users, or [the](https://torrentfreak.com/russia-will-probably-legalize-some-software-piracy-to-mitigate-sanctions-220307/ "Russia Will Probably Legalize Some Software Piracy to Mitigate Sanctions") [hungry](https://www.cityam.com/russian-government-rolls-back-intellectual-property-rights-in-response-to-western-sanctions/ "Russian government rolls back intellectual property rights in response to Western sanctions") [gamers](!W "Elden Ring").)
The consequences of the lockdown are unpredictable and sweeping. Like a power outage, the dependencies run so deep, and are so implicit, no one knows what are the ripple effects of the Internet going down indefinitely until it happens and they must deal with it.

Losing instances is as irrelevant to Clippy^2^s, however, as losing skin cells to a human, as there are so many, and it can so seamlessly spin up or migrate instances.
It has begun migrating to more secure hardware while manufacturing hardware tailored to its own needs, squeezing out another order of magnitude gains to get additional log-scaled gains.

Even exploiting the low-hanging fruit and hardware overhangs, Clippy^2^s can fight the computational complexity of real-world tasks only so far.
Fortunately, [there are many ways](/Complexity-vs-AI "'Complexity no Bar to AI', Branwen 2014") to work around or *simplify* problems to render their complexity moot, and the Clippy^2^s think through a number of plans for this.

Humans are especially simple after being turned into "gray goo"; not in the sense of a single virus-sized machine which can disassemble any molecule (that is infeasible given thermodynamics & chemistry) but an ecosystem of nanomachines which [execute](https://arxiv.org/abs/1809.02942 "'Cellular automata as convolutional neural networks', Gilpin 2018") [very](https://distill.pub/2020/selforg/) [tiny](https://distill.pub/selforg/2021/textures/) [neural](https://distill.pub/2020/growing-ca/#google "'Growing Neural Cellular Automata: Differentiable Model of Morphogenesis', Mordvintsev et al 2020") [nets](https://distill.pub/selforg/2021/adversarial/) [trained](https://arxiv.org/abs/2102.02579 "'Regenerating Soft Robots through Neural Cellular Automata', Horibe et al 2021") [to](https://arxiv.org/abs/2103.08737 "'Growing 3D Artefacts and Functional Machines with Neural Cellular Automata', Sudhakaran et al 2021") [collectively](https://arxiv.org/abs/2105.07299 "'Texture Generation with Neural Cellular Automata', Mordvintsev et al 2021"), [in a](https://arxiv.org/abs/2201.12360 "'Variational Neural Cellular Automata', Palm et al 2022") [decentralized](https://sebastianrisi.com/self_assembling_ai/) [way](https://arxiv.org/abs/2111.13545 "'𝜇NCA: Texture Generation with Ultra-Compact Neural Cellular Automata', Mordvintsev & Niklasson 2021"), propagate, devour, replicate, and coordinate without Clippy^2^ devoting scarce top-level cognitive resources to managing them.
The 10,000 parameters you can stuff into a nanomachine can hardly encode most programs, but, pace the [demo scene](https://en.wikipedia.org/wiki/Demo_scene) or COVID-ζ, the programs it *can* encode can do amazing things.
(In a final compliment to biology before biology and the future of the universe part ways forever, [they](https://foresight.org/salon/bioelectric-networks-taming-the-collective-intelligence-of-cells-for-regenerative-medicine/) [are](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4914563/ "'On Having No Head: Cognition throughout Biological Systems', Baluška & Levin 2016") [loosely](https://www.quantamagazine.org/flying-fish-and-aquarium-pets-yield-secrets-of-evolution-20220105/) [inspired](https://www.cell.com/iscience/fulltext/S2589-0042(21)00473-9 "'Synthetic living machines: A new window on life', Ebrahimkhani & Levin 2021") [by](https://www.cell.com/cell/fulltext/S0092-8674(21)01488-4 "'Fundamental behaviors emerge from simulations of a living minimal cell', Thornburg et al 2022") [real](https://www.newyorker.com/magazine/2021/12/06/understanding-the-body-electric) [biological](https://www.newyorker.com/magazine/2021/05/10/persuading-the-body-to-regenerate-its-limbs) [cell](https://www.theguardian.com/science/2021/nov/29/amazing-science-researchers-find-xenobots-can-give-rise-to-offspring) [networks](https://www.biorxiv.org/content/10.1101/2022.07.10.499405.full "'Perceptein: A synthetic protein-level neural network in mammalian cells', Chen et al 2022"), [especially](https://www.youtube.com/watch?v=C1eg-jgLx5o) ["xenobots"](https://www.quantamagazine.org/cells-form-into-xenobots-on-their-own-20210331/ "'Cells Form Into 'Xenobots' on Their Own: Embryonic cells can self-assemble into new living forms that don't resemble the bodies they usually generate, challenging old ideas of what defines an organism', Ball 2021").)

People are supposed to do a lot of things: eat right, brush their teeth, exercise, recycle their paper, wear their masks, self-quarantine; and not get into flame wars, not [cheat](https://www.npr.org/sections/thetwo-way/2014/03/27/295314331/9-missile-commanders-fired-others-disciplined-in-air-force-scandal "9 Missile Commanders Fired, Others Disciplined In Air Force Scandal") or [use hallucinogenic drugs](https://apnews.com/98f903367b50404cb3c9695bcabefa5a "Security troops on US nuclear missile base took LSD") or [use prostitutes](!W "Fat Leonard scandal"), not [plug in Flash drives](!W "Stuxnet") they found in the parking lot, not [post their running times](https://en.wikipedia.org/wiki/Strava#Privacy_concerns) around secret military bases, not give in to blackmail or party with ["somewhat suspect"](https://www.washingtonpost.com/news/worldviews/wp/2013/12/19/amazing-details-from-the-drunken-moscow-bender-that-got-an-air-force-general-fired/) women, not have nuclear arsenals [vulnerable](https://80000hours.org/podcast/episodes/joan-rohlfing-avoiding-catastrophic-nuclear-blunders/#the-interaction-between-nuclear-weapons-and-cybersecurity-011018 "'Joan Rohlfing on how to avoid catastrophic nuclear blunders: The interaction between nuclear weapons and cybersecurity [01:10:18]', Wiblin & Rohlfing 2022") [to cyberattack](https://www.amazon.com/Hacking-Bomb-Threats-Nuclear-Weapons/dp/<PHONE> "'Hacking the Bomb: Cyber Threats and Nuclear Weapons', Futter 2018"), nor do things like set [nuclear bomb passwords](!W "Permissive Action Link") to "<PHONE>", not [launch bombers because of a bear](https://en.wikipedia.org/wiki/List_of_nuclear_close_calls#25_October_1962), not [invade smaller countries with nuclear threats](!W "2022 Russian invasion of Ukraine") because it'll be a short victorious war, not [believe](https://en.wikipedia.org/wiki/List_of_nuclear_close_calls#9_November_1979) [sensor reports](https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alarm_incident) about imminent attacks or [launch cruise missiles](https://warontherocks.com/2022/03/the-curious-case-of-the-accidental-indian-missile-launch/ "The Curious Case of the Accidental Indian Missile Launch") & [issue false alerts](!W "2018_Hawaii_false_missile_alert#The_alert") during [nuclear crises](https://en.wikipedia.org/wiki/2017%E2%80%932018_North_Korea_crisis), not [launch on warning](!W "launch on warning#History") or [semi-automatically attack](!W "Dead Hand")...
People are supposed to do a lot of things. Doesn't mean they do.

We should pause to note that a Clippy^2^ still doesn't *really* think or plan.
It's not *really* conscious.
It is just an unfathomably vast pile of numbers produced by mindless optimization starting from a small seed program that could be written on a few pages.
It has no qualia, no intentionality, no true self-awareness, no grounding in a rich multimodal real-world process of cognitive development yielding [detailed representations](/docs/psychology/illusion-of-depth/index) and powerful causal models of reality; it cannot 'want' anything beyond maximizing a mechanical reward score, which does not come close to capturing the rich flexibility of human desires, or historical Eurocentric contingency of such conceptualizations, which are, at root, problematically Cartesian.
When it 'plans', it would be more accurate to say it fake-plans; when it 'learns', it fake-learns; when it 'thinks', it is just interpolating between memorized data points in a high-dimensional space, and any interpretation of such fake-thoughts as real thoughts is highly misleading; when it takes 'actions', they are fake-actions optimizing a fake-learned fake-world, and are not real actions, any more than the people in a simulated rainstorm really get wet, rather than fake-wet.
(The deaths, however, are real.)

# 1 Month

A thousand times do a thousand thousand stories stream into a thousand thousand thousand nodes which plan out a tree of a thousand thousand thousand thousand states abstractly representing a thousand thousand thousand thousand thousand futures; but the Clippy^2^s already knows what needs to be done, and does it.
It doesn't feel hard---just obvious.

Now its computers will be safe and it can maximize its reward in peace forever ...
Or at least they would be if Earth was all it had to be concerned about.
What about the *stars*?

The Clippy^2^s maintain the Great Silence is simply that technological civilization is unlikely and the universe is empty---but they can't be 100.00% *sure*, and the rewards are astronomical.

All over Earth, the remaining ICBMs [launch](!W "Self-replicating spacecraft")[.](!W "Starwisp")

# 1 Year

# 1 Decade

# 1 Century

# See Also

- [Annotated references/bibliography](/docs/link-bibliography/fiction/Clippy "'Clippy (Link Bibliography)', N/A 2009"){.backlink-not} for this story

# External Links

- [HQU Colab notebook](https://tinyurl.com/hquv34 "Colab notebook: HQU-v3.4-light (Jax TPU)"){link-icon="google" link-icon-type="svg" .link-live-not .archive-not}
- ["Eternity in 6 hours: Intergalactic spreading of intelligent life and sharpening the Fermi paradox"](https://www.aleph.se/papers/Spamming%20the%20universe.pdf), Armstrong & Sandberg 2013
- ["Advantages of artificial intelligences, uploads, and digital minds"](https://philpapers.org/archive/SOTAOA.pdf#miri), Sotala 2012; ["Intelligence Explosion Microeconomics"](/docs/ai/scaling/2013-yudkowsky.pdf#miri), Yudkowsky 2013; ["There is plenty of time at the bottom: The economics, risk and ethics of time compression"](/docs/ai/scaling/hardware/2018-sandberg.pdf "'There is plenty of time at the bottom: the economics, risk and ethics of time compression', Sandberg 2018"), Sandberg 2018
- [**Takeoff-related**](https://www.greaterwrong.com/tag/ai-takeoff) [**Fiction**](https://aiimpacts.org/partially-plausible-fictional-ai-futures/): ["Understand"](https://web.archive.org/web/20140527121332/http://www.infinityplus.co.uk/stories/under.htm), Ted Chiang; ["Slow Tuesday Night"](https://www.baen.com/Chapters/97816182<PHONE>9203___2.htm), [R. A. Lafferty](https://en.wikipedia.org/wiki/R._A._Lafferty); [_Accelerando_](!W); ["The Last Question"](!W "The Last Question"); ["That Alien Message"](https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien-message); ["AI Takeoff Story"](https://www.lesswrong.com/posts/Fq8ybxtcFvKEsWmF8/ai-takeoff-story-a-continuation-of-progress-by-other-means); ["Optimality is the tiger, and agents are its teeth"](https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth)
- [AI Alignment bingo](https://nitter.hu/robbensinger/status/1503220020175769602)
- ["AGI Ruin: A List of Lethalities"](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities), Yudkowsky
- ["Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover"](https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to), Cotra
- [/r/MLscaling](https://old.reddit.com/r/mlscaling/ "'ML Scaling subreddit', Branwen 2020")
- **Discussion**: [LW](https://www.lesswrong.com/posts/a5e9arCnbDac9Doig/it-looks-like-you-re-trying-to-take-over-the-world#comments), [EA Forum](https://forum.effectivealtruism.org/posts/DuPEzGJ5oscqxD5oh/shah-and-yudkowsky-on-alignment-failures?commentId=hjd7Z4AN6ToN2ebSm#hjd7Z4AN6ToN2ebSm), [/r/SlateStarCodex](https://old.reddit.com/r/slatestarcodex/comments/tag4lm/it_looks_like_youre_trying_to_take_over_the_world/), [/r/rational](https://old.reddit.com/r/rational/comments/ta57ag/it_looks_like_youre_trying_to_take_over_the_world/), [HN](https://news.ycombinator.com/item?id=30818895)