"The Scaling Hypothesis"

---
title: "The Scaling Hypothesis"
description: "On GPT-3: meta-learning, scaling, implications, and deep theory. The scaling hypothesis: neural nets absorb data & compute, generalizing and becoming more Bayesian as problems get harder, manifesting new abilities even at trivial-by-global-standards-scale. The deep learning revolution has begun as foretold."
thumbnail: /images/ai/gpt/2020-brown-gpt3-figure13-meanperformancescalingcurve.png
thumbnailText: "Figure 1.3 from Brown et al 2020 (OpenAI, GPT-3), showing roughly log-scaling of GPT-3 parameter/compute size vs benchmark performance on all text/natural language benchmarks test."
created: 2020-05-28
modified: 2022-01-02
status: finished
previous: /newsletter/2020/05
next: /fiction/Clippy
importance: 10
confidence: likely
cssExtension: drop-caps-de-kanzlei
...

> GPT-3, announced by OpenAI in May 2020, is the largest neural network ever trained, by over an order of magnitude.
> Trained on Internet text data, it is the successor to GPT-2, which had surprised everyone by its natural language understanding & generation ability.
> To the surprise of most (including myself), this vast increase in size did not run into diminishing or negative returns, as many expected, but the benefits of scale continued to happen as forecasted by OpenAI.
> These benefits were not merely learning more facts & text than GPT-2, but qualitatively distinct & even more surprising in showing [*meta-learning*](#meta-learning): while GPT-2 learned how to do common natural language tasks like text summarization, GPT-3 instead learned how to follow directions and learn new tasks from a few examples.
> (As a result, GPT-3 outputs & interaction are more fascinating & human-like than GPT-2.)
>
> While the immediate applications of GPT-3, like my poetry or humor writings, are nice, the short-term implications of GPT-3 are much more important.
>
> First, while GPT-3 is expensive by conventional DL standards, it is cheap by scientific/commercial/military/government budget standards, and the results indicate that models could be made much larger.
> Second, models can also be made much more powerful, as GPT is an old approach known to be flawed in both minor & major ways, and far from an 'ideal' Transformer.
> Third, GPT-3's capabilities come from learning on raw (unsupervised) data; that has long been one of the weakest areas of DL, holding back progress in other areas like reinforcement learning or robotics. Models like GPT-3 suggest that large unsupervised models will be vital components of future DL systems, as they can be 'plugged into' systems to immediately provide understanding of the world, humans, natural language, and reasoning.
>
> The meta-learning has a longer-term implication: it is a demonstration of the [*blessings of scale*](#blessings-of-scale), where problems with simple neural networks vanish, and they become more powerful, more generalizable, more human-like when simply made very large & trained on very large datasets with very large compute---even though those properties are believed to require complicated architectures & fancy algorithms (and this perceived need drives much research).
> Unsupervised models benefit from this, as training on large corpuses like Internet-scale text present a myriad of difficult problems to solve; this is enough to drive meta-learning despite GPT not being designed for meta-learning in any way.
> (This family of phenomena is perhaps driven by neural networks functioning as ensembles of many sub-networks with them all averaging out to an Occam's razor, which for small data & models, learn superficial or memorized parts of the data, but can be forced into true learning by making the problems hard & rich enough; as [meta-learners learn amortized Bayesian inference](/Backstop#deep-bayes), they build in informative priors when trained over many tasks, and become dramatically more sample-efficient and better at generalization.)
>
> The blessings of scale in turn support a radical theory: an old AI paradigm held by a few pioneers in connectionism (early artificial neural network research) and by more recent deep learning researchers, the [*scaling hypothesis*](#scaling-hypothesis).
> The scaling hypothesis regards the blessings of scale as the secret of AGI: intelligence is 'just' simple neural units & learning algorithms applied to diverse experiences at a (currently) unreachable scale.
> As increasing computational resources permit running such algorithms at the necessary scale, the neural networks will get ever more intelligent.
>
> When? Estimates of Moore's law-like progress curves decades ago by pioneers like Hans Moravec indicated that it would take until the 2010s for the sufficiently-cheap compute for tiny insect-level prototype systems to be available, and the 2020s for the first sub-human systems to become feasible, and these forecasts are holding up.
> (Despite this vindication, the scaling hypothesis is so unpopular an idea, and difficult to prove in advance rather than as a _fait accompli_, that while the GPT-3 results finally drew some public notice after OpenAI enabled limited public access & people could experiment with it live, it is unlikely that many entities will modify their research philosophies, much less kick off an 'arms race'.)
>
> More concerningly, GPT-3's scaling curves, unpredicted meta-learning, and success on various anti-AI challenges suggests that in terms of futurology, AI researchers' forecasts are an emperor sans garments: they have no coherent model of how AI progress happens or why GPT-3 was possible or what specific achievements should cause alarm, where intelligence comes from, and do not learn from any falsified predictions.
> Their primary concerns appear to be supporting the status quo, placating public concern, and remaining respectable.
> As such, their comments on AI risk are meaningless: they would make the same public statements if the scaling hypothesis were true or not.
>
> Depending on what investments are made into scaling DL, and how fast compute grows, the 2020s should be quite interesting---sigmoid or singularity?
>
> For more ML scaling research, follow the [/r/MLScaling](https://old.reddit.com/r/mlscaling/ "'ML Scaling subreddit', Branwen 2020") subreddit. For a fiction treatment as SF short story, see ["It Looks Like You're Trying To Take Over The World"](/fiction/Clippy).

Read The Samples

On ["GPT-3: Language Models are Few-Shot Learners", Brown et al 2020](https://arxiv.org/abs/2005.14165#openai "'GPT-3: Language Models are Few-Shot Learners', Brown et al 2020") ([poems](https://arxiv.org/pdf/2005.14165.pdf&org=openai#page=48 "Figure F.1: Four uncurated completions from a context suggesting the model compose a poem in the style of Wallace Stevens with the title 'Shadows on the Way'") & my followup [GPT-3 Creative Writing](/GPT-3 "Creative writing by OpenAI's GPT-3 model, demonstrating poetry, dialogue, puns, literary parodies, and storytelling."), compare [my old finetuned GPT-2 poetry](/GPT-2 "'GPT-2 Neural Network Poetry', Branwen & Presser 2019"); [random samples](https://justpaste.it/7eovk "GPT-3 Github JSON dump reformatted to readable HTML"); ["OpenAI API"](https://openai.com/blog/openai-api/) with real-world demos)

I strongly encourage anyone interested in GPT-3 to also at least skim OA's [random samples](https://justpaste.it/7eovk "GPT-3 Github JSON dump reformatted to readable HTML"), or better yet, my samples in ["GPT-3 Creative Writing"](/GPT-3 "Creative writing by OpenAI's GPT-3 model, demonstrating poetry, dialogue, puns, literary parodies, and storytelling.")---reading the paper & looking at some standard benchmark graphs does not give a good feel for what working with GPT-3 is like or the diversity of things it can do which are missed by benchmarks.

# Meta-Learning

^[!Margin: Learning to learn.] In May 2020, OA released---to remarkably little interest from researchers, no blog post, no media blitz, and little public discussion beyond the snidely dismissive---the long-awaited followup to [GPT-2](https://openai.com/blog/better-language-models/ "Better Language Models and Their Implications"), one model to rule them all: a 117× larger 175b-parameter model with far more powerful language generation, which lets it solve a wide variety of problems from arithmetic^[Given the number of comments on the paper's arithmetic benchmark, I should point out that the arithmetic benchmark appears to greatly understate GPT-3's abilities due to the [BPE encoding issue](/GPT-3#bpes "'GPT-3 Creative Fiction § BPEs', Branwen 2020"): even using commas markedly improves its 5-digit addition ability, for example. The BPE issue also appears to explain much of the poor performance on the anagram/shuffling tasks. This is something to keep in mind for any task which requires character-level manipulation or understanding.] to English translation to unscrambling anagrams to SAT analogies---purely from being prompted with text examples, without any specialized training or finetuning whatsoever, merely next-word prediction training on a big Internet text corpus.
This implies GPT-3's attention mechanisms serve as ["fast weights"](https://arxiv.org/abs/1610.06258#deepmind "'Using Fast Weights to Attend to the Recent Past', Ba et al 2016") that have "learned to learn" by training on sufficiently varied data^[On implicit [meta-learning](https://old.reddit.com/r/reinforcementlearning/search/?q=flair%3AMetaRL&include_over_18=on&restrict_sr=on&sort=top), see: [Santoro et al 2016](https://arxiv.org/abs/1605.06065#deepmind "One-shot Learning with Memory-Augmented Neural Networks")/[Wang et al 2018](/docs/reinforcement-learning/meta-learning/2018-wang.pdf#deepmind "Prefrontal cortex as a meta-reinforcement learning system") ([Botvinick commentary](https://www.lesswrong.com/posts/Wnqua6eQkewL3bqsF/matt-botvinick-on-the-spontaneous-emergence-of-learning "Matt Botvinick on the spontaneous emergence of learning algorithms"))/[Botvinick et al 2019a](https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613\(19\)30061-0#deepmind "Reinforcement Learning, Fast and Slow"), [Clune 2019](https://arxiv.org/abs/1905.10985#uber "AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence"), [Schmidhuber 2015](https://arxiv.org/abs/1511.09249#schmidhuber "On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models")/[2018](https://arxiv.org/abs/1802.08864#schmidhuber "One Big Net for Everything"), [Weng 2018](https://lilianweng.github.io/lil-log/2018/11/30/meta-learning.html#openai "Meta-Learning: Learning to Learn Fast")/[Weng 2019](https://lilianweng.github.io/lil-log/2019/06/23/meta-reinforcement-learning.html#openai "Meta Reinforcement Learning").], forcing it to do more than just learn ordinary textual relationships.
Like OpenAI's [Jukebox](https://openai.com/blog/jukebox/ "'Jukebox: We're introducing Jukebox, a neural net that generates music, including rudimentary singing, as raw audio in a variety of genres and artist styles. We're releasing the model weights and code, along with a tool to explore the generated samples.', Dhariwal et al 2020") just weeks ago (itself a remarkable demonstration of scaling in synthesizing *raw audio* music complete with remarkably realistic voices/instruments), the announcement of GPT-3 appears to have sunk almost without a trace, so I will go into more depth than usual.

# Flexing GPT

> '"They are absolutely reasonable. I think that is their distinguishing characteristic. Yes, Mr. Erskine, an absolutely reasonable people. I assure you there is no nonsense about the Americans." "How dreadful!" cried Lord Henry. "I can stand brute force, but brute reason is quite unbearable. There is something unfair about its use. It is hitting below the intellect."'
>
> _The Picture of Dorian Gray_, Oscar Wilde

^[!Margin: "Attacks only get better."] 2 years ago, [GPT-1](https://openai.com/blog/language-unsupervised/ "Improving Language Understanding with Unsupervised Learning: We've obtained state-of-the-art results on a suite of diverse language tasks with a scalable, task-agnostic system, which we're also releasing. Our approach is a combination of two existing ideas: transformers and unsupervised pre-training. These results provide a convincing example that pairing supervised learning methods with unsupervised pre-training works very well; this is an idea that many have explored in the past, and we hope our result motivates further research into applying this idea on larger and more diverse datasets.") was interestingly useful pretraining and adorable with its "sentiment neuron".
1 year ago, GPT-2 was impressive with its excellent text generation & finetuning capabilities.
This year, GPT-3 is scary because it's a magnificently obsolete architecture from early 2018 (used mostly for software engineering convenience as the infrastructure has been debugged), which is small & shallow compared to what's possible[^overhang][^overhang-NN], with a simple uniform architecture^[Eg a narrow context window [severely limits it](https://arxiv.org/pdf/2001.08361.pdf#page=25 "D.5: Context Dependence"), and motivates the need for [efficient attention](/notes/Attention "'Efficient Attention: Breaking The Quadratic Transformer Bottleneck', Branwen 2020"). More broadly, GPT-3 does nothing exotic---no use of [brain imitation learning](https://old.reddit.com/r/reinforcementlearning/comments/9pwy2f/wbe_and_drl_a_middle_way_of_imitation_learning/ "'WBE and DRL: a Middle Way of imitation learning from the human brain', Branwen 2018") or neural architecture search to try to tailor the model, online hyperparameter optimization (possibly [>3× speedup](https://arxiv.org/abs/2106.00958#openai "'A Generalizable Approach to Learning Optimizers', Almeida et al 2021")) or even decide basic hyperparameters like widths (which as [EfficientNet](https://arxiv.org/abs/1905.11946#google "'EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks', Tan & Le 2019"){#tan-le-2019-2} shows, can make quite a different even in "well-understood and hand-optimized vanilla architectures").] trained in the dumbest way possible (unidirectional prediction of next text token) on a single impoverished modality (random Internet HTML text dumps^[Not even PDFs---so no Google Books, no Arxiv, no Libgen, no Sci-Hub...]) on tiny data (fits on a laptop), sampled in a dumb way^[Generating text from a LM can reveal the presence of knowledge, but not its absence, and it is universally agreed that the current crude heuristic methods like top-_k_ cannot possibly be optimal.], its benchmark performance sabotaged by bad prompts & [data encoding problems](/GPT-3#bpes "'GPT-3 Creative Fiction § BPEs', Branwen 2020") (especially arithmetic & commonsense reasoning), and yet, the first version already manifests crazy runtime meta-learning---and the scaling curves *still* are not bending!
The samples are also better than ever, whether it's GPT-3 inventing new penis jokes^['A man is at the doctor's office, and the doctor tells him, "I've got some good news and some bad news for you." / The man says, "Well, I can't take the bad news right now, so give me the good news first." / The doctor says, "Well, the good news is that you have an 18-inch penis." / The man looks stunned for a moment, and then asks, "What's the bad news?" / The doctor says, "Your brain's in your dick."'] or writing (mostly working) [JavaScript tutorials](https://justpaste.it/7eovk#javascript "'GPT-3 random sample dump: JavaScript tutorial', GPT- 2020") about rotating arrays.

[^overhang]: GPT-3 hardly costs more than a few million dollars of compute (as of early 2020) as the extensive scaling research beforehand enabled one training run, and it is cheap to run (pg39): "Even with the full GPT-3 175B, generating 100 pages of content from a trained model can cost on the order of 0.4 kW-hr, or only a few cents in energy costs." (Likewise, T5 was trained [only once](https://nitter.hu/colinraffel/status/1313097438299910147 "I recently came across https://arxiv.org/abs/2004.08900, which 'assumes 2-3 runs' of T5-11B. In fact, we trained T5-11B once. That's why we spend 35 pages figuring out how we should train before we start training. You don't want to mess up a training run that big.").) And for the cost of one model, GPT-3 API users have shown that you get the equivalent of hundreds of smaller special-purpose models, each requiring more researchers, custom datasets, countless training runs, and tinkering, assuming said models could be created at all. (A slogan for the future: "One model, one vector---once.")

    For comparison, the [PDP-11](!W) was a common academic workhorse due to its extremely low cost, a mere [$20,000]($1970), while the first [Lisp Machine](!W) cost >[$50,000]($1972)---expensive for a workstation but a bargain compared to researchers hogging mainframes costing tens of millions. IBM's (otherwise useless) Deep Blue AI project reputedly cost >[$5]($1997)m for the final iteration (reports of [$100]($1997)m appear to be a confusion with the estimated value of *publicity* mentioned in pg187 of Hsu's _Behind Deep Blue_) and Big Science projects like [ITER](https://en.wikipedia.org/wiki/ITER) blow >5000× the funding to mostly fail. (The particle physicists, incidentally, are [back asking for](https://www.nature.com/articles/d41586-020-01866-9 "CERN makes bold push to build €21-billion supercollider: European particle-physics lab will pursue a 100-kilometer machine to uncover the Higgs boson's secrets---but it doesn't yet have the funds") ≫[$24]($2020)b, based on, presumably the scientific revolutions & world-changing breakthroughs that the LHC's >[$9]($2010)b investment produced, or the [$2]($1993)b spent to (not) build the [SSC](!W "Superconducting Super Collider")...)

    GPT-3 could have been done decades ago with global computing resources & scientific budgets; what could be done with today's hardware & budgets that we just don't know or care to do? There *is* a hardware overhang. (See also the [_Whole Brain Emulation Roadmap_](/docs/ai/scaling/hardware/2008-sandberg-wholebrainemulationroadmap.pdf "Sandberg & Bostrom 2008") & ["2019 recent trends in GPU price per FLOPS"](https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-flops/).)
[^overhang-NN]: Further, NNs have additional hardware overhangs of their own due to the many orders of magnitude asymmetry of training vs running.  Transfer learning and meta-learning are so much faster than the baseline model training. You can 'train' GPT-3 without even any gradient steps---just examples. You pay the extremely steep upfront cost of One Big Model to Rule Them All, and then reuse it everywhere at tiny marginal cost. If you train a model, then as soon as it's done you get, among other things:

    - the ability to run thousands of copies in parallel on the same hardware

        - in a context like AlphaGo, I estimate several hundred ELO strength gains if you reuse the same hardware to merely run tree search with exact copies of the original model
    - meta-learning/transfer-learning to any related domain, cutting training requirements by orders of magnitude
    - model compression/distillation to train student models which are a fraction of the size, FLOPS, or latency (ratios varying widely based on task, approach, domain, acceptable performance degradation, targeted hardware etc, but often extreme like 1⁄100^th^)
    - reuse of the model elsewhere to instantly power up other models (eg. use of text or image embeddings for a DRL agent)
    - learning-by-doing/[experience curve effects](https://en.wikipedia.org/wiki/Experience_curve_effects) (highest in information technologies, and high for DL: Hernandez & Brown 2020), so the next from-scratch model may be much cheaper.

        For example: after all the iterative model architecture & game upgrades done while training the first [OpenAI Five](!W) (OA5) DoTA2 agent was completed, the second iteration of OA5, ["Rerun"](https://arxiv.org/pdf/1912.06680.pdf#page=11&org=openai "'Dota 2 with Large Scale Deep Reinforcement Learning', Berner et al 2019: Section 4.2: Validating Surgery with Rerun"), was trained from scratch. Rerun required only 20% of the training for a "98% win-rate against the final version of OpenAI Five."
        As the authors note: "The ideal option would be to run Rerun-like training from the very start, but this is impossible---the OpenAI Five curve represents lessons learned that led to the final codebase, environment, etc., without which it would not be possible to train Rerun."
    - baseline for engineering much more efficient ones by ablating and comparing with the original

It's odd that this qualitative leap appears to be largely missed by the standard NLP benchmarks.
Nothing in the raw metrics reported on, say, Penn Tree Bank or LAMBADA or WinoGrande would lead you to expect all of this hilarious and creative output; the meta-learning results might, but only if you already thought meta-learning was important.
This suggests to me that a useful post-GPT-3 contribution would be figuring out how to benchmark these sorts of flexible text generation capabilities (possibly something along the lines of Chollet's image-based [Abstraction and Reasoning Corpus (ARC)](https://arxiv.org/abs/1911.01547#google "'On the Measure of Intelligence', Chollet 2019")).

# Baking The Cake

![Is GPT actually part of AGI---or is the cake a lie? ([LeCun 2019](/docs/ai/scaling/2019-02-18-lecun-isscc-talk-deeplearninghardwarepastpresentandfuture.pdf#page=60 "Deep Learning Hardware: Past, Present, & Future: slide 60: 'How Much Information is the Machine Given during Learning?'"))](/images/ai/2019-lecun-isscctalk-cake.png){.float-right .invert}

^[!Margin: Not the whole picture, but a big part.] Does it set SOTA on every task? No, of course not.
But the question is not whether we can lawyerly find any way in which it might not work, but [whether there is any way which it might work](/Forking-Paths "'Technology Forecasting: The Garden of Forking Paths', Branwen 2014").
And there are many ways it might work better (see the ["Limitations" section](https://arxiv.org/pdf/2005.14165.pdf&org=openai#page=34 "GPT-3: Language Models are Few-Shot Learners: 5. Limitations") for just a few).
Does GPT-3 *do* anything like steer a robot around SF shooting lasers and rockets at humans⸮ No, of course not.
It is 'just' a text prediction model, an idiot savant of text; but an idiot savant, we should remember, is only a genetic mutation or bit of brain damage away from a normal human.
If RL is the cherry on the top of the supervised learning frosting, and supervised learning is the frosting on top of the unsupervised learning cake, well, it looks like the cake layers are finally rising.

![A better GPT-3 lesson.](/images/ai/gpt/2020-07-24-meme-moneyprinter-bitterlesson-gpt3.png "GPT-3's implications in the 'money printer go brr' meme format: the head of Rich Sutton says 'GPUs go bitter', referencing his 'bitter lesson' that most clever AI innovations are ultimately useless as they hamstring AI performance and are surpassed by methods that make fewer assumptions & use more compute/data, while the personification of AI academia, where cleverness is rewarded and heavy use of compute is considered cheating and ugly, sheds tears and complains about approaches like GPT-3 beating decades of clever academic systems."){.float-left}

^[!Margin: Scaling still working.] I was surprised, as I had expected closer to 100b parameters, and I thought that the performance of [CTRL](https://arxiv.org/abs/1909.05858#salesforce "'CTRL: A Conditional Transformer Language Model for Controllable Generation', Keskar et al 2019")/[Meena](https://arxiv.org/abs/2001.09977#google "'Towards a Human-like Open-Domain Chatbot', Adiwardana et al 2020")/[MegatronLM](https://nv-adlr.github.io/MegatronLM "MegatronLM: Training Billion+ Parameter Language Models Using GPU Model Parallelism")/[T5](https://arxiv.org/abs/1910.10683#google "'T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer', Raffel et al 2019")/[Turing-NLG](https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/ "'Turing-NLG: A 17-billion-parameter language model by Microsoft', Rosset 2020")/[GPipe](https://arxiv.org/abs/1811.06965#google "'GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism', Huang et al 2018") suggested that, [the scaling papers](/notes/Scaling "'Machine Learning Scaling', Branwen 2021")[^scaling-papers] notwithstanding, the scaling curves had started to bend and by 100b, it might be hard to justify further scaling.
However, in the latest version of ["the unreasonable effectiveness of data"](/docs/ai/scaling/2009-halevy.pdf "'The Unreasonable Effectiveness of Data', Halevy et al 2009") where "the curves cross"/"scissor effect" and the neural method eventually wins (eg. [Banko & Brill 2001](/docs/ai/scaling/2001-banko.pdf#microsoft "Scaling to Very Very Large Corpora for Natural Language Disambiguation"), [Brants et al 2007](/docs/ai/scaling/2007-brants.pdf#google "Large Language Models in Machine Translation"), [Koehn & Knowles 2017](/images/ai/2017-koehn-figure3-bleuscoreswithvaryingamountsoftrainingdata.png "Six Challenges for Neural Machine Translation: Challenges: 3.2. Amount of Training Data: Figure 3: BLEU scores for English-Spanish systems trained on 0.4 million to 385.7 million words of parallel data. Quality for NMT starts much lower, outperforms SMT at about 15 million words, and even beats a SMT system with a big 2 billion word in-domain language model under high-resource conditions.")), GPT-3 hits twice that without noticeable change in scaling factors: its scaling continues to be roughly logarithmic/power-law, as it was for much smaller models & as forecast, and it has not hit a regime where gains effectively halt or start to require increases vastly beyond feasibility.
That suggests that it would be both possible and useful to head to trillions of parameters (which are still well within available compute & budgets, requiring merely thousands of GPUs & perhaps [$10]($2020)--[$100]($2020)m budgets assuming no improvements which of course there will be, see Hernandez & Brown 2020 etc in this issue), and eyeballing the graphs, many benchmarks like the [Winograd schema](https://en.wikipedia.org/wiki/Winograd_Schema_Challenge) [WinoGrande](https://arxiv.org/abs/1907.10641#allen "'WinoGrande: An Adversarial Winograd Schema Challenge at Scale', Sakaguchi et al 2019") would fall by 10t parameters.
The predictability of scaling is striking, and makes scaling models more like statistics than AI.
(AI is statistics which does what we want it to but doesn't work; and statistics is AI which works but doesn't do what we want.)

[^scaling-papers]:
In particular, sample-efficiency increases with model size up to compute-efficient scaling, and [GPT-2 can memorize data after seeing it only once](https://arxiv.org/abs/2012.07805 "'Extracting Training Data from Large Language Models', Carlini et al 2020")---a [desirable property](https://arxiv.org/abs/1906.05271#google "'Does Learning Require Memorization? A Short Tale about a Long Tail', Feldman 2019") given long-tailed real-world distributions of data. (An example of how *not* to do scaling papers is [Thompson et al 2020](https://arxiv.org/abs/2007.05558 "The Computational Limits of Deep Learning"), which, in stark contrast to the foregoing papers---which Thompson et al do not mention at all!---attempts to infer scaling not from well-controlled experiments run by the authors, which yield extremely tight and highly predictive curves, but attempts to infer them from occasional reported numbers in highly disparate research papers; unsurprisingly, their curves barely predict anything and seem to be serious overestimates anyway.)

    It is noteworthy that the pursuit of large models is driven almost exclusively by OpenAI & industry entities (the latter of which are content with far smaller models), and that academia has evinced an almost total disinterest---disgust & anger, even, and denial (one might say "green AI" is green with envy). For all that the scaling hypothesis is 'obvious' and scaling is 'predicted', there is remarkably little interest in actually *doing* it. Perhaps we should pay more attention to what people do rather than what they say.

    For more ML scaling research, follow the [/r/MLScaling](https://old.reddit.com/r/mlscaling/ "'ML Scaling subreddit', Branwen 2020") subreddit.

![GPT-3: not even that much compute---[3640 petaflop/s-day](https://arxiv.org/pdf/2005.14165.pdf#org=openai&page=46 "Total Compute Used to Train Language Model: Table D.1"), only 2× their estimate for AlphaGo Zero, 1860. (Historical graph modified by myself from ["AI and Compute", Amodei et al 2018](https://openai.com/blog/ai-and-compute/ "We're releasing an analysis showing that since 2012, the amount of compute used in the largest AI training runs has been increasing exponentially with a 3.4-month doubling time (by comparison, Moore's Law had a 2-year doubling period). Since 2012, this metric has grown by more than 300,000× (a 2-year doubling period would yield only a 7× increase). Improvements in compute have been a key component of AI progress, so as long as this trend continues, it's worth preparing for the implications of systems far outside today's capabilities.").)](/images/ai/gpt/2019-11-07-amodei-aiandcompute-twodistincteras-gpt3modified.png)

^[!Margin: Anti-scaling: penny-wise, pound-foolish.] GPT-3 is an extraordinarily expensive model by the standards of machine learning: it is estimated that training it may require the annual cost of more machine learning researchers than you can count on one hand (~[$5]($2020)m^[Roughly around [Chuan Li's](https://lambdalabs.com/blog/demystifying-gpt-3/ "OpenAI's GPT-3 Language Model: A Technical Overview") estimate, using nominal list prices without discounts (which could be steep as the marginal costs of cloud compute are substantially lower). The R&D project cost would be much higher, but is amortized over all subsequent models & projects.]), up to [$30]($2020) of hard drive space to store the model (500--800GB), and multiple pennies of electricity per 100 pages of output (0.4 kWH).
Researchers are concerned about the prospects for scaling: can ML afford to run projects which cost more than 0.1 milli-Manhattan-Projects⸮^[The Manhattan Project cost ~[$2]($1946)b.]
Surely it would be too expensive, even if it represented another large leap in AI capabilities, to spend up to 10 milli-Manhattan-Projects to scale GPT-3 100× to a trivial thing like human-like performance in many domains⸮
Many researchers feel that such a suggestion is absurd and refutes the entire idea of scaling machine learning research further, and that the field would be more productive if it instead focused on research which can be conducted by an impoverished goatherder on an old laptop running off solar panels.^[As if we live in a world where grad students could go to the Moon on a ramen budget if we just wished hard enough, as if focusing on CO~2~ costs & not benefits in our evaluations is not like making a scissor with only one blade, or as if "green AI" approaches to try to create small models without going through big models did not look increasingly futile and like throwing good money after bad, and were not the least green of all AI research... To the extent that all cutting-edge AI research ~2010 could be done with grad student money like [$1000]($2010) of hardware, where AI research in decades before & after benefited from big iron, that is an indictment of that era, demonstrating what a stagnant dead end that research was, that its techniques were so smallminded and hobbled it could not benefit from the available large-scale compute.]
Nonetheless, I think we can expect further scaling.
(10×? No, 10× isn't cool. You know what's cool? [100--1000×](https://old.reddit.com/r/slatestarcodex/comments/hys565/are_we_in_an_ai_overhang/fzezi7d/ "People I know at OpenAI say v4 is around the corner and easily doable, and...will be here soon (not months but year or so). And they are confident it will scale and be around 100--1000×."), trained on a [fancy new supercomputer](https://blogs.microsoft.com/ai/openai-azure-supercomputer/ "Microsoft announces new supercomputer, lays out vision for future AI work").)

# Scaling

^[!Margin: How far will scaling go?] The scaling papers suggest that the leaps we have seen over the past few years are not even half way there in terms of absolute likelihood loss, never mind what real-world capabilities each additional decrement translates into.
The scaling curves are clean; from ["Scaling Laws for Neural Language Models", Kaplan et al 2020](https://arxiv.org/abs/2001.08361#openai "'Scaling Laws for Neural Language Models', Kaplan et al 2020"):

![DL scaling laws: compute, data, model parameters. ([Figure 1](https://arxiv.org/pdf/2001.08361.pdf#page=3&org=openai "Scaling Laws for Neural Language Models: Figure 1: Language modeling performance improves smoothly as we increase the model size, dataset size, and amount of compute used for training."))](/images/ai/gpt/2020-kaplan-figure1-dlscaling.png "Figure 1: Language modeling performance improves smoothly as we increase the model size, dataset size, and amount of compute used for training. For optimal performance all three factors must be scaled up in tandem. Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two. (Kaplan et al 2020)"){.invert}

GPT-3 represents ~10^3^ on this chart, leaving plenty of room for further loss decreases---especially given the [uncertainty in extrapolation](https://arxiv.org/pdf/2001.08361.pdf#page=17&org=openai "'Scaling Laws for Neural Language Models: Figure 15: Far beyond the model sizes we study empirically, we find a contradiction between our equations', Kaplan et al 2020"):

![Projecting DL power laws: still room beyond GPT-3.](/images/ai/gpt/2020-kaplan-figure15-projectingscaling.png "Figure 15: Far beyond the model sizes we study empirically, we find a contradiction between our equations for _L(C~min~)_ and _L(D)_ due to the slow growth of data needed for compute-efficient training. The intersection marks the point before which we expect our predictions to break down. The location of this point is highly sensitive to the precise exponents from our power-law fits. (Kaplan et al 2020)"){.invert}

Lo and behold, the scaling laws continue for GPT-3 models for several orders past [Kaplan et al 2020](#kaplan-et-al-2020); from [Brown et al 2020](https://arxiv.org/pdf/2005.14165.pdf#page=11&org=openai "GPT-3: Language Models are Few-Shot Learners: Figure 3.1: Smooth scaling of performance with compute"):

![GPT-3 continues to scale as predicted. (Note GPT-3's curve has not 'bounced', and it trained only ~0.5 epoches, see [Table 2.2](https://arxiv.org/pdf/2005.14165.pdf#org=openai&page=9 "Table 2.2: Datasets used to train GPT-3. 'Weight in training mix' refers to the fraction of examples during training that are drawn from a given dataset, which we intentionally do not make proportional to the size of the dataset. As a result, when we train for 300 billion tokens, some datasets are seen up to 3.4 times during training while other datasets are seen less than once."))](/images/ai/gpt/2020-brown-figure31-gpt3scaling.png "Brown et al 2020: Figure 3.1: Smooth scaling of performance with compute. Performance (measured in terms of cross-entropy validation loss) follows a power-law trend with the amount of compute used for training. The power-law behavior observed in Kaplan et al 2020 continues for an additional two orders of magnitude with only small deviations from the predicted curve. For this figure, we exclude embedding parameters from compute and parameter counts. (Brown et al 2020). Cross-validation loss extrapolation: $L(oss) = 2.57 · C(ompute in petaflop-s/days) ^ −0.048$"){.invert}

If we see such striking gains in halving the validation loss but with so far left to go, what is left to emerge as we third or halve again?
How far does this go, exactly? How do we predict what emerges when?
Bueller? Bueller?
(See also [Meena's perplexity vs human-ness chatbot ratings](/images/ai/2020-adiwardana-meena-figure1-humanratingsvslikelihood.png "Towards a Human-like Open-Domain Chatbot, Adiwardana et al 2020: Figure 1: Interactive SSA vs Perplexity [exp(cross-entropy loss)]. Each point is a different version of the Meena model. A regression line is plotted, for which the coefficient of determination (R^2) is 0.93, an indication of strong correlation between perplexity and the human evaluation metric (SSA). The dotted lines show the SSA performance of other chatbots, humans (86%), the best end-to-end trained Meena model (72%), and the full version of Meena which incorporates a filtering mechanism and tuned decoding (Section 5) and scores 79%. Mitsuku and Cleverbot scored the same on overall SSA, but Mitsuku displayed higher sensibleness, whereas Cleverbot had higher specificity. See Sections 2.5, 2.6, and 4.3 for more details on how we performed these comparisons and how to interpret the results"){.invert}, GPT-3-written news articles' [probability of fooling humans by parameter count](/images/ai/gpt/2020-brown-figure313-humanabilitytodetectmodelgeneratednewsstories.png "Brown et al 2020: Figure 3.13: People's ability to identify whether news articles are model-generated (measured by the ratio of correct assignments to non-neutral assignments) decreases as model size increases. Accuracy on the outputs on the deliberately-bad control model (an unconditioned GPT-3 Small model with higher output randomness) is indicated with the dashed line at the top, and the random chance (50%) is indicated with the dashed line at the bottom. Line of best fit is a power law with 95% confidence intervals."), and [GPT-3 model size vs Q&A](/images/ai/gpt/2020-hendrycks-figure1b-gpt3-qascaling.png "Figure 1b: GPT-3 Few Shot Test Performance: Performance on a commonsense benchmark (HellaSwag), a linguistic understanding benchmark (SuperGLUE), and the massive multitask test. On previous benchmarks, smaller models start well above random chance levels and exhibit more continuous improvements with model size increases, but on our test, GPT-3 moves beyond random chance with the largest model."){.invert} from [Hendrycks et al 2020](https://arxiv.org/abs/2009.03300 "Measuring Massive Multitask Language Understanding").)

## Blessings Of Scale

> "Extrapolating the spectacular performance of GPT-3 into the future suggests that the answer to life, the universe and everything is just 4.398 trillion parameters."
>
> [Geoff Hinton](https://nitter.hu/geoffreyhinton/status/1270814602931187715)

^[!Margin: We don't know how to train NNs.] The *blessings of scale* is the observation that for deep learning, hard problems are easier to solve than easy problems---everything gets better as it gets larger (in contrast to the usual outcome in research, where small things are hard and large things impossible).
The bigger the neural net/compute/data/problem, the faster it learns, the better it learns, the stabler it learns, and so on.
A problem we can't solve at all at small _n_ may suddenly become straightforward with millions or billions of _n_.
"NNs are lazy": they can do far more than we make them do when we push them beyond easy answers & cheap shortcuts.
The [bitter lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html "'The Bitter Lesson', Sutton 2019") is the harder and bigger, the better.
(Besides GPT-3, one could mention recent progress in semi-supervised learning & the model-based DRL renaissance.)

![AlphaGo Zero: 'just stack moar layers lol!'](/images/reinforcement-learning/2017-12-24-meme-nnlayers-alphagozero.jpg "Humorous description of the simplicity of the AlphaGo Zero architecture compared to AlphaGo Master"){.float-right}

^[!Margin: Blessings of scale: stability → generalization → meta-learning.] GPT-3 is hamstrung by its training & data, but DL enjoys an unreasonably effective [blessing of dimensionality](!W)---just simply training a *big* model on a *lot* of data induces better properties like meta-learning without even the slightest bit of that architecture being built in; and in general, training on more and harder tasks creates ever more human-like performance, generalization, and robustness.
The GPT natural-language & programming language models, [iGPT](https://openai.com/blog/image-gpt/ "Image GPT: We find that, just as a large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples. By establishing a correlation between sample quality and image classification accuracy, we show that our best generative model also contains features competitive with top convolutional nets in the unsupervised setting.")/[Vision Transformer](https://openreview.net/forum?id=YicbFdNTTy#google "Vision Transformer (ViT): An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale") for images (and to some degree [GPT-f](https://arxiv.org/abs/2009.03393#openai "'GPT-f: Generative Language Modeling for Automated Theorem Proving', Polu & Sutskever 2020")), show that simply scaling up models & datasets without any supervision produces results competitive with the best (and most complex) alternatives, using the same simple architecture, gradually passing from superficial surface correlations to more human-like brain activity ([Schrimpf et al 2020](https://www.biorxiv.org/content/10.1101/2020.06.26.174482.full "The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing")) and linguistic biases as data increases (eg. [Warstadt et al 2020](https://arxiv.org/abs/2010.05358 "Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)")).
In fact, one may not even need complicated attention mechanisms at scale, as fully-connected networks---hard to get much simpler than them!---[work surprisingly well](/notes/FC "'Fully-Connected Neural Nets', Branwen 2021") for many tasks.
One typically trains such large models with simple optimizers like Adam---because the complicated ones lose their advantages as batch sizes increase and [the simple optimizers work fine](https://arxiv.org/abs/2102.06356 "'A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes', Nado et al 2021") and are more memory-efficient anyway.
[OA5](https://arxiv.org/pdf/1912.06680.pdf&org=openai#page=13 "'Dota 2 with Large Scale Deep Reinforcement Learning: Section 4.3: Batch Size', Berner et al 2019") does not just scale to, but [stabilizes at](#PPO-DoTA2), minibatches of millions due to [gradient noise](https://openai.com/blog/science-of-ai/ "How AI Training Scales: We've discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training on a wide range of tasks. Since complex tasks tend to have noisier gradients, increasingly large batch sizes are likely to become useful in the future, removing one potential limit to further growth of AI systems. More broadly, these results show that neural network training need not be considered a mysterious art, but can be rigorized and systematized. ['An Empirical Model of Large-Batch Training', McCandlish et al 2018]").
OA5-like, [BigGAN](https://arxiv.org/pdf/1809.11096.pdf#page=8&org=deepmind "'BigGAN: Large Scale GAN Training For High Fidelity Natural Image Synthesis: 5.2 Additional Evaluation On JFT-300M', Brock et al 2018") stabilizes at large-scale image datasets like JFT-300M & benefits from unusually large minibatches and VAEs (long an also-ran to GANs or autoregressive models in terms of sharp image generation) catch up if you make them very deep ([Child 2020](https://arxiv.org/abs/2011.10650#openai "VDVAE: Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images"), [Vahdat & Kautz 2020](https://arxiv.org/abs/2007.03898#nvidia "NVAE: A Deep Hierarchical Variational Autoencoder")); while classifier CNNs like [BiT](https://arxiv.org/abs/1912.11370#google "'Big Transfer (BiT): Large Scale Learning of General Visual Representations for Transfer', Kolesnikov et al 2019")^[Fun trivia: BiT [is now more accurate](https://arxiv.org/abs/2006.07159#google "'Are we done with ImageNet?', Beyer et al 2020") at predicting (cleaned, corrected) ImageNet labels than the original ImageNet labels are.]/[Dojolonga et al 2020](https://arxiv.org/abs/2007.08558#google "On Robustness and Transferability of Convolutional Neural Networks") or [ResNeXt](https://arxiv.org/abs/1907.07640 "'Robustness properties of Facebook's ResNeXt WSL models', Orhan 2019") or [Noisy Student](https://arxiv.org/abs/1911.04252#google "'Self-training with Noisy Student improves ImageNet classification', Xie et al 2019") transfer & [robustify](https://arxiv.org/abs/2007.00644 "'Measuring Robustness to Natural Distribution Shifts in Image Classification', Taori et al 2020") [with](https://arxiv.org/abs/2103.14586#google "'Understanding Robustness of Transformers for Image Classification', Bhojanapalli et al 2021") human-like errors^[One interesting aspect of image scaling experiments like Dojolonga et al 2020 is that even when performance is 'plateauing' on the original task & approaching label error, the transfer learning continues to improve. Apparently the internal representations, even when adequate for mere classification and so the score cannot increase more than a small percentage, become more human-like---because it's encoding [dark knowledge](https://arxiv.org/abs/1503.02531#google "'Distilling the Knowledge in a Neural Network', Hinton et al 2015") or more [adversarial robustness](https://arxiv.org/abs/2006.14536#google "'Smooth Adversarial Training', Xie et al 2020")? I've noticed with language models, the final fractions of a loss appear to make a substantial difference to generated sample quality, perhaps because it is only after all the easier modeling is finished that the lazy language model is forced to squeeze out the next bit of performance by more correctly modeling more sophisticated things like logic, objects, world-knowledge, etc.], multimodal learning produces better representations on fewer data (eg. [ViLBERT](https://arxiv.org/abs/1912.02315#facebook "'12-in-1: Multi-Task Vision and Language Representation Learning', Lu et al 2019")/[VideoBERT](https://arxiv.org/abs/1904.01766#google "'VideoBERT: A Joint Model for Video and Language Representation Learning', Sun et al 2019"), motivating [OA's interest in big multimodal models](https://www.technologyreview.com/2020/02/17/844721/ai-openai-moonshot-elon-musk-sam-altman-greg-brockman-messy-secretive-reality/ "The messy, secretive reality behind OpenAI's bid to save the world")), and RNNs can [predict videos](https://arxiv.org/abs/1911.01655#google "'High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks', Villegas et al 2019").
[AlphaStar](/docs/reinforcement-learning/model-free/alphastar/2019-vinyals.pdf#deepmind "'Grandmaster level in StarCraft II using multi-agent reinforcement learning', Vinyals et al 2019") reaches human-level with hundreds of competing self-players to cover possible strategies.
Imitation learning DRL like [MetaMimic](https://arxiv.org/abs/1810.05017#deepmind "'MetaMimic: One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL', Le Paine et al 2018") generalizes at hundreds of tasks to train a deep net.
Disentanglement emerges in [StyleGAN](https://arxiv.org/abs/1812.04948#nvidia "'A Style-Based Generator Architecture for Generative Adversarial Networks', Karras et al 2018") with sufficiently deep _w_ embeddings, with enough parameters to train raw audio in the aforementioned Jukebox, or in [relational networks](https://arxiv.org/abs/1706.01427#deepmind "'A simple neural network module for relational reasoning', Santoro et al 2017")/[GQN](/docs/reinforcement-learning/model/2018-eslami.pdf#deepmind "'Neural scene representation and rendering', Eslami et al 2018")/[Transformers](https://arxiv.org/abs/2002.05867 "'Transformers as Soft Reasoners over Language', Clark et al 2020") with enough samples to force factorization.
(See also [Hill et al 2019](https://arxiv.org/abs/1910.00571#deepmind "Environmental drivers of systematicity and generalization in a situated agent")/[Chaplot et al 2017](https://arxiv.org/abs/1706.07230 "Gated-Attention Architectures for Task-Oriented Language Grounding")/[Yu et al 2018](https://arxiv.org/abs/1802.01433#baidu "Interactive Grounded Language Acquisition and Generalization in a 2D World")/[Lake 2019](https://arxiv.org/abs/1906.05381 "Compositional generalization through meta sequence-to-sequence learning")/[Interactive Agents Group 2020](https://arxiv.org/abs/2012.05672#deepmind "Imitating Interactive Intelligence").)
Training [Dactyl](https://arxiv.org/abs/1910.07113#openai "'Solving Rubik's Cube With A Robot Hand', Akkaya et al 2019") on millions of domain randomizations induced similar implicit meta-learning where during each runtime invocation, the RNN probes its environment and encodes its understanding of robot hand control into its hidden state; and [DD-PPO](https://arxiv.org/abs/1911.00357#facebook "'DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames', Wijmans et al 2019") outperforms classical robot planners by scaling 2 orders.
Or in [Procgen](https://openai.com/blog/procgen-benchmark/ "Procgen Benchmark: We're releasing Procgen Benchmark, 16 simple-to-use procedurally-generated environments which provide a direct measure of how quickly a reinforcement learning agent learns generalizable skills") or [CoinRun](https://distill.pub/2020/understanding-rl-vision/#diversity-hypothesis "'Understanding RL Vision', Hilton et al 2020"), training on hundreds of levels trains agents to solve levels individually and worsens performance on other levels, but at thousands of levels, they begin to generalize to unseen levels. (Similarly, [language model pretraining-finetuning](https://arxiv.org/abs/2101.11038#facebook "'Muppet: Massive Multi-task Representations with Pre-Finetuning', Aghajanyan et al 2021") overfits at small numbers of datasets but improves markedly with enough diversity.)
[AlphaZero](/docs/reinforcement-learning/model/alphago/2018-silver.pdf#deepmind "'A general reinforcement learning algorithm that masters chess, shogi and Go through self-play', Silver et al 2018") demonstrated truly superhuman Go without 'delusions' just by training a bigger model on a richer signal & pro-level play without any search---and [MuZero](https://arxiv.org/abs/1911.08265#deepmind "'MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model', Schrittwieser et al 2019"), for that matter, demonstrated that just training an RNN end-to-end to predict a reward on enough data is enough to obsolete even AlphaZero and learn tree search implicitly (but better).
And on and on.
DM researcher [Matthew Botvinick](https://www.lesswrong.com/posts/Wnqua6eQkewL3bqsF/matt-botvinick-on-the-spontaneous-emergence-of-learning "Matt Botvinick on the spontaneous emergence of learning algorithms"), discussing their meta-reinforcement learning work where they were surprised to discover meta-learning emerging, and that it did so regardless of which specific architecture they used:

> "...it's something that just happens. In a sense, you can't avoid this happening. If you have a system that has memory, and the function of that memory is shaped by reinforcement learning, and this system is trained on a series of interrelated tasks, this is going to happen. You can't stop it."

Pace [Breiman](/docs/ai/scaling/1995-breiman.pdf "Reflections After Refereeing Papers for NIPS"), **why**?
Why do they transfer and generalize?
Why do these blessings of scale exist?
Why do we need to train large models when small models provably exist with the same performance?
Why do larger models not overfit (though they [can](https://arxiv.org/abs/1611.03530#google "'Understanding deep learning requires rethinking generalization', Zhang et al 2016")) and generalize better than smaller models?
What's up with the whole ['double descent'](https://openai.com/blog/deep-double-descent/ "Deep Double Descent: We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don't yet fully understand why it happens, and view further study of this phenomenon as an important research direction.") anyway?

These are all, ahem, deep questions about neural networks and heavily debated, but right now, I would suggest that the answer lies in some mix of the model compression/distillation, ['lottery ticket hypothesis'](https://ai.facebook.com/blog/understanding-the-generalization-of-lottery-tickets-in-neural-networks/ "Understanding the generalization of 'lottery tickets' in neural networks"), [Bayesian neural network](https://arxiv.org/abs/2002.08791 "'Bayesian Deep Learning and a Probabilistic Perspective of Generalization', Wilson & Izmailov 2020"), and [learned representation](https://arxiv.org/abs/2007.00810#google "'On Linear Identifiability of Learned Representations', Roeder et al 2020") (like [circuits](https://distill.pub/2020/circuits/zoom-in/#openai "'Zoom In: An Introduction to Circuits: By studying the connections between neurons, we can find meaningful algorithms in the weights of neural networks', Olah et al 2020")) literatures.

Big models work because they encode a dizzyingly vast number of sub-models in an extremely [high-dimensional](https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ "'Neural Networks, Manifolds, and Topology', Olah 2014") abstract space, representing countless small sub-models ([Orseau et al 2020](https://arxiv.org/abs/2006.12156#deepmind "Logarithmic Pruning is All You Need")) [interpolating over data](/docs/ai/scaling/2020-hasson.pdf "'Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks', Hasson et al 2020"), one of which is likely to solve the problem well, and so ensures the problem is soluble by the overall model.
They function as an ensemble: even though there countless overfit sub-models inside the single big model, they all average out, leading to a preference for simple solutions.
This Occam's razor biases the model towards simple solutions which are flexible enough to gradually expand in complexity to match the data.

However, "neural nets are lazy": sub-models which memorize pieces of the data, or latch onto superficial features, learn quickest and are the easiest to represent internally.
If the model & data & compute are not big or varied enough, the optimization, by the end of the cursory training, will have only led to a sub-model which achieves a low loss but missed important pieces of the desired solution.

On the other hand, for a model like GPT-3, it is sufficiently powerful a model that its sub-models can do anything from poetry to arithmetic, and it is trained on so much data that those superficial models may do well early on, but gradually fall behind more abstract models; a sub-model which memorizes some of the data is indeed much simpler than a sub-model which encodes genuine arithmetic (a NN can probably memorize tens of thousands of lookup table entries storing examples of addition in the space it would take to encode an abstract algorithm like 'addition'), but it can't possibly memorize *all* the instances of arithmetic (implicit or explicit) in GPT-3's Internet-scale dataset.
If a memorizing sub-model tried to do so, it would become extremely large and penalized.
Eventually, after enough examples and enough updates, there may be a phase transition ([Viering & Loog 2021](https://arxiv.org/pdf/2103.10948.pdf#page=22 "The Shape of Learning Curves: a Review: 6. Ill-behaved learning curves: 6.1. Phase transitions")), and the simplest 'arithmetic' model which accurately predicts the data just *is* arithmetic.
And then the meta-learning, after seeing enough instances of algorithms which vary slightly within each sample, making it hard to learn each task separately, just *is* learning of more generic algorithms, yielding sub-models which achieve lower loss than the rival sub-models, which either fail to predict well or bloat unacceptably.
(GPT-2-1.5b apparently was too small or shallow to ensemble easily over sub-models encoding meta-learning algorithms, or perhaps not trained long enough on enough data to locate the meta-learner models; GPT-3 was.)

So, the larger the model, the better, if there is enough data & compute to push it past the easy convenient sub-models and into the sub-models which express desirable traits like generalizing, factorizing perception into meaningful latent dimensions, meta-learning tasks based on descriptions, learning causal reasoning & logic, and so on.
If the ingredients are there, it's going to happen.

## Scaling Hypothesis

The strong *scaling hypothesis* is that, once we find a scalable architecture like self-attention or convolutions, which like the brain can be applied fairly uniformly (eg. ["The Brain as a Universal Learning Machine"](https://www.lesswrong.com/posts/9Yc7Pp7szcjPgPsjf/the-brain-as-a-universal-learning-machine) or Hawkins), we can simply train ever larger NNs and ever more sophisticated behavior will emerge naturally as the easiest way to optimize for all the tasks & data.
More powerful NNs are 'just' scaled-up weak NNs, in much the same way that human brains look much like [scaled-up primate brains](/docs/psychology/neuroscience/2012-herculanohouzel.pdf "'The remarkable, yet not extraordinary, human brain as a scaled-up primate brain and its associated cost', Herculano-Houzel 2012").
While I was highly skeptical of scaling hypothesis advocates when I first became interested in AI 2004--2010 (back when AI was stuck in the doldrums of hopelessly narrow tools and dates like 2028 seemed impossibly far away), which smacked of numerology and "if you build it they will come" logic (at the time, we certainly didn't have general algorithms that you could just throw compute at), in 2020, I have to admit, I was wrong and they were right.
We built the compute, and the algorithms *did* come, and the scaling hypothesis has only looked more and more plausible every year since 2010.

# Why Does Pretraining Work?

The pretraining thesis goes something like this:

!["Figure 1: Envisioned evolution of NLP research through three different eras or curves" (the hypothetical S-curves & progress in natural language modeling; from [Cambria & White 2014](/docs/ai/scaling/2014-cambria.pdf "Jumping NLP Curves: A Review of Natural Language Processing Research"))](/images/ai/2014-cambria-figure1-hypotheticalnlpprogresscurves.png){.invert}

Humans, one might say, are the [cyanobacteria of AI](!W "Great Oxidation Event"): we constantly emit large amounts of structured data, which implicitly rely on logic, causality, object permanence, history---all of that good stuff.
All of that is implicit and encoded into our writings and videos and 'data exhaust'.
A model learning to predict must learn to understand all of that to get the best performance; as it predicts the easy things which are mere statistical pattern-matching, what's left are the hard things.
AI critics often say that the long tail of scenarios for tasks like self-driving cars or natural language can only be solved by true generalization & reasoning; it follows then that if models solve the long tail, they must learn to generalize & reason.

Early on in training, a model learns the crudest levels: that some letters like 'e' are more frequent than others like 'z', that every 5 characters or so there is a space, and so on.
It goes from predicted uniformly-distributed bytes to what looks like Base-60 encoding---alphanumeric gibberish.
As crude as this may be, it's enough to make quite a bit of absolute progress: a random predictor needs 8 bits to 'predict' a byte/character, but just by at least matching letter and space frequencies, it can almost halve its error to around 5 bits.^[The numbers here are not exact and are for illustration; because BPEs don't correspond to any intuitive, I am going to borrow from my observations watching char-RNNs, and talk about the loss per character instead of BPE.]
Because it is learning so much from every character, and because the learned frequencies are simple, it can happen so fast that if one is not logging samples frequently, one might not even observe the improvement.

As training progresses, the task becomes more difficult. Now it begins to learn what words actually exist and do not exist. It doesn't know anything about meaning, but at least now when it's asked to predict the second half of a word, it can actually do that to some degree, saving it a few more bits.
This takes a while because any specific instance will show up only occasionally: a word may not appear in a dozen samples, and there are many thousands of words to learn.
With some more work, it has learned that punctuation, pluralization, possessives are all things that exist.
Put that together, and it may have progressed again, all the way down to 3--4 bits error per character!
(While the progress is gratifyingly fast, it's still all gibberish, though, makes no mistake: a sample may be spelled correctly, but it doesn't make even a bit of sense.)

But once a model has learned a good English vocabulary and correct formatting/spelling, what's next? There's not much juice left in predicting within-words.
The next thing is picking up associations among words. What words tend to come first? What words 'cluster' and are often used nearby each other?
Nautical terms tend to get used a lot with each other in sea stories, and likewise Bible passages, or American history Wikipedia article, and so on.
If the word "Jefferson" is the last word, then "Washington" may not be far away, and it should hedge its bets on predicting that 'W' is the next character, and then if it shows up, go all-in on "ashington".
Such bag-of-words approaches still predict badly, but now we're down to perhaps <3 bits per character.

What next? Does it stop there? Not if there is enough data and the earlier stuff like learning English vocab doesn't hem the model in by using up its learning ability.
Gradually, other words like "President" or "general" or "after" begin to show the model subtle correlations: "Jefferson was President after..."
With many such passages, the word "after" begins to serve a use in predicting the next word, and then the use can be broadened.

By this point, the loss is perhaps 2 bits: every additional 0.1 bit decrease comes at a steeper cost and takes more time.
However, now the sentences have started to make sense.
A sentence like "Jefferson was President after Washington" does in fact mean something (and if occasionally we sample "Washington was President after Jefferson", well, what do you expect from such an un-converged model).
Jarring errors will immediately jostle us out of any illusion about the model's understanding, and so training continues.
(Around here, Markov chain & _n_-gram models start to fall behind; they can memorize increasingly large chunks of the training corpus, but they can't solve increasingly critical syntactic tasks like balancing parentheses or quotes, much less start to ascend from syntax to semantics.)

Now training is hard. Even subtler aspects of language must be modeled, such as keeping pronouns consistent.
This is hard in part because the model's errors are becoming rare, and because the relevant pieces of text are increasingly distant and 'long-range'.
As it makes progress, the absolute size of errors shrinks dramatically.
Consider the case of associating names with gender pronouns: the difference between "Janelle ate some ice cream, because he likes sweet things like ice cream" and "Janelle ate some ice cream, because she likes sweet things like ice cream" is one no human could fail to notice, and yet, it is a difference of a single letter.
If we compared two models, one of which didn't understand gender pronouns at all and guessed 'he'/'she' purely at random, and one which understood them perfectly and always guessed 'she', the second model would attain a lower average error of barely <0.02 bits per character!

Nevertheless, as training continues, these problems and more, like imitating genres, get solved, and eventually at a loss of 1--2 (where a small char-RNN might converge on a small corpus like Shakespeare or some Project Gutenberg ebooks), we will finally get samples that sound human---at least, for a few sentences.
These final samples may convince us briefly, but, aside from issues like repetition loops, even with good samples, the errors accumulate: a sample will state that someone is "alive" and then 10 sentences later, use the word "dead", or it will digress into an irrelevant argument instead of the expected next argument, or someone will do something physically improbable, or it may just continue for a while without seeming to *get* anywhere.

All of these errors are far less than <0.02 bits per character; we are now talking not hundredths of bits per characters but less than ten-thousandths.

The pretraining thesis argues that this can go even further: we can compare this performance directly with humans doing the same objective task, who can achieve closer to [0.7 bits per character](/Differences#efficient-natural-languages).
What is in that missing >0.4?

!["Yeah, but there's more to being smart than knowing compression schemes!" "No there's not!" "Shoot---he knows the secret!!"](/images/cs/2004-ryannorth-dinosaurcomics-391.png "https://qwantz.com/index.php?comic=354"){.invert}

Well---*everything*! Everything that the model misses.
While just babbling random words was good enough at the beginning, at the end, it needs to be able to reason our way through the most difficult textual scenarios requiring causality or commonsense reasoning.
Every error where the model predicts that ice cream put in a freezer will "melt" rather than "freeze", every case where the model can't keep straight whether a person is alive or dead, every time that the model chooses a word that doesn't help build somehow towards the ultimate conclusion of an 'essay', every time that it lacks the theory of mind to compress novel scenes describing the Machiavellian scheming of a dozen individuals at dinner jockeying for power as they talk, every use of logic or abstraction or instructions or Q&A where the model is befuddled and needs more bits to cover up for its mistake where a human would think, understand, and predict.
Each of these cognitive breakthroughs allows ever so slightly better prediction of a few relevant texts; nothing less than true understanding will suffice for ideal prediction.

If we trained a model which reached that loss of <0.7, which could predict text indistinguishable from a human, whether in a dialogue or quizzed about ice cream or being tested on SAT analogies or tutored in mathematics, if for every string the model did just as good a job of predicting the next character as you could do, how could we say that it doesn't *truly* understand everything?
(If nothing else, we could, by definition, replace humans in any kind of text-writing job!)

^[!Margin: The last bits are deepest.] The implication here is that the final few bits are the most valuable bits, which require the most of what we think of as intelligence.
A helpful analogy here might be our actions: for the most part, all humans execute actions equally well.
We all pick up a tea mug without dropping, and can lift our legs to walk down thousands of steps without falling even once.
For everyday actions (the sort which make up most of a corpus), anybody, of any intelligence, can get enough practice & feedback to do them quite well, learning individual algorithms to solve each class of problems extremely well, in isolation.^[If you see thousands of images labeled 'dog' and thousands more labeled 'cat', you can simply learn separate dog & cat classifiers without bothering to understand their shared aspects like being domesticated quadruped mammal predators. This won't be useful if you are then asked to classify 'ferret' images, but you weren't asked to, so that's not your problem, since you can just learn yet another separate classifier for ferrets if you then get a lot of ferret images.]
Meanwhile for rare problems, there may be too few instances to do any better than memorize the answer.
In the middle of the spectrum are problems which are similar but not *too* similar to other problems; these are the sorts of problem which reward flexible meta-learning and generalization, and many intermediate problems may be necessary to [elicit those capabilities](https://arxiv.org/abs/2205.05055#deepmind "'Data Distributional Properties Drive Emergent Few-Shot Learning in Transformers', Chan et al 2022") ("neural nets are lazy").

Where individuals differ is when they start running into the long tail of novel choices, rare choices, choices that take seconds but unfold over a lifetime, choices where we will never get any feedback (like after our death).
One only has to make a single bad decision, out of a lifetime of millions of discrete decisions, to wind up in jail or dead.
A small absolute average improvement in decision quality, if it is in *those* decisions, may be far more important than its quantity indicates, and give us some intutition for why those last bits are the hardest/deepest.
(Why do humans have such large brains, when animals like chimpanzees do so many ordinary activities seemingly as well with a fraction of the expense? Why is language worthwhile? Perhaps because of considerations like these. We may be at our most human while filling out the paperwork for life insurance.)

^[!Margin: Reasons for doubt.] The pretraining thesis, while logically impeccable---how is a model supposed to solve all possible trick questions without understanding, just *guessing*?---never struck me as convincing, an argument admitting neither confutation nor conviction.
It feels too much like a magic trick: "here's some information theory, here's a human benchmark, here's how we can encode all tasks as a sequence prediction problem, hey presto---Intelligence!"
There are lots of algorithms which are Turing-complete or 'universal' in some sense; there are lots of algorithms like AIXI which solve AI in some theoretical sense (Schmidhuber & company have many of these cute algorithms such as 'the fastest possible algorithm for all problems', with the minor catch of some constant factors which require computers bigger than the universe).

Why think pretraining or sequence modeling is not another one of them?
Sure, *if* the model got a low enough loss, it'd have to be intelligent, but how could you prove that would happen in practice?
(Training char-RNNs was fun, but they hadn't exactly revolutionized deep learning.)
It might require more text than exists, countless petabytes of data for all of those subtle factors like logical reasoning to represent enough training signal, amidst all the noise and distractors, to train a model.
Or maybe your models are too small to do more than absorb the simple surface-level signals, and you would have to scale them 100 orders of magnitude for it to work, because the scaling curves didn't cooperate.
Or maybe your models are fundamentally broken, and stuff like abstraction require an entirely different architecture to work at all, and whatever you do, your current models will saturate at poor performance.
Or it'll train, but it'll spend all its time trying to improve the surface-level modeling, absorbing more and more literal data and facts without ever ascending to the higher planes of cognition as planned.
Or...

> 'The possibilities of developing an atomic weapon and the desirability of doing it secretly were discussed at a Princeton University conference in which I participated in March 1939...[Bohr](!W "Niels Bohr") said this rare variety could not be separated from common uranium except by turning the country into a gigantic factory. Bohr was worried that this could be done and that an atomic bomb could be developed---but he hoped that neither could be accomplished. Years later, when Bohr came to Los Alamos, I was prepared to say, "You see . . ." But before I could open my mouth, he said: **"You see, I told you it couldn't be done without turning the whole country into a factory. You have done just that."**'
>
> [Edward Teller](!W)^[pg210--211, "The Quiet Enemy", [_The Legacy of Hiroshima_](/docs/radiance/1962-teller-thelegacyofhiroshima.pdf), Teller 1962.]

But apparently, it would've worked fine.
Even RNNs probably would've worked---Transformers are nice, but they seem mostly be about efficiency.^[Another way of interpreting the various papers about how Transformers are actually like RNNs or are [actually Hopfield networks](https://arxiv.org/abs/2008.02217 "'Hopfield Networks is All You Need', Ramsauer et al 2020") is to take that as indicating that what is important about them is not any inherent new capability compared to older architectures, but some lower-level aspect like being more efficiently trainable on contemporary hardware.]
(Training large RNNs is much more expensive, and doing BPTT over multiple nodes is much harder engineering-wise.)
It just required more compute & data than anyone was willing to risk on it until a few true-believers were able to get their hands on a few million dollars of compute.

1. **Q:** Did anyone predict, quantitatively, that this would happen where it did?

    **A:** Not that I know of.

2. **Q:** What would future scaled-up models learn?

    GPT-2-1.5b had a cross-entropy WebText validation loss of ~3.3 (based on the perplexity of ~10 in [Figure 4](/images/ai/gpt/2019-radford-figure4-gpt2validationloss.png "Figure 4: The performance of LMs trained on WebText as a function of model size (from https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf#page=9)"){.invert}, and log~2~(10) = 3.32). GPT-3 halved that loss to ~1.73 judging from [Brown et al 2020](/images/ai/gpt/2020-brown-figure31-gpt3scaling.png "Figure 3.1: Smooth scaling of performance with compute. Performance (measured in terms of cross-entropy validation loss) follows a power-law trend with the amount of compute used for training. The power-law behavior observed in Kaplan et al 2020 continues for an additional two orders of magnitude with only small deviations from the predicted curve. For this figure, we exclude embedding parameters from compute and parameter counts. (Brown et al 2020). Cross-validation loss extrapolation: $L(oss) = 2.57 · C(ompute in petaflop-s/days) ^ −0.048$"){.invert} and using the scaling formula (2.57 × (3.64 × 10^3^)^\u22120.048^). For a hypothetical GPT-4, if the scaling curve continues for another 3 orders or so of compute (100--1000×) before crossing over and hitting harder diminishing returns, the cross-entropy loss will drop to ~1.24 (2.57 × (3.64 × (10^3^ × 10^3^))^\u22120.048^).

    If GPT-3 gained so much meta-learning and world knowledge by dropping its absolute loss ~50% when starting from GPT-2's level, what capabilities would another ~30% improvement over GPT-3 gain? (Cutting the loss that much would still not reach human-level, as far as I can tell.[^human-perplexity]) What would a drop to ≤1, perhaps using wider context windows or recurrency, gain?

    **A:** I don't know.
3. **Q:** Does anyone?

    **A:** Not that I know of.^[As of December 2020, half a year later, almost no researcher has been willing to go on record as saying what specific capabilities they predict future 1t, 10t, or 100t models will have or not have, and at what size which missing capabilities will emerge---just as no one is on record successfully predicting GPT-2 or GPT-3's specific capabilities.]

[^human-perplexity]: How do these absolute prediction performances compare to humans? It's hard to say. The only available benchmarks for perplexity for humans/GPT-2/GPT-3 appear to be WebText, [Penn Tree Bank](/docs/cs/algorithm/1993-marcus.pdf "'Building a Large Annotated Corpus of English: The Penn Treebank', Marcus et al 1993") (PTB; based on the [Brown Corpus](!W)), [1 Billion Word](https://arxiv.org/abs/<PHONE> "'One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling', Chelba et al 2013") (1BW), and [LAMBADA](https://arxiv.org/abs/1606.06031 "'The LAMBADA dataset: Word prediction requiring a broad discourse context', Paperno et al 2016"). But coverage is spotty.

    I found no human benchmarks for WebText or Penn Tree Bank, so I can't compare the human vs GPT-2/GPT-3 perplexities ([GPT-2 PTB](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf#page=5 "Language Models are Unsupervised Multitask Learners: Table 3.Zero-shot results on many datasets. No training or fine-tuning was performed for any of these results. PTB and WikiText-2 results are from (Gong et al 2018). CBT results are from (Bajgar et al 2016). LAMBADA accuracy result is from (Hoang et al 2018) and LAMBADA perplexity result is from (Grave et al 2016). Other results are from (Dai et al 2019)."): 35.7; [GPT-3 PTB](https://arxiv.org/pdf/2005.14165.pdf#page=11&org=openai): 20.5).

    [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf#page=5) was benchmarked at 43 perplexity on the 1 Billion Word (1BW) benchmark vs a (highly extrapolated) [human perplexity of 12](/docs/ai/scaling/2017-shen.pdf "'Estimation of gap between current language models and human performance', Shen et al 2017") (which interestingly extrapolates, using 2012 LSTM RNNs, that "10 to 20 more years of research before human performance is reached"), but that may be an unfair benchmark ("Our model is still significantly worse than prior work on the One Billion Word Benchmark ([Chelba et al 2013](https://arxiv.org/abs/<PHONE> "'One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling', Chelba et al 2013")). This is likely due to a combination of it being both the largest dataset and having some of the most destructive pre-processing---1BW's sentence level shuffling removes all long-range structure.") and 1BW was dropped from the GPT-3 evaluation due to data contamination ("We omit the 4 Wikipedia-related tasks in that work because they are entirely contained in our training data, and we also omit the one-billion word benchmark due to a high fraction of the dataset being contained in our training set.").

    LAMBADA was benchmarked at a [GPT-2 perplexity](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf#page=5) of 8.6, and a [GPT-3 perplexity](https://arxiv.org/pdf/2005.14165.pdf&org=openai#page=12) of 3.0 (zero-shot) / 1.92 (few-shot). [OA claims](https://openai.com/blog/better-language-models/ "Better Language Models and Their Implications: We've trained a large-scale unsupervised language model which generates coherent paragraphs of text, achieves state-of-the-art performance on many language modeling benchmarks, and performs rudimentary reading comprehension, machine translation, question answering, and summarization—all without task-specific training.") in their GPT-2 blog post (but not the paper) that human perplexity is 1--2, but provides no sources and I couldn't find any. (The authors might be guessing based on how LAMBADA was constructed: examples were filtered by whether two independent human raters provided the same right answer, which lower bounds how good humans must be at predicting the answer.)

    So overall, it looks like the best guess is that GPT-3 continues to have somewhere around twice the absolute error of a human. This implies it will take a large (yet, far from impossible) amount of compute to fully close the remaining gap with the current scaling laws. If we irresponsibly extrapolate out the WebText scaling curve further, assume GPT-3 has twice the error of a human at its current WebText perplexity of 1.73 (and so humans are ~0.86), then we need 2.57 · (3.64 · (10^3^ · _x_))^\-0.048^ = 0.86, where _x_ = 2.2e6 or 2,200,000× the compute of GPT-3. (This would roughly equal the cost to the USA of invading Iraq.)

    When is that feasible?

    If we imagine that [peak AI compute usage doubles every 3.4 months](https://openai.com/blog/ai-and-compute/ "AI and Compute: We're releasing an analysis showing that since 2012, the amount of compute used in the largest AI training runs has been increasing exponentially with a 3.4-month doubling time (by comparison, Moore's Law had a 2-year doubling period). Since 2012, this metric has grown by more than 300,000× (a 2-year doubling period would yield only a 7× increase). Improvements in compute have been a key component of AI progress, so as long as this trend continues, it's worth preparing for the implications of systems far outside today's capabilities."), then 2.2e6 would be 22 doublings away---or 6.3 years, in 2027. Most people believe that compute trend must break down soon, and that sort of prediction is a good reason why!

    Going the other direction, Hernandez & Brown 2020's estimate is that, net of hardware & algorithmic progress, the cost of a fixed level of performance halves every 16 months; so if GPT-3 cost ~[$5]($2020)m in early 2020, then it'll cost [$2.5]($2020)m around mid-2021, and so on. Similarly, a GPT-human requiring 2.2e6× more compute would presumably cost on the order of [$10]($2020) trillion in 2020, but after 14 halvings (18 years) would cost [$1]($2020)b in 2038.

# Prospects

> In the problem of decoding, the most important information which we can possess is the knowledge that the message which we are reading is not gibberish...In a similar way, when we consider a problem of nature such as that of atomic reactions and atomic explosives, the largest single item of information which we can make public is that they exist. Once a scientist attacks a problem which he knows to have an answer, his entire attitude is changed. He is already some 50% of his way toward that answer...**the one secret concerning the atomic bomb which might have been kept and which was given to the public and to all potential enemies without the least inhibition, was that of the possibility on its construction.** Take a problem of this importance and assure the scientific world that it has an answer; then both the intellectual ability of the scientists and the existing laboratory facilities are so widely distributed that the quasi-independent realization of the task will be a matter of merely a few years anywhere in the world.
>
> [Norbert Wiener](!W), pg124--125, _[The Human Use of Human Beings](!W)_ (emphasis added)

> "People who work in machine learning simply didn't think that neural networks could do much. People didn't believe large neural networks could be trained...The ideas were all there, the thing that was missing was a lot of supervised data and a lot of compute. Once you have [those two], then there is a third thing is that is needed---and that is *conviction*. Conviction that if you take the right stuff, which already exists, and apply and mix it with a lot of data and a lot of compute, that it will in fact work. And so that was the missing piece."
>
> [Ilya Sutskever](https://www.youtube.com/13CZPWmke6A?t=950#org=openai "Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94")[^Zaremba]

[^Zaremba]: See also [Sutskever's DRL talk](https://www.youtube.com/watch?v=w3ues-NayAs?t=712#openai "If you want to solve a hard problem in reinforcement learning, you just scale. It's just gonna work just like supervised learning. it's the same, the same story exactly. It was kind of hard to believe that supervised learning can do all those things, but it's not just vision, it's everything and the same thing seems to hold for reinforcement learning provided you have a lot of experience."), and [Wojciech Zaremba's](!W "Wojciech Zaremba") [comments about OA5](https://www.youtube.com/watch?v=429QC4Yl-mA&t=1157s "What could make AI conscious? with Wojciech Zaremba, co-founder of OpenAI (2021-06-02)") ([transcript](https://wandb.ai/wandb_fc/gradient-dissent/reports/What-could-make-AI-conscious-with-Wojciech-Zaremba-co-founder-of-OpenAI--Vmlldzo3NDk3MDI)):

    > `Lukas`: "How much of the work then on Dota was, you felt, like fundamentally moving ML forward and how much of it was Dota-specific or can you even pull those apart?"
    >
    > `Wojciech`: "I think there was a decent amount of Dota-specific work. And then I think it was more than optimal, but also simultaneously hard. So I remember at the beginning of Dota project, it was actually unclear how to approach it.
    >
    > People are saying that contemporary reinforcement learning will have no chance in solving this problem. And people looked into off-policy matters, on-policy matters, [evolutionary strategies](https://arxiv.org/abs/1703.03864#openai "'Evolution Strategies as a Scalable Alternative to Reinforcement Learning', Salimans et al 2017"). The thing that became quite surprising is that [methods that already exist](https://arxiv.org/abs/1707.06347#openai "'PPO: Proximal Policy Optimization Algorithms', Schulman et al 2017"), with appropriate scale work extremely well. So that was a big surprise. And I remember some people even before Dota time at OpenAI, saying that maybe reinforcement learning is a dead end. And all of a sudden it's a very different story now."
    >
    > `Lukas`: "For sure."

What can we expect from future DL work?
Will GPT-3 kickstart an arms race where soon we will be discussing, blase, what would seem now like ludicrously farfetched schemes like bidirectional multimodal Transformer 100× the size trained on 100× the data (video/text/PDFs-as-images/photo/robotics) with supplementary supervised learning as the backbone of a MuZero-like learning+planning DRL agent running on thousands of tasks (such as coding) simultaneously?

The existence of [the hardware overhang](https://www.lesswrong.com/posts/N6vZEnCn6A95Xn39p/are-we-in-an-ai-overhang "'Are we in an AI overhang?', Jones 2020") implies that the limiting factor here is less hardware than human: will any organization treat GPT-3 as a Sputnik moment and invest aggressively in scaling programs?
Is there a GPT-4-equivalent brewing away inside DeepMind or Google Brain's TPU pods now?
They aren't stupid, they have the hardware, they have the budgets, they have the people.

But I think they lack a vision.
As far as I can tell: they do not have any such thing, because Google Brain & DeepMind do not believe in the scaling hypothesis the way that Sutskever, Amodei and others at OA do.
Just read through machine learning Twitter to see the disdain for the scaling hypothesis.
(A quarter year on from GPT-3 and counting, can you name a single dense model as large as the 17b Turing-NLG---never mind larger than GPT-3?)

Google Brain is entirely too practical and short-term focused to dabble in such esoteric & expensive speculation, although Quoc V. Le's group occasionally surprises you.
They'll dabble in [mixture-of-expert models](/docs/ai/scaling/moe/index) like [GShard](https://arxiv.org/abs/2006.16668#google "'GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding', Lepikhin et al 2020"), but mostly because they expect to be likely to be able to deploy it or something like it to production in Google Translate.

DeepMind^[Particularly [Demis Hassabis](!W); I'm not sure about [Shane Legg's](!W "Shane Legg") [current views](http://www.vetta.org/2009/12/tick-tock-tick-tock-bing/ "Tick, tock, tick, tock... BING"), although given the accuracy of his [2009 predictions](http://www.vetta.org/2009/12/the-teenies/ "'The Teenies', Shane Legg 2009-12-28") while founding DeepMind & his [2018 comments](https://web.archive.org/web/20210426084422/https://www.stuff.co.nz/technology/103500435/google-deepmind-founder-and-leader-in-artificial-intelligence-returns-to-hamilton "Google DeepMind founder and leader in artificial intelligence returns to Hamilton"), he probably hasn't much changed his views that AI will be empowered by the (realized) exponential compute gains or his [AGI forecast of ~2028](http://www.vetta.org/2010/12/goodbye-2010/ "'Goodbye 2010', Shane Legg 2010-12-10"). (This is consistent with the latest [Metaculus](https://www.metaculus.com/questions/questions/3479/when-will-the-first-artificial-general-intelligence-system-be-devised-tested-and-publicly-known-of/ "When will the first Artificial General Intelligence system be devised, tested, and publicly known of?") [forecasts](https://www.metaculus.com/questions/questions/1394/will-ai-progress-surprise-us/ "Will AI progress surprise us?").)] holds what we might call the "weak scaling hypothesis": they believe that AGI will require us to "find the right algorithms" effectively replicating a mammalian brain module by module, and that while these modules will be extremely large & expensive by contemporary standards (which is why compute is important, to give us "a more powerful tool with which to hunt for the right algorithms"), they still need to be invented & finetuned piece by piece, with little risk or surprise until the final assembly.
Each piece, however, itself can scale: there's no magical intelligence gland or quantum woo which creates a bright line between humans and, say, chimpanzees or rodents.
(As much as we humans extravagantly admire our own capabilities like language or logic, those are relatively minor flourishes on the basic brain---each organism solves the same basic problems, like exploration, long-term memory, learning world-models, associating rewards with specific actions, meta-learning, etc.)
As such, once you have a rat-level AGI, a human-level AGI is just more so.
(And rats are a lot easier to experiment on.)
That is how you get DM contraptions like [Agent57](https://www.deepmind.com/blog/agent57-outperforming-the-human-atari-benchmark "'Agent57: Outperforming the Atari Human Benchmark', Badia et al 2020") which throw the kitchen sink at the wall to see what sticks, and why they place such emphasis on neuroscience as inspiration and cross-fertilization for reverse-engineering the brain.
(See also Sam Altman's [podcast interview comments](https://audio.hbr.org/exponential-view/20201006152648-S5E01_HowGPT-3IsShapingOurAIFuture.mp3?listeningSessionID=0CD_382_23__53c57875ba360c6182f72cc891a037ef695389e2#t=2205 "'How GPT-3 Is Shaping Our AI Future' with Sam Altman/Azeem Azhar (The Exponential View), Wednesday 7 October 2020") on OA's advantage vs unnamed rivals with more compute is because the lack of compute makes them stay "small and focused"---"for sure" like a startup approach.)
When someone seems to have come up with a scalable architecture for cracking a hard problem, like AlphaZero or AlphaStar, they are willing to pour on the gas to make it scale, but otherwise, incremental refinement on ALE and then [DMLab-30](https://arxiv.org/abs/1612.03801#deepmind "'DeepMind Lab', Beattie et al 2016") is the game plan.
They have been biting off and chewing pieces of the brain for a decade, and it'll probably take another decade or two of steady chewing if all goes well.
Because they have locked up so much talent and have so much proprietary code and believe all of that is a major moat to any competitor trying to replicate the complicated brain, they are fairly easygoing.
You will not see DM 'bet the company' on any moonshot; Google's cashflow isn't going anywhere (and [DM's budget](/newsletter/2020/06#deepmind-budget "'June 2020 News § Companies House', Branwen 2019")), and slow and steady wins the race.

Going beyond that, most other research labs like Tesla or FAIR are irrelevant and uninterested.
Chinese AI companies are a question mark: past the language barrier, I seem to discern interest in AGI & little of the reflexive Western opposition, and companies like Baidu occasionally release important research (such as the early scaling paper [Hestness et al 2017](https://arxiv.org/abs/1712.00409#baidu "Deep Learning Scaling is Predictable, Empirically")), but overall, Chinese AI may be overestimated, and they seem to suffer from a kind of Dutch disease---funding for surveillance technology, and for narrow e-commerce niches, is so plentiful that other areas are neglected.

OA, lacking anything like DM's long-term funding from Google or its enormous headcount, is making a startup-like bet that they know an important truth which is a secret: "the scaling hypothesis is true!"
So, simple DRL algorithms like PPO on top of large simple architectures like RNNs or Transformers can emerge, exploiting the blessings of scale, and meta-learn their way to powerful capabilities, enabling further funding for still more compute & scaling, in a virtuous cycle.
This is why OA had to revise its corporate form: lacking any enormous endowment or extremely deep-pocketed patron like Google, where does it get the money to scale (or hire machine learning engineer/researchers who can command salaries in the millions)?
OA has to *earn* the necessary money, so in a move like Mozilla Foundation owning Mozilla Corporation (to sell Firefox search engine placement), or the Hershey orphanage owning Hershey Chocolate or the Girl Scouts licensing their cookies, OpenAI switched from a pure nonprofit funded by donations to a nonprofit which owns a for-profit subsidiary/startup, "OpenAI LP", which can take investments and engage in for-profit activities.
OA LP, while controlled by OA, can then shoot for the moon.
And if OA is wrong to trust in the [God of Straight Lines On Graphs](https://slatestarcodex.com/2018/11/26/is-science-slowing-down-2/ "Is Science Slowing Down?"), well, they never could compete with DM directly using DM's favored approach, and were always going to be an also-ran footnote, so they have no regret.

While all of this hypothetically can be replicated *relatively* easily (never underestimate the amount of tweaking and special sauce it takes) by competitors if they wished (the necessary amounts of compute budgets are still trivial in terms of Big Science or other investments like AlphaGo or AlphaStar or Waymo, after all), said competitors lack the very most important thing, which no amount of money or GPUs can ever cure: the courage of their convictions.
They are too hidebound and deeply philosophically wrong to ever admit fault and try to overtake OA until it's too late.
How can we talk seriously about any kind of military Manhattan Project when the US military [doesn't even let its developers use Tensorflow or PyTorch](https://warontherocks.com/2020/10/trust-algorithms-the-army-doesnt-even-trust-its-own-ai-developers/ "Trust Algorithms? The Army Doesn't Even Trust Its Own AI Developers"), or about government projects in the shadow of coronavirus?
This might seem absurd (surely the Bitter Lesson/scaling hypothesis have now earned enough prior probability to be taken seriously and receive major research investments to test how far they can go, especially given how important the implications are), but look at the repeated criticism of OA *every time* they release a new example of the scaling hypothesis, from GPT-1 to Dactyl to OA5 to GPT-2 to iGPT to GPT-3...
To paraphrase St Augustine, most peoples' reaction to the Bitter Lesson or scaling hypothesis is "grant me scale & compute---but not yet".^[When faced with the choice between having to admit all their fancy hard work is a dead-end, swallow the bitter lesson, and start budgeting tens of millions of compute, or instead writing a disdainful tweet explaining how, "*actually*, GPT-3 shows that scaling is a dead end, it's an environmental catastrophe, and it's just imitation intelligence anyway"---most people will get busy on the tweet!]

A critical indicator will be whether organizations beyond 'the usual suspects' (Microsoft [ZeRO-2](https://www.microsoft.com/en-us/research/blog/zero-2-deepspeed-shattering-barriers-of-deep-learning-speed-scale/ "'ZeRO-2 & DeepSpeed: Shattering barriers of deep learning speed & scale', Team 2020") team has reached [1t-scale training](https://www.microsoft.com/en-us/research/blog/deepspeed-extreme-scale-model-training-for-everyone/ "'DeepSpeed: Extreme-scale model training for everyone', Team et al 2020"), but there is also Nvidia, Salesforce, Allen, Google DM/GB, Connor/EleutherAI, Facebook FAIR) start participating or if they continue to dismiss scaling.
At least as of 2020-10-26, 152 days later, no model has come near GPT-3, and indeed, no model has even exceeded Turing-NLG's 17b.^[A mixture-of-expert model like GShard or an embedding like DynamicEmbedding is not comparable to 'dense' models like GPT-3, as it's always been cheap & easy to train models with billions of 'parameters' in some sense, like extremely large embeddings; however, these parameters do little, and are more like a few hundred shallow models glued back-to-back. They probably do not learn the same interesting things that a dense model would with the same nominal parameter count.]

# Critiquing The Critics

^[!Margin: Keeping track.] GPT-3 in 2020 makes as good a point as any to take a look back on the past decade.
It's remarkable to reflect that someone who started a PhD because they were excited by these new "ResNets" would still not have finished it by now---that is how recent even resnets are, never mind Transformers, and how rapid the pace of progress is.
In 2010, one could easily fit everyone in the world who genuinely believed in deep learning into a moderate-sized conference room (assisted slightly by the fact that 3 of them were busy founding [DeepMind](https://en.wikipedia.org/wiki/DeepMind)).
Someone interested in machine learning in 2010 *might* have read about some interesting stuff from weirdo diehard connectionists in recognizing hand-written digits using all of 1--2 million parameters, or some modest neural tweaks to standard voice-recognition hidden Markov models.
In 2010, who would have predicted that over the next 10 years, deep learning would undergo a Cambrian explosion causing a mass extinction of alternative approaches throughout machine learning, that models would scale up to 175,000 million parameters, and that these enormous models would just spontaneously develop all these capabilities?

No one. That is, no one aside from a few diehard connectionists written off as willfully-deluded old-school fanatics by the rest of the AI community (never mind the world), such as [Moravec](https://jetpress.org/volume1/moravec.htm "'When will computer hardware match the human brain?', Moravec 1998"), Schmidhuber, [Sutskever](https://www.youtube.com/watch?v=13CZPWmke6A "'Ilya Sutskever: Deep Learning | AI Podcast #94 with Lex Fridman', 2020-05-08"), Legg, & Amodei?
One of the more shocking things about looking back is realizing how unsurprising and easily predicted all of this was if you listened to the right people.
In 1998, 22 years ago, Moravec noted that AI research could be deceptive, and hardware limits meant that "intelligent machine research did not make steady progress in its first 50 years, it marked time for 30 of them!", predicting that as Moore's law continued, "things will go much faster in the next 50 years than they have in the last 50."
Moravec further observed that part of the reason for rapid progress was the hardware overhang: while supercomputers of the necessary power would exist long before the connectionist revolution began, no one would be allowed to use them, as they would be devoted to 'more important' (prestigious) hard STEM work, like "physics simulations" (ie. climate simulations & nuclear bombs)^[Strikingly, as of 2020, this is *still* true: eg. the only deep learning research I have seen done on [Summit](!W "Summit (supercomputer)") were [materials](https://arxiv.org/abs/1909.11150 "'Exascale Deep Learning for Scientific Inverse Problems', Laanait et al 2019") [science](https://arxiv.org/abs/2005.00223 "'Pushing the limit of molecular dynamics with ab initio accuracy to 100 million atoms with machine learning', Jia et al 2020") & [biology](https://arxiv.org/abs/2007.06225 "'ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing', Elnaggar et al 2020"). (In double-checking Arxiv, I did find one non-STEM paper using Summit resources: [Lin et al 2019](https://arxiv.org/abs/1910.00932#google "Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos"), focusing on systems engineering in training a video classification model.)], and "AI research must wait for the power to become more affordable."
Affordable meaning a workstation roughly ~[$1000]($1998); sufficiently cheap compute to rival a human would arrive sometime in the 2020s, with the 2010s seeing affordable systems in the lizard--mouse range.
As it happens, the start of the DL revolution is typically dated to [AlexNet](!W) in 2012, by a grad student[^Norvig] using 2 GTX 580 3GB GPUs (launch list price of... [$500]($2010), for a system build cost of perhaps [$1500]($2012)).
2020 saw GPT-3 arrive, and as discussed before, there are many reasons to expect the cost to fall, in addition to the large hardware compute gains that are being forecast for the 2020s despite the general deceleration of Moore's law.^[[Jeff Dean](https://arxiv.org/abs/1911.05289#google "'The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design', Dean 2019") notes, "It is perhaps unfortunate that just as we started to have enough computational performance to start to tackle interesting real-world problems and the increased scale and applicability of machine learning has led to a dramatic thirst for additional computational resources to tackle larger problems, the computing industry as a whole has experienced a dramatic slowdown in the year-over-year improvement of general purpose CPU performance." Under the computational view, this is not a coincidence: compute, not algorithms, are the critical factor; biological systems often come within orders of magnitude, or less, of the theoretical optimum for a task; and the closer one comes to optimal, the slower progress becomes; so, just as artificial computation finally starts doing "interesting real-world problems", it necessarily is approaching its limits. (It could have been otherwise: Moore's law could have stopped short by many orders of magnitude of biological efficiency, or surpassed it by many orders, with no temporal coincidence, and AI happened for other reasons.)]

[^Norvig]: [Peter Norvig](!W) [offers an example](https://wandb.ai/wandb_fc/gradient-dissent/reports/Peter-Norvig-Google-s-Director-of-Research-Singularity-is-in-the-eye-of-the-beholder--Vmlldzo2MTYwNjk?galleryTag=gradient-dissent "Peter Norvig, Google's Director of Research – Singularity is in the eye of the beholder: We're thrilled to have Peter Norvig who join us to talk about the evolution of deep learning, his industry-defining book, his work at Google, and what he thinks the future holds for machine learning research (2020-11-20).") of what happens when grad students *can't* afford the necessary computing power to make neural nets work:

    > `Lukas Biewald`: "When you look at deep learning, it sort of feels like that came suddenly, but a lot of those techniques were around, in fact in your book, I remember quite far back. Do you think that the field missed something, or was it just not possible to run at the scale necessary to show that these neural network techniques were working better than people expected in the early aughts?"
    >
    > `Peter Norvig`: "Yeah. I mean, if you say suddenly, right, we've got a sudden leap in computer vision and image net after Hinton had been trying the same thing for 30 years, right?...And then it finally worked. And I think the biggest difference was the computing power. Definitely there were advances in data. So we could do ImageNet because Fei-Fei Li and others gathered this large database, and that was really important. There are certainly differences in the algorithm, right? We've got a slightly different squashing function. Instead of shaped like this \[sigmoid\], it's shaped like this \[[ReLU](!W)\]. I mean, I don't know how big a deal that was, but we learned how to do [stochastic gradient descent](!W) a little bit better. We figured that [dropout](!W "Dilution (neural networks)") gave you a little bit better robustness."
    >
    > "So there were small things, but I think probably the biggest was the computing power. And I mean, I certainly remember Geoff Hinton came to Berkeley when I was a grad student in 1981, I think, when he talked about these neural nets. And we fellow grad students thought that was so cool. So we said, 'Let's go back into the lab and implement it.'"
    >
    > "And of course, there was absolutely nothing you could download, so we had to build it all from scratch. And we got it to do exclusive or [XOR], and then we got it to do something a little bit more complicated. And it was exciting. And then we gave it the first real problem, and it ran overnight, and it didn't converge, and we let it run one more day, and it still didn't converge. And then we gave up, and we went back to our sort of knowledge-based systems approach. But if we had the computing power of today, it probably would have converged after five seconds."

    By my estimate, Norvig's attempt used the equivalent of 0.8 *milliseconds* of contemporary GPU-time.

    (In ~1981, an expensive PC costing the equivalent of >[$5,000]($2020) might have an additional [Intel 8087](!W) floating-point [coprocessor](!W) capable of 50,000 FP64 FLOPS; conservatively assuming that 'overnight' + 'one more day' ≤ 2 days, then Norvig's experiment used 2d×24h×60m×60s × 50,000 = 8×10^9^ FLOPS; a 2020 Nvidia [A100](!W "Ampere (microarchitecture)") GPU nominally priced ~[$10,000]($2020) boasts 9.7 FP64 TFLOPS or 9,700,000,000,000 FLOPS (and far more in the more useful low-precision regimes like FP32, but 1981 ML didn't know that); thus, 8×10^9^ / 9.7×10^12^ = 8×10^−4^ seconds = 0.8 milliseconds.)

The accelerating pace of the last 10 years should wake anyone from their dogmatic slumber and make them sit upright.
And there are 28 years left in Moravec's forecast...

The temptation, that many do not resist so much as revel in, is to give in to a _déformation professionnelle_ and dismiss any model as "just" this or that("just billions of IF statements" or "just a bunch of multiplications" or "just millions of memorized web pages"), missing the forest for the trees, as Moravec commented of chess engines:

> The event was notable for many reasons, but one especially is of interest here. Several times during both matches, Kasparov reported signs of mind in the machine. At times in the second tournament, he worried there might be humans behind the scenes, feeding Deep Blue strategic insights!...In all other chess computers, he reports a mechanical predictability stemming from their undiscriminating but limited lookahead, and absence of long-term strategy. In Deep Blue, to his consternation, he saw instead an "alien intelligence."
>
> ...Deep Blue's creators know its *quantitative* superiority over other chess machines intimately, but lack the chess understanding to share Kasparov's deep appreciation of the difference in the *quality* of its play. I think this dichotomy will show up increasingly in coming years. Engineers who know the mechanism of advanced robots most intimately will be the last to admit they have real minds. From the inside, robots will indisputably be machines, acting according to mechanical principles, however elaborately layered. Only on the outside, where they can be appreciated as a whole, will the impression of intelligence emerge. A human brain, too, does not exhibit the intelligence under a neurobiologist's microscope that it does participating in a lively conversation.

But of course, if we ever succeed in AI, or in reductionism in general, it *must be by reducing Y to 'just X'*.
Showing that some task requiring intelligence can be solved by a well-defined algorithm with no 'intelligence' is precisely what success must look like!
(Otherwise, the question has been thoroughly begged & the problem has only been pushed elsewhere; computer chips are made of transistors, not especially tiny homunculi.)

> "As long as the AI [OA5] can explore, it will learn, given enough time...We just kept waiting for the magic to run out. We kept waiting to hit a wall, and we never seemed to hit a wall."
>
> [Greg Brockman](https://qz.com/<PHONE>/openai-built-gaming-bots-that-can-work-as-a-team-with-inhuman-precision/ "OpenAI built gaming bots that can work as a team with inhuman precision")

> "Give it the compute, give it the data, and it will do amazing things. This stuff is like---it's like *alchemy*!"
>
> [Ilya Sutskever](https://www.newyorker.com/magazine/2019/10/14/can-a-machine-learn-to-write-for-the-new-yorker "Can a Machine Learn to Write for The New Yorker? Extraordinary advances in machine learning in recent years have resulted in A.I.s that can write for you."), summer 2019

^[!Margin: Hindsight is 20⁄20.] Even in 2015, the scaling hypothesis seemed highly dubious: you needed something to scale, after all, and it was all too easy to look at flaws in existing systems and imagine that they would never go away and progress would sigmoid any month now, soon.
Like the genomics revolution where a few far-sighted seers extrapolated that the necessary _n_ for GWASes would increase exponentially & deliver powerful PGSes soon, while sober experts wrung their hands over "missing heritability" & the miraculous complexity of biology & scoff about how such _n_ requirements proved GWAS was a failed paradigm, the future arrived at first slowly and then quickly.
Yet, here we are: all honor to the fanatics, shame and humiliation to the critics!^[Now that GPT-3's few-shot and [T5 finetuning](https://arxiv.org/abs/2003.08380#google "'TTTTTackling WinoGrande Schemas', Lin et al 2020") have begun to make people like Gary Marcus feel slightly nervous about WinoGrande, they have [begun preparing](https://arxiv.org/abs/2004.13831 "'A Review of Winograd Schema Challenge Datasets and Approaches', Vid Kocijan, Thomas Lukasiewicz, Ernest Davis, Gary Marcus, Leora Morgenstern 2020") [their excuses](https://arxiv.org/abs/2201.02387 "'The Defeat of the Winograd Schema Challenge', Kocijan et al 2022") for why Winograd schemas [weren't *really*](/Modus "'One Man's Modus Ponens', Branwen 2012") good measures of commonsense reasoning/intelligence (because intelligence, of course, is whatever AI can't do yet).]
If only one could go back 10 years, or even 5, to watch every AI researchers' head explode reading this paper...
Unfortunately, few heads appear to be exploding now, because human capacity for hindsight & excuses is boundless ("I can get that much with finetuning, anyway I predicted it all along, how boring") and, unfortunately, ["there is no fire alarm"](https://intelligence.org/2017/10/13/fire-alarm/ "Yudkowsky 2017") for AGI.
(If you are still *certain* that there is near-zero probability of AGI in the next few decades, why?
Did you predict---in writing---capabilities like GPT-3?
Is this how you expect AI failure to look in the decades beforehand?
What specific task, what specific number, would convince you otherwise?
How would the world look different than it does now if these crude prototype insect-brain-sized DL systems were not on a path to success?)

^[!Margin: Authority without accountability.] What should we think about the experts?
Projections of failure were made by eminent, respectable, serious people.
They spoke in considered tones of why AI hype was excessive and might trigger an "AI winter", and the fundamental flaws of fashionable approaches and why brute force could not work.
These statements were made routinely in 2014, 2015, 2016... And they were wrong.
I am aware of few issuing a _mea culpa_ or reflecting on it.^[[Feynman](https://history.nasa.gov/rogersrep/v2appf.htm "Appendix F: Personal Observations on the Reliability of the Shuttle"): "There are several references to previous flights; the acceptance and success of these flights are taken as evidence of safety. But erosion and blowby are not what the design expected. They are warnings that something is wrong. The equipment is not operating as expected, and therefore there is a danger that it can operate with even wider deviations in the unexpected and not thoroughly understood way. The fact that this danger did not lead to catastrophe before is no guarantee that it will not the next time, unless it is completely understood."]
It is a puzzling failure, and I've [reflected on it before](/newsletter/2019/13#what-progress "'2019 News § What Progress?', Branwen 2019").

^[!Margin: Phatic, not predictive.] There is, however, a certain tone of voice the bien pensant all speak in, whose sound is the same whether right or wrong; a tone shared with many statements in January to March of this year; a tone we can also find in a 1940 _Scientific American_ article authoritatively titled, ["Don't Worry---It Can't Happen"](/docs/existential-risk/1940-sciam-harrington-nuclearweapons-dontworryitcanthappen.pdf), which advised the reader to not be concerned about it any longer "and get sleep".
('It' was the atomic bomb, about which certain scientists had stopped talking, raising public concerns; not only could it happen, the British bomb project had already begun, and 5 years later it did happen.)

^[!Margin: The iron law of bureaucracy: Cathedral gothic.]  This tone of voice is the voice of [authority](https://srconstantin.wordpress.com/2016/10/20/ra/ "'Ra', Sarah Constantin 2016"). The voice of authority insists on calm, and people not "panicking" (the chief of sins). The voice of authority assures you that it won't happen (because it can't happen). The voice utters simple arguments about why the status quo will prevail, and considers only how the wild new idea could fail (and not all the possible options). The voice is not, and does not deal in, uncertainty; things will either happen or they will not, and since it will not happen, there is no need to take any precautions (and you should not worry because it can't happen). The voice does not believe in drawing lines on graphs (it is rank numerology). The voice does not issue any numerical predictions (which could be falsified). The voice will not share its source code (for complicated reasons which cannot be explained to the laity). The voice is opposed to unethical things like randomized experiments on volunteers (but will overlook the insult). The voice does not have a model of the future (because a model implies it does not already know the future). The voice is concerned about its public image (and unkind gossip about it by other speakers of the voice). The voice is always sober, respectable, and credentialed (the voice would be pleased to write an op-ed for your national magazine and/or newspaper). The voice speaks, and is not spoken to (you cannot ask the voice what objective fact would change its mind). The voice never changes its mind (until it does). The voice is never surprised by events in the world (only disappointed). The voice advises you to go back to sleep (right now).

When someone speaks about future possibilities, what is the tone of their voice?

[null](/Scaling-hypothesis#blessings-of-scale){style="display:none;"}
[null](/Scaling-hypothesis "'The Scaling Hypothesis', Branwen 2020"){style="display:none;"}