# The Coming Wave

Book review: The Coming Wave: Technology, Power, and the Twenty-first
Century's Greatest Dilemma, by Mustafa Suleyman.

An author with substantial AI expertise has attempted to discuss AI in
terms that the average book reader can understand.

The key message: AI is about to become possibly the most important event
in human history. Maybe 2% of readers will change their minds as a
result of reading the book.

A large fraction of readers will come in expecting the book to be mostly
hype. They won't look closely enough to see why Suleyman is excited.

### Danger

How much danger does Suleyman see?

Mostly he's vague about what failure looks like.

> AI, synthetic biology, and other advanced forms of technology ...
> could present an existential threat to nation-states - risks so
> profound they might disrupt or even overturn the current geopolitical
> order.

Much later, he says we could even face "an existential risk to the
species."

However, these isolated quotes don't reflect the tone of the book,
which seems calmer than those quotes led me to expect.

Does that tone reflect an assumption that most readers wouldn't believe
a more frightened tone? Or that he's resigned to the inevitability of
the human race taking a big gamble? Most likely it's some combination
of the two.

He claims that advances in AI technology are impossible to stop. Three
independent sets of incentives are pushing it forward: the profit
motive, the need for nations to protect themselves from other nations,
and competition for prestige. Wise governments would be our only hope of
stopping the effects of those forces. Suleyman has mostly given up hope
that governments are that competent.

The book's confusing impressions are best summarized by this quote:

> And yet, while there is compelling evidence that containment is not
> possible, temperamentally I remain an optimist.

### Would believing him help the average reader?

I suppose we should presume unless proven otherwise that people are
better off understanding more.

Yet there's depressingly little that the book's target audience can do
about AI.

### Containment

What does Suleyman mean by containment? I can only guess, based on how
he advises trying to achieve it.

He says we should avoid recursive self-improvement and autonomy. But he
devotes just one sentence to that. Does he think such rules are no more
important than a prohibition on using AI for electioneering? Maybe.

What does he think about the difficulty of enforcing those rules? I see
plenty about why containment in general is hard, but little about
whether some rules are more enforceable than others.

### Distinguishing AI Prospects from Hype

Suleyman's warnings look hard for an average person to distinguish from
the usual hype that surrounds new technologies.

I don't know what he could have done better to convince the average
reader that this time is different.

I made a few attempts last year to [persuade
superforecasters](https://bayesianinvestor.com/blog/index.php/2023/07/17/existential-risk-persuasion-tournament/)
that AI would produce transformative changes in a decade or so. I mostly
failed. The unusual nature of my claims about AI advances caused many
people to demand a higher burden of proof than could be met with a few
hours worth of effort.

The situation feels like how it must have felt in February 2020 to those
who saw that COVID wasn't going to be contained. That time, I was one
of the people who was tired of reading about possible SARS, Ebola, swine
flu, etc. pandemics. What could have convinced me to more carefully
investigate how COVID was spreading? The most promising approach that
comes to mind is people offering to bet me. But there's limits to how
many bet offers I can pay attention to, so even a culture of betting
might have been inadequate. With AI, it's somewhat tricky to figure out
what to bet on.

I wish Suleyman had found some way to show a graph of accelerating
generality in AIs.

A decade ago, it was hard to see how a program designed to play Go could
also look better than a total idiot at chess.

Nowadays, systems are sufficiently general purpose that a good deal of
the development effort goes into suppressing unwanted abilities. I'll
guess that about a third of the human effort devoted to training GPT-4
went to tasks such as persuading it not to generate porn or racist
jokes. That feels like a significant change, which distinguishes it from
most new technologies.

If we could get a good graph of AI progress, that would go a long way
toward clarifying disagreements over whether AI will become powerful
soon.

If I'd been writing this book, I'd have shown some sort of graph of
generality versus time, with an implication that AI will reach human
levels in the early 2030s. The numbers that I would use would be
somewhat arbitrary and hard to defend in detail. But the overall pattern
is hard to deny. That pattern reflects the most important aspects of my
reasoning about AI timelines.

The book makes many forecasts about a variety of technologies. I
encourage readers to pay careful attention to which ones have clear
dates associated with them, and take those forecasts more seriously than
the forecasts with vague timelines.

His most interesting forecast involves what he labels the Modern Turing
Test. It tests whether the AI fulfill this request:

> Go make $1 million on Amazon in a few months with just a $100,000
> investment. ... [caveats about bank accounts] ... I think it will
> be done with a few minor human interventions within the next year, and
> probably fully autonomously within three to five years.

This seems like his second most important forecast:

> Pause for a moment and imagine a world where robots with the dexterity
> of human beings that can be "programmed" in plain English are
> available at the price of a microwave. ... utterly inevitable over a
> twenty-year horizon, and possibly much sooner.

### Comparing AI to a Prior Wave of Futurism

The middle part of the book is a survey of other possibly imminent
technological advances. It's somewhat reminiscent of Drexler's 1986
book [The Engines of
Creation](https://en.wikipedia.org/wiki/Engines_of_Creation).

Suleyman's forecasts predict somewhat more dramatic changes than
Drexler predicted back in 1986.

How does AI today compare to how nanotech looked back then? Talk about
AI has spread a bit wider than talk about nanotech ever did, but that
might be due to more people understanding software than chemistry.

The difference that strikes me the most is the reaction of venture
capital. 15 years after Drexler predicted that nanotech would become
important in 15 to 30 years, a few VCs started talking to nanotech
companies about maybe investing a few million over a few years. The dot
com crash convinced them to be more cautious. Nanotech researchers
mostly continued to focus on academic publications. Most nanotech
funding came from government grants, and went to chemistry and physics
projects that were renamed as nanotech in order to sound more cutting
edge.

Compare that to 2022. VCs threw billions at AI startups, some of them
months-old companies whose founders had limited track records. At least
[one of those startups](https://www.conjecture.dev/) is too committed to
secrecy to be confused with a project that's focused on academic
prestige. Academia and government grants seem only marginally relevant
to recent AI progress.

And of course there's OpenAI. Clearly motivated in part by prestige,
but more by prestige among trendy startups than prestige among academia.

### Digressions

Parts of the book wander far from Suleyman's areas of expertise.

One example:

> a jobs recession will crater tax receipts, damaging public services
> and calling into question welfare programs just as they are most
> needed.

Our experience at the start of the COVID pandemic suggests that many
governments are able to handle a sudden jobs recession, at least for a
year. I see a good chance that capital gains from leading AI company
stocks will offset the decline in taxes on wages. A massive AI workforce
will create plenty of wealth. It's not obvious why governments would
have difficulty grabbing enough of that wealth to support increasing
welfare payments.

There will be important political problems, but they'll likely be quite
different from what Suleyman worries about.

### The Book's Recommendations

I'll comment on an arbitrary sample of the book's recommendations for
containing AI.

Suleyman wants an Apollo Program on technical safety. This might be the
right way to think about some subsets of AI safety research, where
there's something resembling a consensus as to what we want (e.g.
interpretability).

But having one big centrally organized project is risky, because there
are still some potentially critical areas where we don't know who is
even asking the right questions. E.g. there's little agreement over how
to tell an AI what goals to follow. The leading strategy,
[RLHF](https://www.lesswrong.com/tag/rlhf), appears to be adequate only
in worlds where AI alignment is fairly easy. Having one big project take
charge of this could discourage the pursuit of novel approaches that
might be safer.

Suleyman recommends limits on copying AI weights, using computer
security techniques. I see some resemblance to how telomeres arguably
limit cancer. I don't quite see how that can be implemented without
dramatic restrictions on where those weights can be used. In particular,
how can we reliably ensure that an AI can't copy it's own weights?

He wants a culture of openness around failures that's similar to what
the world has achieved for plane crashes. That reflects a somewhat
optimistic guess about how gradually failures will become more serious.

Suleyman says critics should work on advancing AI capabilities. Suleyman
believes that's the most likely way someone can influence the course of
AI. Alas, that relies on controversial assumptions that Suleyman
doesn't adequately explain. If he does have explanations, they would
likely be hard to fit into a laymen-oriented book.

### Conclusion

I rate this book as four stars, mainly for being a serious attempt by a
reputable authority to convince laymen that big changes are imminent.

His advice is vaguely good, but likely too ordinary to address the
magnitude of our coming challenges.

Nearly all the value lies in the first third or the first half of the
book. Feel free to quit partway through.