# Book review: Human Compatible

Book review: Human Compatible, by Stuart Russell.

Human Compatible provides an analysis of the long-term risks from
artificial intelligence, by someone with a good deal more of the
relevant prestige than any prior author on this subject.

What should I make of Russell? I skimmed his best-known book,
[Artificial Intelligence: A Modern
Approach](https://en.wikipedia.org/wiki/Artificial_Intelligence:_A_Modern_Approach),
and got the impression that it taught a bunch of ideas that were popular
among academics, but which weren't the focus of the people who were
getting interesting AI results. So I guessed that people would be better
off reading Deep Learning by Goodfellow, Bengio, and Courville instead.
Human Compatible neither confirms nor dispels the impression that
Russell is a bit too academic.

However, I now see that he was one of the pioneers of [inverse
reinforcement
learning](https://en.wikipedia.org/wiki/Reinforcement_learning#Inverse_reinforcement_learning),
which looks like a fairly significant advance that will likely become
important someday (if it hasn't already). So I'm inclined to treat him
as a moderately good authority on AI.

The first half of the book is a somewhat historical view of AI, intended
for readers who don't know much about AI. It's ok.

### Key proposals

Russell focuses a moderate amount on criticizing what he calls the
**standard model** of AI, in which someone creates an intelligent agent,
and then feeds it a goal or utility function.

I'm not too clear how standard that model is. It's not like there's a
consensus of experts who are promoting it as the primary way to think of
AI. It's more like people find the model to be a simple way to think
about goals when they're being fairly abstract. Few people seem to be
defending the standard model against Russell's criticism (and it's
unclear whether Russell is claiming they are doing so). Most of the
disagreements in this area are more about what questions we should be
asking, rather than on how to answer the questions that Russell asks.

Russell gives a fairly cautious overview of why AI might create risks
that are as serious as the risks gorillas face from humans. Then he
outlines an approach that might avoid those risks, using these three
rules:

> 1.  The machine's only objective is to maximize the realization of
>     human preferences.
> 2.  The machine is initially uncertain about what those
>     preferences are.
> 3.  The ultimate source of information about human preferences is
>     human behavior.

Note that these are high-level guidelines for researchers; he's not at
all claiming they're rules that are ready to be written into an AI.

Russell complains that the AI community has ignored the possibility of
creating AIs that are uncertain about their objective, and calls that "a
huge blind spot".

I'm unclear on whether this qualifies as a blind spot. I can imagine a
future in which it's important. But for AI as it exists today, it looks
like uncertainty would add complexity, without producing any clear
benefit. So I think it has been appropriate for most AI researchers to
have postponed analyzing it so far.

An aside: Russell points out that uncertainty provides an interesting
way to avoid wireheading: if the reward is defined so that it can't be
observed directly, then the AI will know that hacking the AI's signal
won't create more brownie points in heaven.

### Feasibility?

Russell is fairly convincing in his claim that AIs which are designed
according to his rules will relatively safe. That's a much better
achievement than most authors manage on this topic.

I'm a bit less convinced that this approach is easy enough to implement
that it will be competitive with other, possibly less safe, approaches.

Some of my doubt derives from the difficulty, using current techniques,
of encoding the relevant kind of abstract objectives into an AI.

The objectives that Russell wants don't look much like the kind of
objectives that AI researchers know how to put into an AI.

It's fairly well known how to give an AI objectives either by using a
large number of concrete examples of the "correct" result, or by
specifying a readily quantifiable reward. Even a dilettante such as
myself knows the basics of how to go about either of those approaches.

In contrast, it's unclear how to encode an objective that depends on
high-level concepts such as "human" or "preference that is inferred from
behavior" without the AI already having done large amounts of learning.

Maybe there's some way to use predictions about observed preferences as
if the predictions quantified the actual objective? That looks partly
right. But how do we tell the AI that the predictions aren't the real
objective? If we don't succeed at that, we risk something like the King
Midas problem: a naive new AI might predict that King Midas's
preferences will be better satisfied if everything he touches turns to
gold. But if that prediction becomes the AI's objective, then the AI
will resist learning that the King regrets his new ability, since that
might interfere with it's objective of turning anything he touches into
gold.

AI researchers have likely not yet tried to teach their systems about
hard-to-observe concepts such as utopia, or heaven. Teaching an AI to
value not-yet-observed preferences seems hard in roughly the same way.
It seems to require using a much more sophisticated language than is
currently used to encode objectives.

I'll guess that someone would need to hard code many guesses about what
human preferences are, to have somewhere to start, otherwise it's
unclear how the AI would initially prefer any action over another. How
is it possible to do that without the system already having learned a
lot about the world? And how is it possible for the AI to start learning
without already having some sort of (possibly implicit) objective?

Is there some way to start a system with a much easier objective than
maximizing human preferences, then switch to Russell's proposed
objective after the system understands concepts such as "human" and
"preference"? How hard is it to identify the right time to do that?

I gather that some smart people believe some of these questions need to
be tackled head on. My impression is that most of those people think AI
safety is a really hard problem. I'm unclear on how hard Russell thinks
AI safety is.

It's quite possible that there are simple ways to implement Russell's
rules, but I'm moderately confident that doing so would require a fairly
large detour from what looks like the default path to human-level AI.

Compare Russell's approach to [Drexler's
ideas](http://www.bayesianinvestor.com/blog/index.php/2019/01/30/drexler-on-ai-risk/)
of only putting narrow, short-term goals into any one system. (I think
Drexler's writings were circulating somewhat widely before Russell
finished writing Human Compatible, but maybe Russell finished his book
before he could get access to Drexler's writings).

If Drexler's approach is a good way to generate human-level AI, then I
expect it to be implemented sooner than Russell's approach will be
implemented.

Still, we're still at a stage where generating more approaches to AI
safety seems more valuable than deciding which one is best. Odds are
that the researchers who actually implement the first human-level AIs
will have better insights than we do into which approaches are most
feasible. So I want to encourage more books of this general nature.

Russell's rules show enough promise to be worth a fair amount of
research, but I'm guessing they only have something like a 5% or 10%
chance of being a good solution to AI risks.

### Miscellaneous

Russell ideas often sound closer to those of
[Bostrom](http://www.bayesianinvestor.com/blog/index.php/2014/07/28/superintelligence/)
and [MIRI](https://intelligence.org) than to those of mainstream AI, yet
he dismisses recursive self-improvement and fast takeoff. His reasons
sound suspicious - I can't tell whether he's got good intuitions that he
has failed to explain, or whether he ignores those scenarios because
they're insufficiently mainstream.

Russell makes the strange claim that, because existing AI is poor at
generalizing across domains,

> when people talk about "machine IQ" increasing rapidly and threatening
> to exceed human IQ, they are talking nonsense.

But Russell seems to take the opposite position 100 pages later, when
he's dismissing Kevin Kelly's [The Myth of a Superhuman
AI](https://www.wired.com/2017/04/the-myth-of-a-superhuman-ai/). I'm
disappointed that Russell didn't cite the
[satire](http://aleph.se/andart2/technology/why-fears-of-supersizing-are-misplaced/)
of Kelly that argues against the feasibility of bigger than human
machines.

Russell has a strange response to Bostrom's proposal to use one good AI
to defend against any undesirable AIs. Russell says that we'd end up
"huddling in bunkers" due to the "titanic forces" involved in battles
between AIs. Yet Bostrom's position is clearly dependent on the
assumption of a large power asymmetry between the dominant AI (or
possibly a dominant coalition of AIs?) and any new bad AI - why would
there be much of a battle? I'd expect something more like
[Stuxnet](https://en.wikipedia.org/wiki/Stuxnet).

There are lots of opinions about how much power disparity there will be
between the most powerful AI and a typical new AI, and no obvious way to
predict which one is correct. Russell says little about this issue.

But suppose such battles are a big problem. How is this concern specific
to Bostrom's vision? If battles between AI are dangerous to bystanders,
what's the alternative to good AI(s) fighting bad AIs? Does someone have
a plan to guarantee that nobody ever creates a bad AI? Russell shows no
sign of having such a plan. Russell might be correct here, but if so,
the issue deserves more analysis than Russell's dismissal suggests.

### Philosophy

Russell concludes with a philosophical section that tackles issues
relating to morality.

It includes some good thoughts about the difficulties of inferring
preferences, and some rather ordinary ideas about utilitarianism,
including some standard worries about [the repugnant
conclusion](http://www.bayesianinvestor.com/blog/index.php/2009/09/25/turning-the-repugnant-conclusion-into-utopia/).

Here's one of Russell's stranger claims:

> in a sense, *all* humans are [utility
> monsters](https://en.wikipedia.org/wiki/Utility_monster) relative to,
> say, rats and bacteria, which is why we pay little attention to the
> preferences of rats and bacteria in setting public policy.

Is that why we ignore their preferences? My intuition says it's mostly
because we're selfish and not trying to cooperate with them. I don't
think I'm paying enough attention to their preferences to have figured
out whether we're utility monsters compared to them.

### Conclusion

I'll end with a more hopeful note (taken from right after Russell
emphasizes that machines won't imitate the behavior of people they
observe):

> It's possible, in fact, that if we humans find ourselves in the
> unfamiliar situation of dealing with purely altruistic entities on a
> daily basis, we may learn to be better people ourselves - more
> altruistic and less driven by pride and envy.

Human Compatible will be somewhat effective at increasing the diversity
of AI safety research, while heading off risks that AI debate will
polarize into two tribes.

See also [this
review](https://www.lesswrong.com/posts/FuGDYNvA6qh4qyFah/thoughts-on-human-compatible)
from someone who, unlike me, is doing real AI safety research.