# Foom Liability

Robin Hanson suggests, partly in response to calls for a pause in
development of AGI, [liability
rules](https://www.overcomingbias.com/p/foom-liability) for risks
related to AGI rapidly becoming powerful.

My intuitive reaction was to classify foom liability as equivalent to a
near total ban on AGI.

Now that I've found time to think more carefully about it, I want to
advocate foom liability as a modest improvement over any likely pause or
ban on AGI research. In particular, I want the most ambitious AI labs
worldwide to be required to have insurance against something like $10
billion to $100 billion worth of damages.

Liability has the obvious advantage that whoever is financially
responsible for the liability (AI labs? insurance companies?) will have
better incentives to study the risks carefully than would politicians,
bureaucrats, or standards committees composed of industry
representatives.

A more subtle advantage: limited risk that it will be used to protect
incumbents or permanently stifle good technologies.

I see somewhat less risk of liability being hijacked by culture
warriors.

Liability wouldn't produce ideal incentives, due to scenarios where AGI
accidents cause more damage than any company can compensate for (e.g.
human extinction). I'll guess that liability would provide one tenth of
the optimal incentive.

But remember that there's not too much reason to worry about improving
the financial incentives to avoid extinction. I'm not too concerned
with the risk that DeepMind will train a system that they think has a 5%
chance of killing us all. I'm more concerned about a less responsible
lab training a system that their insurance company would think had a 5%
chance of starting a minor war, and which I think has a 5% chance of
killing us all. I.e. my main concern is with AI labs that are
overconfident about safety, but who are rational enough to respond to
projected penalties for medium-sized accidents.

For the risks that worry me most, there would need to be an
international agreement on such liability / insurance. That creates
difficulties beyond those that would confront a direct ban. Some
countries would allow fly-by-night insurance companies to provide
"insurance" that would not, in practice, cover much of the liability.

### How Much Liability

Key variables are: who needs to buy liability insurance (or provide the
equivalent evidence of self-insurance), and how much harm does the
insurance policy need to cover?

It feels pretty arbitrary to assign a specific number to many of the AI
accidents that I can imagine.

Suppose an AI makes a serious attempt at world conquest: It hacks and
blackmails its way into apparent control of several nuclear arsenals. It
releases scandalous evidence about influential people who warned about
AIs gaining power. Yet it gets shut down without any clear harm to
innocent people.

I'm guessing that penalties in the vicinity of $10 billion, give or
take a factor or 10, would be in the right ballpark.

Much more than that would mean that only the wealthiest organizations
could afford to work on AGI. I was tempted to claim that such
organizations will on average be more responsible than smaller
organizations. Then I remembered Elon Musk, and decided it was unclear.

It feels harder to adequately describe what entities would need to be
insured. E.g. open source clones of leading LLMs are unlikely to be
smarter than the original LLMs (compute-intensive training seems likely
to remain important), but might be more dangerous because they attract
more reckless tinkerers who make them more agenty. That is a problem for
any proposal to regulate or slow AI development. I'm focusing in this
post on how foom liability compares to other options. I'm not claiming
to know whether any of these proposals are feasible.

### Scenarios

I'll now imagine several scenarios as to how foom liability would play
out. I'm giving probabilities to indicate my vague intuitions about
which scenarios are most likely. All probabilities are conditional on
some sort of international agreement to require strict liability for
foom-like accidents.

### Scenario 1: Permanent Ban on AGI

I am reminded of what liability rules have done to innovation in small
aircraft, and of what regulation has done to nuclear power. From [Where
is my Flying
Car?](https://www.bayesianinvestor.com/blog/index.php/2018/10/14/where-is-my-flying-car/):

> One of the more ironic regulatory pathologies that has shaped the
> world of general aviation is that most of the planes we fly are either
> 40 years old or homemade - and that we were forced into that position
> in the name of safety.

In this scenario, I imagine an airtight requirement that key types of AI
research get an insurance company to write a policy covering that
research. I also imagine that insurance companies are reluctant to write
those policies. The existence of somewhat special rules for foom
liability reinforces widespread concern over the risks, in ways that
cause a perpetual upward creep in expected damage awards from arguably
dysfunctional courts.

This requires suppression of unauthorized research that is a good deal
more effective than what I know how to implement. Yet if most professors
at leading universities decided that AGI would make their jobs obsolete
(while GPT-4 won't), then I wouldn't want to bet against their ability
to devise an airtight ban.

I'll give this scenario a 5% chance.

### Scenario 2: Leaky Ban on AGI

More likely, it will be hard to fully enforce any insurance requirement.
Most of the problem is likely to come from the difficulty of identifying
what kind of research is risky enough to require insurance.

An analogy would be rules that outlaw unlicensed taxis and hotels. Uber
and Airbnb created businesses that compete with incumbents in those
industries, without meeting the formal definition of taxis and hotels,
defeating the goal of protecting those incumbents from competition.

I imagine that it's hard to draft an insurance requirement that
reliably distinguishes between safe software and software that might
foom. Any attempt to strike a reasonable balance between allowing good
software to remain unregulated, and restricting all risky software, will
leave opportunities for clever startups to create unlicensed AGIs that
will foom to whatever extent foom is possible.

I'm imagining in this scenario that AI labs mostly keep roughly the
same mix of focus on capability and safety research that they have now.
They postpone risky training of large systems. That slows down
capability advances, and slows down some safety research that depended
on the availability of more powerful AIs.

I expect this scenario would buy AI safety researchers a year or five.
That would come at the cost of an increased risk that AGIs will be
designed by more reckless developers. It's unclear what net affects
this scenario would have on our safety.

I'll give this scenario a 55% chance.

### Scenario 3: Goldilocks

In my hoped for scenario, foom liability is effective at delaying AGI
development for a year or two, and at spurring increased safety
research.

Insurance companies initially indicate that they would charge more than
$10 billion for a policy covering the most reputable AI labs. A year
later, they sell one policy for $5 billion. Six months after that,
multiple policies are sold for $3 billion.

Many people in the industry end up agreeing that the insurance
requirement caused the industry to reduce some important risks, at a
fairly acceptable cost in delaying benefits.

I'll give this scenario a 10% chance.

### Scenario 4: Oligopoly

The main effect might be to slow development, by prohibiting small
organizations from doing any important AI development.

I'm imagining here that a handful of companies are able to buy
insurance for maybe $10 billion each. They were already doing most of
what they were able to do to minimize risks. The insurance was a
nuisance due to the need to articulate safety measures, most of which
required expert knowledge to understand. The insurance companies didn't
learn enough to provide any useful ideas about safety beyond whatever
was the default path. Everyone ends up agreeing that there are important
risks, but we find no way to reach any consensus on how to handle the
risks.

My best guess is that this makes us slightly safer, via stopping a few
reckless companies from competing, and via slowing down competitive
races between leading AI labs.

That safety comes at a cost of increased concentration of power in a few
big companies.

I'll give this scenario a 10% chance.

### Scenario 5: Full Speed Ahead

Foom liability might be ineffective. The benefits of AI could persuade
many companies to pursue powerful AGI regardless of the insurance costs.

This is a clear possibility if companies are allowed to self-insure, or
if small companies are able to compete.

I'll give this scenario a 20% chance.

### Scenario 6: China versus West

I'm unclear whether this scenario is affected by the difference between
liability versus a temporary ban on development, so maybe it doesn't
belong in this post at all. China seems slower than the US to treat
smarter-than-human AI as a near-mode issue. China seems at least as
willing in general to ban new technologies, and somewhat more likely to
enforce those bans effectively. Most likely Chinese concern over
smarter-than-human AI will follow US concern with a delay of a year or
so.

I'd be pretty optimistic about an international agreement if China was
mainly concerned with a commercial balance of power. But I see a strange
interplay between AGI risk and conflict over Taiwan.

My biggest concern is that China will see restrictions on AGI (including
foom liability) as partly an attempt to keep China from participating in
the AI revolution. That would increase the already important pressure on
China to at least blockade Taiwan. The resulting GPU shortage would
delay AI progress by a year or so. That would be a high-risk way of
buying time for safety research.

There will likely be political pressure in the US for advocates of
restrictions on AI to ally with forces that want to cripple China.

I'll treat this as a subset of the Leaky Ban scenario, and not give it
a separate probability.

### Conclusion

There are still many details that would need to be clarified. Imagine
that an AGI manipulates South Korea into liberating North Korea, causing
100k immediate deaths, but the AGI projects doing that will save lives
in the longer run. How do we decide whether to penalize the AGI's
creators? I suspect we get decent incentives whichever way we decide
such questions, as long as we have relatively clear rules for deciding
on those penalties.

A foom insurance requirement looks hard to implement well, but only a
little bit harder than a more direct pause or ban on AGI development.

I'll guess that a foom insurance has a 5% chance of producing an
important safety benefit. Given how precarious our position looks, that
seems like a great deal.