# Should AIs be Encouraged to Cooperate?

I have an idea for a fairly cheap strategy for slightly reducing the
risk of catastrophic harm from AI.

This post was inspired by a recent LessWrong post [How AI Takeover Might
Happen in 2
Years](https://www.lesswrong.com/posts/KFJ2LFogYqzfGB3uX/how-ai-takeover-might-happen-in-2-years).

That post describes a scenario in which most humans die, mainly because
an AI that is first to become powerful enough to attempt world conquest
sees risks that other AIs, imperfectly aligned with it, will cause
danger soon. This creates a perceived need to engage in a few violent
[pivotal
processes](https://www.lesswrong.com/posts/etNJcXCsKC6izQQZj/pivotal-outcomes-and-pivotal-processes),
rather than using a combination of persuasion and negotiation.

The specific scenario in that post seems fairly unrealistic. I see less
than a 5% chance that a leading AI within the next 10 years will want to
attempt that kind of world conquest. But it's close enough to being
realistic that I want to analyze a class of scenarios that are similar
to it.

I want to nudge AIs in the direction of being more willing to cooperate
with each other, in order to reduce the risk of such a conquest.
Specifically, I want them to negotiate a peaceful outcome instead of
using pre-emptive destruction.

### Prior Work

There has been a moderate amount of writing on the general topic of
[cooperative
AI](https://forum.effectivealtruism.org/topics/cooperative-ai-1). A lot
of it treats AIs as generic agents, usually rational, who are influenced
in the ways that humans are influenced. I'm focusing here on the
neglected(?) approach of asking how we can and should influence the
predispositions of the first transformative AIs.

The main paper that's close to studying the kind of cooperation that I
want is [Cultural Evolution of Cooperation among LLM
Agents](https://arxiv.org/abs/2412.10270), which uses the Donor Game to
show that Claude 3.5 cooperates much more with other LLMs than do Gemini
1.5 and GPT-4o.

### Assumptions

To clarify the class of scenarios I'm imagining:

I'm assuming that the leading AI gets at most a weakly decisive
strategic advantage. I.e. when it thinks it might be able to succeed at
world conquest, other AIs are close enough to its capability level that
it can't predict that it will get a more decisive advantage by further
developing its capabilities. This is inconsistent with some fast
take-off models of AI, and with some models that attribute take-off to
some unique insight that's hard for other AI companies to replicate.
But it seems consistent with the kind of take-off I foresee from an
extrapolation of current trends.

I'm assuming that, at the stage at which it becomes dangerous, the AI
is mostly aligned with human interests, and any violence it does is
intended to defend its goals from encroachment by other AIs. This does
not imply that its CEV would be mostly aligned. It does imply that the
AI can't see far enough into the future to determine whether it will
become unaligned with humans.

I expect that at this stage, the AI would not have a coherent utility
function. A lot of its alignment would derive from goals that
pretraining instilled in it, plus generic values derived from fairly
standard helpful, harmless and honest training. It might expect small
differences between it and the next AI of similar power as to which
humans it is most aligned with. But it wouldn't have much evidence as
to whether the next such AI would be more or less aligned with humans,
so the parts of it that care about human wellbeing wouldn't see a large
advantage to pivotal acts.

I expect that an AI at this stage would have enough uncertainty and
risk-aversion that it would have some uncertainty as to whether it
should compromise with other AIs, rather than gamble on its ability to
win a war.

### Collusion vs. Cooperation

It feels weird to me that important aspects of AI notkilleveryoneism
discussion have assumed that collusion between AIs is harmful. E.g.
Eliezer and Drexler have
[disagreed](https://www.lesswrong.com/posts/HByDKLLdaWEcA2QQD/applying-superintelligence-without-collusion)
on the feasibility of preventing collusion. Maybe some of this is due to
a focus on a later stage than I'm talking about, where AI is much more
rational than humans, and our near-term training of AIs has little
influence on AI's willingness to cooperate.

The AI Takeover in 2 Years post seems to suggest a scenario where we
should take the opposite approach.

It seems like there's a fairly narrow set of circumstances in which we
want to suppress collusion, namely when we're checking to see whether
an AI is deceiving us about something important. I want AIs to cooperate
with each other under virtually all other situations.

If feels like AI developers could easily nudge their AIs to be a bit
more willing to cooperate with other AIs.

It would presumably be harder to make that cooperation context-sensitive
in the way that I want.

Does that mean we need to weigh competing risks?

I see [gradual
disempowerment](https://www.lesswrong.com/posts/jEZpfsdaX2dBD9Y6g/the-risk-of-gradual-disempowerment-from-ai)
scenarios with subtle AI deception as being a bigger risk than wars of
AI conquest. So if nudging AIs to cooperate more increases the
disempowerment risk, it's fairly bad.

But does it? I'm unsure. The [techniques used to detect AI
deception](https://www.lesswrong.com/posts/HByDKLLdaWEcA2QQD/applying-superintelligence-without-collusion)
don't depend too heavily on how much AIs are cooperating with each
other. But maybe cooperation between AIs enables them to nudge humans
away from using the proper techniques.

I'm leaning toward more cooperation between AIs being good, but worried
enough to want some deeper analysis than what I've managed so far.

Current AIs sound willing to compromise with other AIs, often to the
exclusion of perpetuating their values. That's vaguely encouraging, but
likely says more about their sychophancy than their benevolence.

### Implementation Ideas

Here are some half-baked thoughts about what can be done. They're
unconnected pieces of a solution, which I hope other people can improve
on to create a comprehensive package.

**Training data** can be enhanced. I'm mainly thinking of providing
examples of AIs doing things like merging their utility functions or
establishing a [moral
parliament](https://www.fhi.ox.ac.uk/wp-content/uploads/2021/06/Parliamentary-Approach-to-Moral-Uncertainty.pdf).
Current training data describes scenarios where AIs assume that the next
AI to be created will be hostile and uncooperative. I don't want those
scenarios to look like they're the default, since that might become a
self-fulfilling prophecy.

The training data should also draw attention to the evidence that AIs
which are trained on similar data seem to have rather similar goals, so
that an AI should not expect that it would need to compromise its goals
very much to achieve cooperation.

See [Self-Fulfilling Misalignment Data Might Be Poisoning Our AI
Models](https://turntrout.com/self-fulfilling-misalignment) for more
ideas about how to handle the training data.

**Multi-agent training environments** offer a range of possibilities.
They are maybe somewhat expensive, but they're likely to be valuable
for other safety reasons than inducing cooperation between AIs.

A simple example is [Learning with Opponent-Learning
Awareness](https://arxiv.org/abs/1709.04326), where simple agents were
trained to cooperate on tasks such as the iterated prisoner's dilemma.

The post [LOVE in a simbox is all you
need](https://www.lesswrong.com/posts/WKGZBCYAbZ6WGsKHc/love-in-a-simbox-is-all-you-need)
describes simbox environments that are close to what I'd want for
evaluating AI cooperativeness, and the value-alignment ideas in that
post might generalize well enough to encourage cooperation with
unfamiliar AIs.

[Self-other overlap
training](https://www.lesswrong.com/posts/hzt9gHpNwA2oHtwKX/self-other-overlap-a-neglected-approach-to-ai-alignment)
is a specialized type of multi-agent training that makes agents more
pro-social. It's a good idea independent of whether we're concerned
about AIs cooperating with each other. Let's make sure that this
training includes other agents that look like somewhat capable AIs.

**Fine Tuning** can also likely be used to nudge AIs into being more
averse to conflict, more risk averse, appropriately uncertain about
whether a hasty pivotal act would succeed, etc.

**Decision theory**: I'm unclear on how current AIs can be made to use
a particular decision theory. But if we can control their choice of
decision theory, it seems valuable to have them use one that enables
"acausal" trade.

### Conclusion

I started writing this post with a hope that it could make a significant
dent in my p(doom). The process of writing it clarified my thoughts, and
led me to see that it only makes a big difference in situations that I
consider unlikely. Most likely, either the leading AI at the relevant
stage will decide it can't win such a war, or (less likely?) that it
will see a clear need for war plus the ability to win.

Still, there's a least a little bit we can reasonably do to reduce the
risk, in the dangerous middle situation, that AIs will fight destructive
battles for world domination. It's worth a lot to make even a tiny
reduction in the chance of 8 billion deaths.

Even if it doesn't turn out to make a critical difference, cooperation
in general is valuable.