# Some Comments on Recent AI Safety Developments

## Overview of New Developments

***Update (9th of August, 2025):*** *I was asked to unpublish and republish this post. Since the original writing of the post, the US military has signed deals with xAI, Anthropic, Google, and OpenAI, with the [xAI deal](https://www.theguardian.com/technology/2025/jul/14/us-military-xai-deal-elon-musk) worth 200 million USD alone. AI and national security entanglement also features heavily in scenarios like [AI 2027](https://ai-2027.com/), which seems likely to become a self-fulfilling prophecy at least in part.*

*At the same time, Chinese AI developers have proven their ability to produce SOTA or close-to-SOTA AI models. While my initial analysis focused on US AI companies, this now no longer seems correct. Given these updates, the dynamics of anti-alignment are likely to play out symmetrically across both sides of the Pacific instead of just in the US. **Unless concerted efforts are taken to deescalate, we may no longer be able to avert an AI military arms race.***

***Edit (4th of December, 2024):*** *OpenAI has now signed up with Anduril, fulfilling my prediction that all three major AI developers will enter into agreements with the US military. See <https://www.anduril.com/article/anduril-partners-with-openai-to-advance-u-s-artificial-intelligence-leadership-and-protect-u-s/>*

As of today, November 9th, 2024, three major AI developers have made
clear their intent to begin or shortly begin offering tailored AI
services to the United States Department of Defense and related
personnel.

-   Anthropic:
    <https://arstechnica.com/ai/2024/11/safe-ai-champ-anthropic-teams-up-with-defense-giant-palantir-in-new-deal/>

-   Meta: <https://scale.com/blog/defense-llama> <https://www.theregister.com/2024/11/06/meta_weaponizing_llama_us/>

-   OpenAI:
    <https://fortune.com/2024/10/17/openai-is-quietly-pitching-its-products-to-the-u-s-military-and-national-security-establishment/>

These overtures are not only directed by AI companies. There are also
[top down
pressures](https://www.bloomberg.com/news/articles/2024-10-24/white-house-pushes-agencies-to-adopt-ai-for-military-spy-uses)
to expand the military use of AI from the US government under Joe Biden.
Ivanka Trump has [posted in favour
of](https://x.com/IvankaTrump/status/1839002887600370145) Leopold
Aschenbrenner's Situational Awareness document, suggesting that the
incoming Trump administration will continue this trend.

For obvious reasons, I believe the same is happening in China, and
possibly (to a lesser extent) other nations with advanced AI labs (UK).
I focus on the American case because most of the most advanced AI labs
are in fact American companies.

## Analysis and Commentary

*Note on terminology: When I say "security community" or "national security community", this is a catch-all term for US Department of Defense personnel, defense contractors, and possibility the military forces of NATO or other allied states (e.g. Five Eyes).*

In some sense this is not new information. In January 2024, the paper
["Escalation Risks from Language Models in Military and Diplomatic
Decision-Making"](https://arxiv.org/abs/2401.03408) warned that
"Governments are increasingly considering integrating autonomous AI
agents in high-stakes military and foreign-policy decision-making". Also
in January, OpenAI removed language [prohibiting the
use](https://theintercept.com/2024/01/12/open-ai-military-ban-chatgpt/)
of its products in military or warfare-related applications. In
September, after the release of o1, OpenAI also claimed that its tools
[have the capabilities](https://openai.com/index/openai-o1-system-card/)
to assist with CBRN (Chemical, Biological, Radiological, or Nuclear
Weapons) development. The evaluation was carried out with the assistance
of experts who were familiar with the procedures necessary for
"biological threat creation", who rated answers based on both their
factual correctness as well as "ease of execution in wet lab". From this
I infer that at least some of these experts have hands-on experience
with biological threats, and the most likely legal source of such
experts is Department of Defense personnel. Given these facts, to
demonstrate such capabilities and then deny the security community
access to the model seems both unviable and unwise for OpenAI.

Furthermore, it seems unlikely that the security community does not have
access already to consumer facing models, given the weak controls placed
on the dissemination of Llama model weights and the ease of creating
OpenAI or Anthropic accounts. Therefore, I proceed from the assumption
that the offerings in these deals are tailor made to the security
community's needs. This claim is explicitly corroborated in the case of
Defense Llama, which claims to be trained on "a vast dataset, including
military doctrine, international humanitarian law, and relevant policies
designed to align with the Department of Defense (DoD) guidelines for
armed conflict as well as the DoD's Ethical Principles for Artificial
Intelligence". The developers also claim it can answer operational
questions such as "how an adversary would plan an attack against a U.S.
military base", suggesting access to confidential information beyond
basic doctrine. We also know that the PRC has developed [similar
tools](https://www.reuters.com/technology/artificial-intelligence/chinese-researchers-develop-ai-model-military-use-back-metas-llama-2024-11-01/)
based on open weights models, further reducing the likelihood of such
offerings not being made to the US military.

## Likely Future Developments

There have been calls for accelerated national security involvement in
AI, most notably from writers such as [Leopold
Aschenbrenner](https://situational-awareness.ai/) (a former OpenAI
employee) and [Dario
Amodei](https://darioamodei.com/machines-of-loving-grace) (CEO of
Anthropic). Amodei in particular favours an "entente" strategy in which
a coalition of democratic nations races ahead in terms of AI development
to outpace the AI developments of autocracies and ensure a democratic
future. These sentiments echo the race to create the atomic bomb and
other similar technologies during World War II and the Cold War. I will
now explore the impact of such sentiments, which appear to be accepted
in security circles given the information above.

The first matter we must consider is what national security involvement
in AI will mean for the current AI development paradigm, which consists
of both open weights, open source, and closed source developers
organised as private companies (OpenAI, Anthropic), subsidiaries of
existing companies (Google Deepmind, Meta FAIR), and academic
organisations (EleutherAI). Based on ideas of an AI race and looming
great power conflict, many have proposed a "Manhattan Project for AI".
My interpretation of this plan is that all further AI development would
be centrally directed by the US government, with the Department of
Defense acting as the organiser and funder of such development. However,
I believe that such a Manhattan Project-style scenario is unlikely. Not
only is access to AI technology already widespread, many of the
contributions are coming from open source or otherwise non-corporate or
non-academic contributors. If acceleration is the goal, putting the
rabbit back in the hat would be counterproductive and wasteful.

Instead, I suggest that the likely model for AI development will be
similar to the development of cryptographic technology in the 20th
century. While it began as a military exercise, once civilian computing
became widespread and efforts to constrain knowledge of cryptanalysis
failed, a *de facto* dual-track system developed. Under this scheme, the
NSA (representing the government and security community) would protect
any [techniques and
advancements](https://en.wikipedia.org/wiki/Differential_cryptanalysis)
it developed to maintain a competitive edge in the geopolitical sphere.
Any [release of technology for open
use](https://en.wikipedia.org/wiki/Data_Encryption_Standard) would be
carefully arranged to maintain this strategic advantage, or only be
conducted [after secrecy was
lost](https://en.wikipedia.org/wiki/Ghidra). At the same time,
developments from the open source, commercial, or academic community
would be permitted to continue. Any successes from the public or
commercial spheres would be incorporated, whether through hiring
promising personnel, licensing technology, or simply implementing their
own implementations now that feasibility was proven. A simple analogy is
that of a one way mirror in an interrogation room: those in the room
(i.e. the public, corporate researchers, academics) cannot look behind
the mirror, but those behind the mirror (the national security
community) can take advantage of any public developments and bring them
"behind the mirror" if necessary. A one way filter is established
between the public sphere and the security sphere, an arrangement we see
mirrored in the ways AI technology is now being licensed, via
IL6-compliant AWS services (in the case of Anthropic) and secured
private servers (in the case of Defense Llama). The public-facing
companies are still active, but a veil of secrecy is established at the
interface with the national security community. *To be clear: there is
no public indication known to me that proprietary model weights are
being shared with the security community. On the other hand, such
agreements would likely live behind the one-way mirror.*

Having established that AI developers will likely continue to exist in
their current form, we must next ask how the development of AI systems
will be influenced by a growing rapproachment with the security
community. It is well known that existing AI alignment efforts are
post-hoc. That is to say, a base or "unaligned" system is created which
has both desired and undesired (dual-use) capabilities, which is then
"brought into compliance" via methods like RLHF and Constitutional AI.
Therefore, the existence of the commercial facing "aligned" models
implies the necessary existence of "base" models, which have not gone
through this post-training modification process. Furthermore, RLHF and
Constitutional AI are both value-agnostic methods. They merely promote
the likelihood of certain types of responses while suppressing others,
with the desired type of response monitored either by an oracle
emulating human feedback or an fully self-supervised model. This means
that it would be feasible to create "anti-aligned" models that feature
*more* undesirable responses. Such models might, for example, specialise
in developing novel cyberweapons or creating plausible propaganda. I
should also expect specialised scaffolding to make use of and enhance
harmful capabilities to be developed. **Amongst these enhanced
capbilities are those most associated with profoundly harmful outcomes,
e.g. the ability to autonomously replicate and take over computer
systems, the ability to develop novel bioweapons, and the ability to
cripple infrastructure.**

Why would these systems be created? While such capabilities would
normally be regarded as harmful per (for example) OpenAI's risk
evaluation metrics, in a military context they are useful, expected
behaviours for an AI system to be regarded as "helpful". In particular,
under the great power conflict/arms race mindset, the possibility of
China or another enemy power developing these tools first would be
unthinkable. On the other hand, the ownership of such a tool would be a
powerful bargaining chip and demonstration of technological superiority.
Therefore, I believe that once the cordon of secrecy is in place, there
will be a desire and incentive for AI developers to produce such
anti-aligned systems tailored for military use. To return to the
cryptography metaphor, while easier methods to break cryptosystems or
intrude into secured networks are regarded as criminal and undesirable
outcomes in general society, for the NSA they are [necessary operational
tools](https://en.wikipedia.org/wiki/PRISM).

Some of you will protest that the standards of the US security community
will prevent them from creating such systems. This is a hypothesis that
can never be tested: Because of the one-way mirror system, we (actors in
the public sphere) will never know if such models are developed without
high-level whistleblowers or leaks. Furthermore, in a great power
conflict context the argument for anti-alignment is symmetric: any great
power with national security involvement in AI development will be aware
of these incentives. Perhaps other powers will not be so conscientious,
and perhaps American corporations will be happy to have two one-way
mirrors
installed: See https://en.wikipedia.org/wiki/Dragonfly_(search_engine\).

## What specifically has changed?

For many of you, these arguments will likely be familiar. However, the
specific agreements between US AI developers and the Department of
Defense, complete with the implementation of the one-way mirror, signals
the proper entry of the US into the AI militarisation race. Even if the
US military does not develop any anti-aligned capabilities beyond those
of commercially available models, other great powers will take notice of
this development and make their own estimations.

Furthermore, the one-way mirror effect is localised: Chinese
anti-alignment efforts cannot benefit from American public-sphere
developments as easily as American anti-alignment efforts can. This is
because of several factors including access to personnel, language and
cultural barriers, and protective measures like information security
protocols and [limits to foreign
access](https://www.rfa.org/english/news/china/openai-07102024145316.html).
Thus far, American AI development efforts are world leading (notice how
the Chinese military uses
[Llama](https://www.reuters.com/technology/artificial-intelligence/chinese-researchers-develop-ai-model-military-use-back-metas-llama-2024-11-01/)
for their purposes rather than local LLM models!). This means that
American anti-alignment efforts, if they ramp up properly, will be world
leading as well.

## Implications for AI safety work

Based on the above, I make several inferences for AI safety work, the
most important of which is this: **Under the present AI development
paradigm, the capabilities of the best anti-aligned AI system will be
lower bounded by the capabilities of the best *public* aligned AI
system.** Notice the asymmetry in knowledge we will possess about
anti-aligned and aligned systems: any advances behind the anti-aligned
systems need not be public, again because of the one-way mirror.

The second inference is this: **Any new *public* capabilities
innovations will be symmetrically applied.** In other words, any attempt
to increase the capabilities of aligned models will be applied to
anti-aligned models so long as alignment is a post-training
value-agnostic process. Any method discovered behind the one-way mirror,
however, need not be shared with the public.

The third inference is this: **Most new *public* alignment work will
also become anti-alignment work.** For example, improvements to
RLHF/post-training alignment or new methods of model control can be
directly employed in service of anti-alignment. Similarly, developments
that make AI systems more reliable and effective are dual-use by their
nature. Mechanistic interpretability and other instrumental science work
will remain as it has always been, effectively neutral. Better
explanations of how models work can likely benefit both alignment and
anti-alignment because of the aforementioned symmetric nature of current
alignment methods.

The final inference is this: ***Public* AI safety advocates are now
definitively not in control of AI development.** While there have always
been AI developers who resisted AI safety concerns (e.g. Yann LeCun at
FAIR), and recently developers like OpenAI have signalled moves away
from an AI safety focus to a commercial AI focus, for a long time it
could be plausibly argued that most consequential AI development is
happening somewhat in the open with oversight and influence from figures
like Yoshua Bengio or Geoffrey Hinton who were concerned about AI
safety. it is now clear that AI safety advocates who do not have
security clearance will not even have full knowledge of cutting edge AI
developments, much less any say about their continued development or
deployment. The age of public AI safety advocates being invited to the
table is over.

There are exceptions to these inferences. For example, if a private AI
developer with no one-way mirror agreement works on their own to develop
a private, aligned model with superior capabilities, this would be a
triumph over anti-alignment. However, not only have all major AI
developers rapidly acquiesced to the one-way mirror arrangement (with
some [notable
exceptions](https://www.theverge.com/2024/8/22/24226161/google-deepmind-staff-call-for-end-to-military-contracts)),
any such development would likely inspire similar anti-alignment
efforts, especially due to the porous nature of the AI development and
AI safety communities. It is also possible that advocates in the
military will resist such developments due to a clear eyed understanding
of the relevant risks, and instead push for the development of positive
alignment technologies with military backing. At the risk of repeating
myself, this is a fight we will not know the outcome of until it is
either inconsequential or too late.

## What should we do?

To be short: I don't know. Many people will argue that this is a
necessary step, that US anti-alignment efforts will be needed to counter
Chinese anti-alignment efforts, that giving autocracies leverage over
democracy in the form of weaponised AI is a death sentence for freedom.
Similar arguments have been deployed by the NSA in defense of its spying
and information gathering efforts. However, others have pointed out the
[delusion](https://www.lesswrong.com/posts/oJQnRDbgSS8i6DwNu/the-agi-entente-delusion)
of trying to control an AI borne of race dynamics and malicious intent,
and point to historical [overreaches and abuses of
power](https://theintercept.com/series/snowden-archive/) by the American
national security community. Indeed, the existence of high-profile leaks
suggests that escape or exfiltration of dangerous AI systems from behind
the one-way mirror is possible, making them a risk even if you have
faith in the standards of the US military. Perhaps now is also an
opportune time to mention the links between the incoming administration
and its ties to [neo-reactionary
politics](https://www.dailymaverick.co.za/article/2024-10-27-some-of-my-best-friends-are-trump-supporters-and-who-on-earth-is-curtis-yarvin/),
as ably demonstrated in the [Project 2025](https://www.project2025.org/)
transition plan.

One thing I think we should all *not* do is continue with
business-as-usual under the blithe assumption that nothing has changed.
A clear line has been crossed. Even if the national security community
renounces all such agreements the day after this post goes live, steps
have been taken to reinforce the existing race dynamic and other nations
will notice. And as for the public? We are already staring at the
metaphorical one way mirror, trying to figure out what is staring back
at us from behind it.