RESEARCH BRIEF

AI Risks Taxonomy
Paving the Path for ConfidenceBuilding Measures
IOANA PUSCASAbout this Brief
This research brief is a short version of the UNIDIR report on risks of artificial intelligence (AI) publishedin October 2023.1
This brief, and the full report it is based upon, are part of the UNIDIR project on Confidence-Building
Measures for Artificial Intelligence, which has two phases. The first phase of the project consisted ofa risk-mapping exercise, which elaborated a comprehensive analysis of risks of AI in the context of international peace and security. The second phase aims to build on this analysis, and to convene multiple
stakeholders to explore options for confidence-building measures (CBMs) for AI that are realistic andfeasible, and drawing on lessons from other processes, and from inputs from diverse actors.

Notes on Scope and Structure
An underlying objective of this project, and implicitly of this phase in the research, is to provide a clear
framework to understand the risks of AI in the context of international peace and security. Various understandings of risks of AI technology have been known and part of the vocabulary of international andmultilateral discussions for the past years, yet a comprehensive mapping and exploration of the risks
landscape has largely been absent from such deliberations.
The taxonomy developed in this study covers key and critical areas of risks of AI, both related to thetechnology, and to its use and effects. To advance discussions about CBMs, and to manage the risksof AI, the international community must develop shared understandings of the risks of the technology,
and how they are related or mutually reinforcing. There are many ways to classify risks of AI and thattask is rendered more complex as AI is a highly scalable, general-purpose technology, with uses acrossdomains and across (weapons) systems. For the purpose of this project, this risk taxonomy accounts
for risks of AI to international peace and security and outlines the main areas of vulnerabilities of thetechnology as well as its potential for misuse, or for escalation in conflict.

Author
Ioana Puscas (@IoanaPuscas1) is Researcher on artificial intelligence with UNIDIR’s
Security & Technology Programme.

1 Full report is available here: https://unidir.org/publication/ai-and-international-security-understanding-the-risks-and-paving-the-path-for-confidence-building-measures/

A I R I S K S TA X O N O M Y

2Overview of the Taxonomy
This taxonomy identifies two large clusters of risks:
RISKS O F AI T E C H N O LO G Y

RI S KS O F A I TO G LO B A L
S E C U R I TY

These include vulnerabilities that are inherent to

These encompass risks that AI technologies pose

how artificial intelligence systems are built andto global security, including risks that emanate

deployed, or which arise in the context of humanfrom the uses of AI in military contexts and ininteraction with AI-enabled systems. Three main

weapon systems, or more broadly, in the context

categories of risks are identified:of AI’s convergence with other technologies anddomains of warfare. This cluster of risks includes:

S A F E TY R I S K S

M I S C A LC U L AT I O N R I S K S

Failures of the technology which are due toUses of AI that lead to incorrect or biased in-inherent limitations of AI systems as technicalterpretations of evolving operational contexts,

systems.

adversary intent, or more generally, of global competition dynamics. Risks of miscalculation are notnew, but AI can magnify their scope and scale.

CY B E R S E C U R I TY R I S K S

E S C A L AT I O N R I S K S

Malicious, intentional attacks on AI systems,

AI can prompt decisions to escalate in conflict,

and which can derail its functioning or outputs inand its potential integration into decision support

some way.

or weapons systems can create direct, accidental
or inadvertent forms of escalation.

H U M A N – M AC H I N E I N T E R ACT I O N
RISKSP R O L I F E R AT I O N R I S K S

Undesired effects or ineffective use of AI systems

cantly enhance risks of proliferation of weapons,due to factors typically encountered in the interaction between humans and AI systems.

AI can alter global security dynamics and signifiincluding weapons of mass destruction.

A I R I S K S TA X O N O M Y

3

AI Risks Taxonomy
EXAMPLES

SPECIFICATIONPROBLEMSSAFETY RISKSBRITTLENESSCONFIDENTIALITY
ATTACKSRISKS OFCYBERSECURITYAI TECHNOLOGYRISKSINTEGRITY ATTACKSAVAILABILITY ATTACKSAUTOMATION BIAS
HUMAN–MACHINEINTERACTION RISKS
TRUST CALIBRATIONRISKS

A I R I S K S TA X O N O M Y

4EXAMPLESAI FORECASTINGTOOLSMISCALCULATION
RISKS
LLMs/MULTI-MODAL
MODELS FOR INTELNUCLEAR RISKSRISKS OF AI TO
GLOBAL SECURITY

ESCALATION RISKSDISINFORMATIONWMDPROLIFERATIONRISKS
AI/AWSA I R I S K S TA X O N O M Y

5Part I. Risks of AI Technology
This cluster of risks includes vulnerabilities that are inherent to how
artificial intelligence systems are built and deployed, or which arise in thecontext of human interaction with AI-enabled systems.

A I R I S K S TA X O N O M Y

61. AI Safety: Inherent Risks of theTechnology
Safety risks are typically defined as unintended

For example, the system can find ways to ‘cheat’

failures of AI systems, causing them to performthe specification by finding an easier method toincorrectly. These are inherent challenges in AI

formally complete the task but in a way that issystems. Safety risks generally mean a systemoutside of the intended goal (this is known as ais not ‘attacked’ by a malicious actor, but it fails‘reward hacking’).

or under-performs due to one or a combinationof factors.

In more complex systems, such as those relying

A common issue of safety across AI appli-

lenges can be mitigated but other issues cancations is brittleness, which means that aemerge, such as spurious correlations, which

system cannot generalize or adapt adequately

are arbitrary connections between variables

to new conditions or when it is presented withthat are not causally related. For example, thenew data. It is this issue that has made manynavigation system in an uncrewed groundAI systems at times appear highly capable invehicle deployed for silent watch operations

testing, only to fail dramatically when deployed

could encounter trees close to a warehouse

under real-world conditions. Some AI systems,

and learn that association although the twosuch as large language models (LLMs) arevariables are likely unrelated.

on neural networks, some specification chal-better at generalizing, yet that capability introduces additional safety testing requirements.
Failures in AI systems may also occur due toso-called task specification issues. Specification in AI refers to the task of conveying to AI
systems what they need to do; in other words,
to translate the intent of the designer of thesystem into specific actions or behaviours of
the system. In practice, aligning human representation of the task to that of a machinelearning (ML)2 system is challenging, and there
are many ways in which mismatches of specification can occur.

2

ML is at the core of modern AI and the two terms are used rather interchangeably in this research.

A I R I S K S TA X O N O M Y

7

2. Cybersecurity Risks
ML systems are vulnerable to cyberattacks typically encountered in other digital domains. These
attacks are generally known by the acronym CIA—confidentiality, integrity, and availability attacks.
The table below provides an overview of the main types of attacks on AI systems.

C I A AT TA C K S

W H AT I S T H E G O A L?

Confidentiality

Confidentiality attacks

HOW IS IT CARRIED OUT? EXAMPLES

extract

hiddeninformation about the model.

Model extraction – this type of attack is carried out
by recording inputs and outputs of the ‘victim’ model
until the attacker obtains enough information so thatthey can recreate a close copy of the stolen model.
Membership inference – studying the inputs andoutputs of the ML system to determine whether acertain data sample was part of the training dataset.
Model inversion – attackers recover (or reconstruct)output categories of the model and try to understandkey features of the input; this type of attack has been
most common for image recognition systems.

Integrity

Integrity attacks compromise an AI system

Data poisoning attacks – attackers manipulate thetypically by altering data in some way. They

training dataset, which causes the system to learn

have received most policy attention to date.

counter-productively.
Evasion attacks – malicious, extremely subtle
changes to the input of the system that lead toerroneous outputs. This can be done with adversarial examples which are perturbations that cause thesystem to change the output (e.g., misclassify an
object). In LLMs, adversarial attacks have exposed
many vulnerabilities, which can steer the models
towards unintended behaviours.

Availability

Availability attacks can lead the ML

Sponge attacks – in compute-intense systems,

component to run slowly or completely stop.

sponge examples exploit the system’s dependency
on hardware and soak up energy, forcing the underlying hardware to underperform.

A I R I S K S TA X O N O M Y

8Limitations of AI systems against cyberattacks

Attacks on AI systems can be relatively easy to

expose vulnerabilities that are inherent to ML

execute, and often require less expertise thanmodels as well as vulnerabilities present acrossbuilding the model in the first place. Vulnerabil-

the technology’s life cycle. Attackers need notities in AI systems cannot always be patched

always break into the model itself as other op-

like in traditional software and although

portunities for attack can be found throughout

defences exist, and they can increase the costthe supply chain. For example, causing a drone

for attackers, a persistent problem remains thatto misclassify targets can be done without

while solving one set of problems, other vulner-

taking control of the drone, for example byabilities may become apparent, or the trade-offs

breaking into the model used by the company(i.e., between safety and performance) are un-that developed the drone.

acceptable. As in the cyber domain, generally,

Attacks on ML are expected to become more
frequent as the technology’s uses expand,as is the range of methods used for attack.

it is broadly recognized that defences for AI
systems offer limited advantages before adversaries move to exploit new vulnerabilities.

Attackers can aim to target many points of vulnerabilities, taking advantage of weaknesses inexisting practices (e.g., transfer learning, whichleverages trained models that are repurposed),or for example, through emerging methods
such as carefully crafted camouflaging techniques specifically designed for AI systems,
which can be used to mislead object detectors.

A I R I S K S TA X O N O M Y

93. Human–Machine Interaction
Risks
Even as a system performs as expected from

There are many factors that contribute to cal-a technical standpoint, there are many chal-

ibrating trust appropriately, ranging from in-lenges of human–machine interaction, which

dividual and situational factors to training

can lead to misuses of the technology. The way

protocols, and system design, including thehumans interact with AI systems comes withdesign of the system’s interface, which plays

important and, in some cases unique, risks thatan important role in the development of situa-

require mitigation at various stages, including

tional awareness and in the ability of the user/

in the interface design of the system and inoperator to monitor and control a system.

training requirements.

Over the past decades, several incidents with

Challenges of human–machine interaction

autonomous systems in the military context

have been explored for over five decades,

have been attributed to problems of human–

as automation and autonomous functions

machine interaction. Operators over- or un-

become more widely embedded in industrial

der-estimated the capabilities of the systemprocesses as well as in weapons systems. The

they were operating and did not intervene at thedevelopment of autonomous vehicles in recent

right time or in the right manner. For example,

years has further expanded the understanding

when the US Army’s Patriot missile systemof risks related to human–machine interaction

was involved in fratricide incidents in the earlyin the context of complex learning systems.

2000s, the system did not fail, technically

A key goal in human–machine interaction istrust calibration, which means effectively
matching a person’s (e.g., operator) trust witha system’s capabilities. In practice, however,speaking. Several issues of human–machineinteraction were at play, including operators
over-trusting the system’s capabilities andpoorly designed interfaces.

trust calibration remains an enduring challenge
not only in the context of human interaction
with autonomous systems, but more broadly,as AI is embedded in a wide range of systems,
including in intelligence analysis (exploredfurther in Part II).
At the lower end, excessive distrust in an AI
system can lead to algorithmic aversion and under-reliance on the technology. Uncritical trust,or over-trust, conversely, manifests as over-reliance on the technology, an issue also known
as automation bias.

A I R I S K S TA X O N O M Y

10Part II. AI and Global Security
This cluster concerns risks that AI technology introduces to global
security. While the first cluster identified risks inherent to AI technology
and in the context of its use, this cluster unpacks risks that AI poses orexacerbates in the context of global security.

A I R I S K S TA X O N O M Y

111. Risks of Miscalculation
The incorporation of AI across intelligence, de-The capabilities of LLMs are also leveraged

cision-support, and forecasting tools means

for battle management software tools andthat AI can impact human decision-making into provide complex and integrated functional-

varied ways, and with direct implications on

ities, combining intelligence collection, interac-decisions to use force. AI’s potential to interfere

tive functions and options for courses of action.

with the entire information space raises the risk

While the use of AI in decision-support is notof miscalculation in warfare, though miscalcu-

new, LLMs may fill a gap in the technology’s

lation can be more broadly conceptualized inprevious shortcomings at providing a more in-the context of global security, including in how

tegrated picture of the operational space.

States assess the capabilities and behavioursof adversaries.

Applications of AI in existing and emerging

AI can be used at scale across intelligencesion-making processes carry significant risks ofprocesses, including in open-source intelli-

miscalculation. Relative to uses of AI for auton-

gence, which makes up an overwhelming partomous weapons, the risks of using AI in intelli-of all intelligence activities in many States

gence and decision-support tools may appear(typically, between 80 to 90 per cent). ML

smaller, yet the consequences can be no lessanalytics can, for example, identify patterns indevastating. Outputs of AI-powered systems

enormous data pools, or extract key information

can impact decisions to use force, includingfrom a wide range of types of media, including

lethal force.

images and videos. Breakthroughs in LLMs,
particularly since 2022, have also been touted
for use in intelligence work. LLMs may be used,
for example, as assistant tools to generatesummaries or to compile intelligence reports
from a large body of sources. LLMs, however,are essentially probabilistic models and cannot
grasp the reasoning processes that underlie intelligence work. Use of AI tools in intelligence
will need careful scrutiny, both in light of thecapabilities that directly serve to inform deci-

At a broader level of analysis, AI can also
introduce

uncertainties

ininternational

relations, globally. Developments in AI cancreate new perceptions of threats and vulnerabilities, and hasten the fielding of AI
systems, including autonomous systems
whose operation in combat may not lend itself
to clear understandings of applicable rules ofengagement.

technology’s current limitations (discussed inPart I) and of, quite simply, its inability at times
to provide useful results.

A I R I S K S TA X O N O M Y

122. Risks of Escalation
Escalation is a central concept in international

AI’s potential for escalation is not limited to therelations, and it refers to the increase in scope

nuclear domain alone. AI can impact how con-and intensity of interactions between States,

ventional weapons are deployed, such as

which can be intentional or unintentional (acci-

by increasing their speed and lethality, which

dental – due to an incorrect use of a weapon, for

may spiral into other pathways for escalation.

example, or inadvertent – when an adversary

For example, a State may feel legitimized toacts intentionally over a threshold it considers

employ nuclear weapons when an adversary

benign, but which is unacceptable for the otherhas gained a disproportionate advantage in itsside).

conventional forces.

AI has raised significant concerns about esca-

Risks of escalation can also be aggravated bylation in the context of international security,

uses of AI to spread disinformation, which can

including by increasing the tempo of warfare incause a range of escalatory reactions in times

a way that complicates de-escalation, by trigger-of crisis or in combat. The use of AI to create hy-

ing accidents that may spiral out of control, or by

per-realistic synthetic videos (‘deepfakes’) or

prompting an escalatory chain of events evenother tools to disseminate deceptive or false in-when used in less-than-lethal applications.

formation can complicate military campaigns,

The use of AI in nuclear command and control,
as well as across the nuclear deterrence ar-including at the tactical level, or increase thelikelihood of pre-emptive attacks.

chitecture, has prompted grave concerns
about AI’s potential role in escalation. While itis difficult to assess the progress of AI integration in nuclear command and control, which isless likely to be swift simply because the technology is not considered predictable and robust
enough, the use of AI appears promising inother areas, such as early warning systems. AI
could be used to identify unusual movements
or to accelerate and improve the processingof sensor data to gauge adversary behaviour.
The risks remain, however, that even below
firing capabilities, the use of AI might paint anincorrect picture of the adversary and raise thealert status.

A I R I S K S TA X O N O M Y

13

3. Risks of Proliferation
AI raises important risks of proliferation of new

Finally, proliferation risks also concern theweapons, including weapons of mass destruc-proliferation of AI itself and of autonomous

tion, either through AI’s convergence with otherweapons. AI software can be repurposed

domains of science and technology, or as aat minimal cost once developed, and it can

result of capabilities enabled by AI itself.

virtually diffuse without restriction, across

The convergence of AI with biology anddomains of use, and across borders.

chemistry holds promise for advances in theDevelopments with AI technology can also

medical field, but has also exposed emerging

enablerisks of weapons proliferation. LLMs used forweapons. The capacity of some contemporary

research and drug discovery—for example,

systems to carry out a number of autonomous

a class of LLMs called ‘chemical language

functions at machine speed already raises

models’ are being used to look for new drugconcerns about escalatory consequences.

molecules—can also create opportunities for
misuse. Further, research has demonstrated
that open-source ML software can be used forthe de novo design of new molecules, including
toxic ones. It is, of course, important to understand that computational proof in the lab
may not be very easily translated into physicalweapons. However, the easing of knowledge
barriers, combined with a growing number ofcommercial companies which provide chemical
synthesis, means the risks of proliferation of
bioweapons cannot be overlooked.
In the cyber domain, AI can be used to enhance
the scope and scale of malicious activities incyberspace. AI can effectively turbocharge
cyber threats, and the integration of ML acrosstechnical systems means that attacks can be
executed at scale. Furthermore, the risks of developing and proliferating malicious code havefurther magnified in the context of LLMs. Guard-

theproliferationofautonomousConcerns about the proliferation of autonomous systems have grown with the surge in useof uncrewed systems over the past decades,
which was also bolstered by advances inother technical domains that enabled relatively low costs of production and deployment.
The breadth of risks multiplies further as morecapable AI-powered systems, fitted with an
expanding array of autonomous functions(including potential for operation without ahuman in the loop) and lethal payloads, can befielded.
Comparative research on loitering munitions,for example, demonstrates a trend for continuous development of autonomous functions,in addition to the use of large and integrated
systems. Some autonomous functions remainlatent for now but that may change in light of operational needs or competitive pressures.

rails normally prevent users from requesting
LLMs to generate malware, but this risk cannot
be ruled out, especially as open-source models
are becoming more widespread.

A I R I S K S TA X O N O M Y

14

Conclusion
This research taxonomized the risks of AI in

To begin a meaningful conversation about

the context of international peace and security

CBMs for AI, it is critical first to map the risks ofin two clusters: first, risks of the technology

the technology and to understand their inherent(safety, security, human–machine interaction

complexity. Risk mitigation must be under-

risks), and second, risks of AI to global security

scored by a comprehensive and informed un-(miscalculation, escalation, proliferation risks).

derstanding of risks. As a next step, efforts to

While it is technically and conceptually possible
to present risks in different categories, it isimportant to highlight that risks are closely in-

explore options for CBMs must engage diverse
stakeholders and ensure that the process isforged through their co-ownership.

terrelated. For example, a safety failure can be
exploited by malicious actors to further compromise the system, or the same vulnerability can be aggravated during use, for example
if operators trust the system too much, or are
inappropriately trained to understand that thesystem is underperforming or that it is effectively under attack.

A I R I S K S TA X O N O M Y

15

@unidir/unidir/un_disarmresearch/unidirgeneva/unidir

Palais des Nations
1211 Geneva, Switzerland© UNIDIR, 2023
W W W. U N I D I R . O R G