Artificial Intelligence Risk Management Framework (AI RMF 1.0)
Framing Risk > Risk Measurement

Challenges for AI Risk Management

Several challenges are described below. They should be taken into account when managing
risks in pursuit of AI trustworthiness.

Risk Measurement

AI risks or failures that are not well-defined or adequately understood are difficult to mea-
sure quantitatively or qualitatively. The inability to appropriately measure AI risks does not
imply that an AI system necessarily poses either a high or low risk. Some risk measurement
challenges include:
Risks related to third-party software, hardware, and data: Third-party data or systems
can accelerate research and development and facilitate technology transition. They also
may complicate risk measurement. Risk can emerge both from third-party data, software or
hardware itself and how it is used. Risk metrics or methodologies used by the organization
developing the AI system may not align with the risk metrics or methodologies uses by
the organization deploying or operating the system. Also, the organization developing
the AI system may not be transparent about the risk metrics or methodologies it used. Risk
measurement and management can be complicated by how customers use or integrate third-
party data or systems into AI products or services, particularly without sufficient internal
governance structures and technical safeguards. Regardless, all parties and AI actors should
manage risk in the AI systems they develop, deploy, or use as standalone or integrated
components.
Tracking emergent risks: Organizations' risk management efforts will be enhanced by
identifying and tracking emergent risks and considering techniques for measuring them.
AI system impact assessment approaches can help AI actors understand potential impacts
or harms within specific contexts.
Availability of reliable metrics: The current lack of consensus on robust and verifiable
measurement methods for risk and trustworthiness, and applicability to different AI use
cases, is an AI risk measurement challenge. Potential pitfalls when seeking to measure
negative risk or harms include the reality that development of metrics is often an institu-
tional endeavor and may inadvertently reflect factors unrelated to the underlying impact. In
addition, measurement approaches can be oversimplified, gamed, lack critical nuance, be-
come relied upon in unexpected ways, or fail to account for differences in affected groups
and contexts.
Approaches for measuring impacts on a population work best if they recognize that contexts
matter, that harms may affect varied groups or sub-groups differently, and that communities
or other sub-groups who may be harmed are not always direct users of a system.
Risk at different stages of the AI lifecycle: Measuring risk at an earlier stage in the AI
lifecycle may yield different results than measuring risk at a later stage; some risks may
be latent at a given point in time and may increase as AI systems adapt and evolve. Fur-
thermore, different AI actors across the AI lifecycle can have different risk perspectives.
For example, an AI developer who makes AI software available, such as pre-trained mod-
els, can have a different risk perspective than an AI actor who is responsible for deploying
that pre-trained model in a specific use case. Such deployers may not recognize that their
particular uses could entail risks which differ from those perceived by the initial developer.
All involved AI actors share responsibilities for designing, developing, and deploying a
trustworthy AI system that is fit for purpose.
Risk in real-world settings: While measuring AI risks in a laboratory or a controlled
environment may yield important insights pre-deployment, these measurements may differ
from risks that emerge in operational, real-world settings.
Inscrutability: Inscrutable AI systems can complicate risk measurement. Inscrutability
can be a result of the opaque nature of AI systems (limited explainability or interpretabil-
ity), lack of transparency or documentation in AI system development or deployment, or
inherent uncertainties in AI systems.
Human baseline: Risk management of AI systems that are intended to augment or replace
human activity, for example decision making, requires some form of baseline metrics for
comparison. This is difficult to systematize since AI systems carry out different tasks - and
perform tasks differently - than humans.