Enabling effective AI policies: Launchof the OECD Framework forClassifying AI Systems
International Conference on AI in Work, Innovation,Productivity and Skills, 22 February 2022

Why classify AI systems?

A variety of systems and policy implications

OECD AI System Definition (OECD, 2019)“An AI system, is a machine-based system that is capable of influencing
the environment by producing an output (recommendations, predictions ordecisions) for a given set of objectives.
i) perceives environments through data or input;ii) abstracts these perceptions into models;iii) uses the models to formulate
options for outcomes.”

OECD Framework for Classifying AI systems:
Key dimensions characterise AI systems’ policy impact
DATA & INPUTECONOMICCONTEXTCONTEXTPEOPLE &PLANETTASK & OUTPUT

AI MODEL

Linking the classification & AI system lifecycle actors
Framework
dimensions

People& Planet

Economic Context

Data& Input

AI ModelTask& Output

ActorsincludeEnd-users &stakeholders

Systemoperators

Data collectors& processors

Developers& modellers

Systemintegrators

Lifecycle
stage

Use or are
impact byPlan& design

Operate& monitor

Collect& process data

Build& use

Build& validate

DeployUses of the OECD AI Classification Framework
APPLICABILITY: Most relevant to classifying specific AI applications, rather than genericAI systems
GOAL: Provide a baseline framework to help support and advance :
1.
2.
3.
4.
5.

a common understanding of AI, and metrics.
structure registries or inventories of AI systems.
sector-specific frameworks, e.g. in healthcare (NICE).
risk assessment and incident reporting (next steps).
risk management & work on accountability along the AI system lifecycle (next steps).

Sincere thanks forinvaluable input to all
who commented• Testing & public consultation May-June 2021:
and tested the> 850 comments & survey responses => Adapted framework.
framework.
PROCESS:• Consensus of group of 60+ experts

Each AI framework
dimension has its
own properties andattributes…

DATA & INPUT- Provenance, collection, dynamic nature- Rights and ‘identifiability’ (personal dataon , proprietary etc.)- Appropriateness and quality
AI actors include data collectors & processors

CONTEXTECONOMIC CONTEXT- Industrial sector- Business function & model- Critical function- Scale & maturity
AI actors include system operators…and involves
specific actors

PEOPLE & PLANET- Users of the system- Impacted stakeholders- Optionality & redress- Human rights,incl. privacy
Human- Well-being & environment- Displacement potentialActors include end-users& stakeholders

TASK & OUTPUT- System task (recognise; personalise etc)
- System action (autonomy level)- Combining tasks and action- Core application areas (computer vision etc)
AI actors include system integrators

AI MODEL- Model characteristics- Model building(symbolic, machinelearning, hybrid)- Model inferencing / use
AI actors include developers& modellersTesting the framework with real AI systems
Key conclusions from survey responses :• The framework is best suited to specificapplications of AI systems rather than to genericAI systems => the more specific the applications,the more consistent the survey responses.
• Respondents were better at classifying criteria inPeople & Planet and Economic Context.
Classifying Data & Input, AI Model, and Task &Output often requires more technical information
than is available publicly.

DATA &INPUTECONOMICCONTEXTPEOPLE &PLANETTASK & OUTPUTAI MODELExample 1: Credit-scoring AI systems
Selected criteria:• System users – Amateur (bank employee)• Optionality – Cannot opt out• Human rights impact – Yes• Sector of deployment – Financial system (e.g., banking, insurance)• Critical function – Critical function/activity (availability of financial services, inclusion)• Data collection – Human (set of rules) and automated sources (e.g. profiles, loan payments)• Rights – Mix of proprietary and public data
• “Identifiability” – often personally identifiable data• Model building – e.g., statistical/hybrid model; learns from provided data, augmented byhuman knowledge• Model evolution – Can evolve during operation• System task – Forecasting: uses past & existing behavior to predict future outcomes• Level of action autonomy – Medium (human on-the-loop)Example 2: GPT-3, text generation
Selected criteria:
Caveat: general purpose AI system, so nearly all responses depend on thespecific application context! Medical advice, content filter, creative writing…
• System users – Primary users are amateur• Impacted stakeholders – workers, consumers• Sector of deployment – Information & communication• Critical function – None
• Data collection – Human sources (text strings)• Rights – Largely public data sources (some proprietary)• Model building – Learn from provided data
• Model evolution – Evolution during operation• System task – Goal-driven optimization, Reasoning with knowledge structures, interaction
support, recognition, personalisation• Level of action autonomy – Low autonomy [human action required e.g., to use
generated text]

Using the framework to frame evidence standards for healthcare
1Scoping Review

2Questionnaire & Interview
Study

3

Delphi Consensus Study

4

Public Consultation• OECD classification framework was
independently ranked as most complete
system from a shortlist of 21 candidates
when mapped against the 9 core domainsof HTA (EUnetHTA)• Highest rated by a global multistakeholder
panel of experts in both a questionnaire
and interview study• Framework now pivotal in developing
evidence standards to underpin postmarket evaluation in UK health sector

Using the framework for health technology assessment

Clinical liability

Current use
Changes to data flows
Local deployment factors
Local performanceEnd-user training
End-user well being
End-user/patient autonomy
Consent
Communication strategies

Input description
Model reporting
Training data reporting
Performance in different
groupsHandling outlier data
Harm
Autonomy
Output description
Post deployment change
management plan
Benchmarking
Oversight committees

Next steps at the OECD:• Refine classification criteria• Add more real-world AI systems and identify possible indicators• Develop a risk assessment framework to facilitate global interoperability• leveraging the classification plus possible governance at the corporate, institutionor AI systems level• Leveraging work in partner organisations, including EU, US, ISO• Leveraging risk assessment work in other parts of the OECD
• Develop a common framework for reporting about AI incidents.

• Support risk management: Inform related work on mitigation, complianceand enforcement along the AI system lifecycle, and responsible business-impactassessment.
18Reminder: Risk categorisationof uses of AI in the draft AI Act of the EU

Details will be needed on European
regulation/standardisation aspects …
• Rules:
Mapping of uses of AI to these four categories (detailedrules, examples …)• Governance:
AI integration into / extension of frameworks for –- risk management- risk mitigation• Tools:
Assessment tools to operationalize mapping rules, due
diligence, AI governance etc.

Harmonised standards
created by ESOs,
especially
CEN-CENELEC JTC21

Mix of actors,
including private
sector, European
Commission, NGOs,…

For more information visit
www.oecd.ai/classification
email: ai@oecd.org