Human Rights,
Democracy,
and the Rule of Law
Assurance Framework
for AI Systems:
A proposal prepared for theCouncil of Europe’s Ad hoc
Committee on Artificial
Intelligence1

Table of Contents
Abbreviations ................................................................................. 4
Introduction and Overview ............................................................... 5

A Proposal for a Coherent and Integrated Approach................................................................. 6
Shape and Delivery of the Framework............................................................................................ 8

Preliminary Context-Based Risk Analysis (PCRA) ................................ 12
Preparing a Project Summary Report in the Planning and Scoping Phase ...... 12

Activity 1: Outline Project, Use Context, Domain, and Data................................................ 14
Activity 2: Identify Stakeholders..................................................................................................... 15
Activity 3: Scope Impacts to Human Rights, Democracy, and Rule of Law ................... 17
Activity 4: Map Governance Workflow .......................................................................................... 43

Project Summary Report Template .................................................. 46
PCRA: Document Overview.................................................................... 57

Introduction ............................................................................................................................................. 57
PCRA Section One: Identifying Risk Factors ............................................................................... 58
PCRA Section Two: Risks of Adverse Impacts on the Human Rights and Fundamental
Freedoms of Persons, Democracy, and the Rule of Law ........................................................ 61

Preliminary Context Based Risk Analysis Template ............................. 71
PCRA Output Example (Please access accompanying video file to view digitalplatform interface for the completion of the PCRA).................................. 203

Stakeholder Engagement Process ................................................... 211
Step 1: Carry Out Stakeholder Analysis ................................................. 212
Step 2: Engage in Positionality Reflection .............................................. 214
Step 3: Establish an Engagement Objective ........................................... 215
Step 4: Determine Engagement Method ................................................ 216
Step 5: Workflow Revisitation and Reporting.......................................... 220

Stakeholder Engagement Process (SEP) Template ............................ 223
Human Rights, Democracy, and the Rule of Law Impact Assessment ... 227
Getting the HUDERIA Process Right ...................................................... 228
Three Steps in the HUDERIA Process .................................................... 232
Building the Content of your HUDERIA .................................................. 232

Identifying Potential Adverse Impacts ........................................................................................ 233
Assessing the Severity of Potential Impacts............................................................................. 234
Additional Questions for the Project Team ................................................................................ 241
Mitigating Potential Impacts and Setting Up Access to Remedy....................................... 242

Revisiting the HUDERIA across the Project Lifecyle ................................. 245
HUDERIA Template ............................................................................ 247

2

Human Rights, Democracy, and Rule of Law Assurance Case (HUDERAC)................................................................................................. 273
Section Outline .................................................................................. 273
Introduction ...................................................................................... 274
What is an Assurance Case? ................................................................ 274
Structure of a HUDERAC ..................................................................... 276
Components of a HUDERAC ................................................................. 277

Goal(s)..................................................................................................................................................... 277
Properties and (Project or System Property) Claims ............................................................. 279
Evidence ................................................................................................................................................. 286

Building a HUDERAC ........................................................................... 287
A HUDERAC Template ......................................................................... 289

Reflect...................................................................................................................................................... 289
Act ............................................................................................................................................................. 290
Justify ...................................................................................................................................................... 291
Goals, Actions and Claims ............................................................................................................... 292

Appendix 1: Summary of Processes, Steps, and User Activities ........... 301
Appendix 2: HUDERAF Project Lifecycle Process Map ......................... 302
Appendix 3: A Sociotechnical Approach to the ML/AI Project Lifecycle .. 303(Project) Design Tasks and Processes ................................................... 305

Project Planning ................................................................................................................................... 305
Problem Formulation .......................................................................................................................... 306
Data Extraction or Procurement .................................................................................................... 307
Data Analysis ........................................................................................................................................ 307(Model) Development Tasks and Processes ............................................ 307

Preprocessing and Feature Engineering ..................................................................................... 307
Model Selection .................................................................................................................................... 308
Model Training ...................................................................................................................................... 308
Model Validation and Testing .......................................................................................................... 308
Model Reporting ................................................................................................................................... 308(System) Deployment Tasks and Processes ........................................... 309

Model Productionalization ................................................................................................................ 309
User Training......................................................................................................................................... 309
System Use and Monitoring ............................................................................................................ 309
Model Updating or De-provisioning .............................................................................................. 310

Glossary ..................................................................................... 311
References .................................................................................. 321

3

Abbreviations
CAHAI
CoE
DHRI
HUDERAC
HUDERAF
HUDERIA
HRIA
IMP
PCRA
PS Report
RIN
SEP
UNGP
UNHROHCCouncil of Europe’s Ad hoc Committee on
Artificial Intelligence
Council of Europe
Danish Human Rights Institute
Human Rights, Democracy, and the Rule of
Law Assurance Case
Human Rights, Democracy, and the Rule of
Law Assurance Framework
Human Rights, Democracy, and the Rule of
Law Impact Assessment
Human Rights Impact Assessment
Impact Mitigation Plan
Preliminary Context Based Risk Assessment
Project Summary Report
Risk Index Number
Stakeholder Engagement Process
United Nations Guiding Principles on Businessand Human Rights
UN Human Rights Office of the HighCommissioner

4

Introduction and Overview
Dear members of the CAHAI-PDG and other esteemed colleagues and excellencies,

What follows is a proposal for a Human Rights, Democracy, and the Rule of Law
Assurance Framework (HUDERAF) for AI systems that endeavours tooperationalize the outline of a Model for a Human Rights, Democracy, and the Ruleof Law Impact Assessment (HUDERIA) presented in CAHAI-PDG(2021)05rev. The
purpose of the latter document was to “define a methodology to carry out impact
assessments of Artificial Intelligence (AI) applications from the perspective ofhuman rights, democracy, and the rule of law, based on relevant Council of Europe(CoE) standards and the work already undertaken in this field at the internationaland national level…, and to develop an impact assessment model.”The challenge put forward in CAHAI-PDG(2021)05rev was fourfold:
1. To develop an impact assessment model that integrates the practices ofhuman right due diligence contained in general Human Rights Impact
Assessments (HRIAs) with AI-centred approaches to algorithmic impact
assessment and the assurance of trustworthy AI innovation practices,
2. To apply a risk-based approach guided by the precautionary principles todetermine use-cases that trigger pre-emptive measures where high risks tohuman rights, democracy and the rule of law are present and cannot be
mitigated,
3. To formulate a methodology of impact assessment that follows the
proportionality principle, namely, that helps AI innovators and project
teams to establish a proportionate approach to the impact assessment
process and to the practices of stakeholder engagement, risk management,
impact mitigation, and innovation assurance that derive therefrom, and
4. To develop a methodology for assessing and grading the likelihood andextent of risks associated with an AI system that takes into account theuse-contexts and purposes of the system, its underlying technology, theactors involved in the production of the system and its stage ofdevelopment, and the views of potentially impacted stakeholders.
As stated in CAHAI-PDG(2021)05rev, taken together, these desired elements ofthe HUDERIA demand a “coherent and integrated” approach: “The model for a
HUDERIA should provide a coherent and integrated approach for assessing
adverse impact on human rights, democracy and the rule of law generated by AI
systems, addressing simultaneously the risks arising from the specific andinherent characteristics of AI systems and the impact of such systems on humanrights, rule of law and democracy.” A four-step process is suggested to achieve
this:

5

STEP ONE: Identify relevant human rights—and rights proxies for
democracy and the rule of law—that could be adversely impacted
STEP TWO: Assess the impact on those rights, encompassing bothtechnical and socio-technical dimensions
STEP THREE: Assess governance mechanisms to ensure the mitigation ofrisks,
appropriate
stakeholder
engagement,
effective
remedy,accountability, and transparency
STEP FOUR: Continuously monitor and evaluate the system to ensure that
impact assessment, impact mitigation, and governance mechanisms are
sufficiently responsive to changes in context and operating environment.
The framework we present here builds on and further develops these important
ideas in the ends of constructing a cohesive set of practicable tools and processes
that bring the HUDERIA to life. One of the main challenges we have faced in trying
to do this has been that the “coherent and integrated” approach needed to bringall of the HUDERIA functions and purposes together requires that several different
streams of activity be developed independently but, at the same time, remainfunctionally interwoven. We have focused much of our efforts on working out how
best to integrate distinctive tasks of risk analysis, stakeholder engagement,
impact assessment, risk management, impact mitigation, and innovation
governance and assurance into a coherent and integrated whole.
Another related challenge has been figuring out, in accordance with the goals set
out in CAHAI-PDG(2021)05rev, how to combine technical and socio-technicaldesiderata for responsible AI innovation with the procedural requirements forprinciples-based human rights due diligence. There is no clear precedence for thiskind of synthesis, and building the connective tissue between the evaluative anddialogical processes needed to establish human rights due diligence and thepractical principles and governance mechanisms needed to set up guardrails forresponsible AI innovation practices is a complex undertaking.

A Proposal for a Coherent and Integrated Approach
Our strategy in confronting these challenges has been to build several clearly
articulated but sensibly interrelated processes (and instruments) that cover all of
the functions and goals that were sketched out by the PDG. The result is the
proposed Human Rights, Democracy, and the Rule of Law Assurance Framework(HUDERAF). There are four main elements of the HUDERAF:
1. The Preliminary Context-Based Risk Analysis (PCRA) provides aninitial indication of the context-based risks that an AI system could pose tohuman rights, fundamental freedoms, and elements of democracy and therule of law. The main purpose of the PCRA is to help project teams establish
a proportionate approach to risk management and assurance practices andto the level of stakeholder engagement that is needed across the project
lifecycle. The PCRA process begins with a set of project scoping andplanning activities in which members of the project team (a) consolidate
information about the project-, use-, domain-, and data-contexts of theprospective system, (b) identify relevant stakeholders, (c) begin to scope6the potential impacts of their systems on human rights, democracy, and therule of law by identifying salient rights and freedoms, and (d) map thegovernance workflow.
After these activities are recorded in a Project Summary Report, the project
team is ready to complete its PCRA. The PCRA is a set of questions andprompts that help project teams to pinpoint risk factors and potential
adverse impacts of their systems on human rights, fundamental freedoms,
and elements of democracy and the rule of law. When the PCRA is finished,
it automatically generates a report (through triggers built on simple
conditional rules) that directs project teams to specific actions that they
then need to take in their impact assessment process (HUDERIA) so thatsufficient consideration can be given to each of the risk factors detected
and to salient rights and freedoms that could be impacted by theprospective AI system. It also directs them to specific goals, properties, andareas that they should focus on in their subsequent risk management andassurance processes (HUDERAC) to reduce and mitigate associated risks.
The report additionally assists project teams to understand, in a provisionalway, the risk level of their system, so that they can take a proportionate
approach to risk management actions and stakeholder engagement.
2. The Stakeholder Engagement Process (SEP) helps project teams toidentify stakeholder salience and to facilitate proportionate stakeholder
involvement and input throughout the project workflow. This process
safeguards the equity and the contextual accuracy of the PCRA and othersubsequent governance processes through stakeholder involvement,
revisitation, and evaluation. The SEP involves a set of steps that include (a)carrying out stakeholder analysis, (b)engaging in positionality reflection, (c)establishing engagement objectives, (d) determining engagement
methods, and (e) involving rights-holders in workflow revisitation andreporting.
3. The Human Rights, Democracy, and the Rule of Law Impact
Assessment (HUDERIA) provides an opportunity for project teams andengaged stakeholders to come together to produce detailed evaluations of
the potential and actual impacts that the design, development, and use of
an AI system could have on human rights, fundamental freedoms andelements of democracy and the rule of law. This process (a) contextualizes
and corroborates potential harms that have been previously identified, (b)enables the discovery of further harms through the dialogical integration of
stakeholder perspectives, (c) makes possible the collaborative and contextsensitive assessment of the severity of potential adverse impacts identified,(d) facilitates the co-design of an impact mitigation plan and sets up accessto remedy, and (e) establishes monitoring and impact re-assessment
protocols.
4. The Human Rights, Democracy, and Rule of Law Assurance Case
(HUDERAC) enables project teams build a structured argument that7

provides demonstrable assurance to stakeholders that claims about theattainment of top-level normative goals (like safety, sustainability,
accountability, and fairness) in processes of building and using an AI system
are warranted given available evidence. Building an assurance case involves
three iterative steps: (a) establishing the goals that are requisite for risk
management and impact mitigation and determining the properties needed
to assure these goals, (b) taking actions to operationalize these properties
in the design, development, and use of the system, and (c) compilingevidence of these actions. The process of developing an assurance caseassists internal reflection and deliberation, promoting the adoption of bestpractices and integrating these into design, development, and deployment
lifecycles. It also provides an accessible way to convey all of this information
to impacted stakeholders. A diligently executed assurance case offers aclear and understandable mechanism of risk management and impact
mitigation that supports appropriate levels of public accountability andtransparency.

Shape and Delivery of the Framework
Seen as a single project governance regime—which provides assurance that theuse of an AI system, as well as the processes behind its design, development anddeployment, respect human rights, fundamental freedoms, democratic principles,
and the rule of law—the proposed HUDERAF can be viewed as a collection ofinterrelated processes, practical steps, and user activities:

Summary of Processes, Steps, and User Activities

A larger image of this Summary of Processes can be found in Appendix 1.

Beyond such a summary view, this set of processes, steps, and user activities canalso be seen as extending across the AI design, development, and deployment
lifecycle. The production and implementation of AI technologies is a multi-phased,
iterative, and often cyclical undertaking, and so effectively safeguarding thealignment of that complex enterprise with human rights, democracy, and the ruleof law necessitates that risk analysis, impact assessment, and assurance methods
are sufficiently responsive to the dynamic character of the AI innovation

8

workflow. 1 The HUDERAF has been developed with this in mind. Stakeholder
Engagement, HUDERIA, and HUDERAC processes are iterative and organised torespond appropriately to changes in production and implementation factors as wellas in the operating environments of AI systems. Here is schematic mapping ofwhat this looks like as such processes and steps move from the design to thedevelopment to the deployment phases of a project workflow:

HUDERAF Project Lifecycle Process Map

A larger image of this Project Lifecycle Process Map can be found in Appendix 2.

It should be noted that, as we have been constructing the elements of thisframework, we have come to believe that the delivery of its processes andinstruments would be best served in the medium of a single digital platform thatallows for the myriad governance goals articulated by the CAHAI and its Feasibility
Study to be all in one place and as user friendly as possible. As they are presented
here, the description and explanation of each process (PCRA, SEP, HUDERIA,
HUDERAC) is accompanied by a template that could be made into a simple andeasy-to-use form. Each template consolidates the questions, prompts, and actions
that need to be covered for the completion of the process at each step along theway. The PCRA Template, in particular, has been created as an interactive digitalform that generates bespoke summary reports based upon conditional/branching
For an overview of the AI/ML project lifecycle through a socio-technical lens, see
Appendix 3.

1

9

logic and the compiling of results. These templates could be brought together andinterconnected in one digital environment, so that users could have a streamlined
way to ensure that their AI innovation practices properly align with human rights,
fundamental freedoms, democracy, and the rule of law and to readily demonstrate
this to potentially impacted stakeholders or their proxies.
A final comment on our mode of presentation: This proposal has been written in
the style of a user’s guide or technical manual, so that all the elements of theframework could be presented with detailed explanations and elaborations of therationale behind the mechanisms, tools, and concepts chosen. This reflects moreof our attempt to convey the details of the framework to the CAHAI and CoEreadership than a representation of what a finished product might look like.
Pragmatically speaking, presenting the framework in this way will make theincorporationofpolicy-level
decisionsandadjustments
fromthe
CoE/PDG/LFG/plenary much easier for those elements of the framework that are
deemed worthy of including.

Working Notes for the PDG
As we have been working to develop these processes and instruments, we have
hewed closely to the findings and policy content presented in the CAHAI’s
Feasibility Study, to the subsequent multi-stakeholder consultation, and to other
CAHAI and Council of Europe documents. This notwithstanding, we ask reviewers
and readers of this proposal to make sure that areas (where interpretive choices
have had to be made) line up properly with the policy goals and developments of
the PDG, the LFG, and the CAHAI, more generally. In building these processes andtemplates, our goal has been to provide the CAHAI with a coherent frame on which
to hang its policy decisions and through which it could operationalize its
governance goals, rather than to put forward any policy innovations as such.
Similarly, we have done our best to reflect the most recent positions articulatedby the CAHAI on open policy issues (such as redlines and triggers of the
precautionary principle). We ask that reviewers and readers keep this in mind, sothat any necessary adjustments to current categories or policy content can be
spotted and made.
Substantively, we ask that reviewers and readers look at Section 2 of the PCRA
(Risks of Adverse Impacts on the Human Rights and Fundamental Freedoms of
Persons, Democracy, and the Rule of Law) closely. In this section, we tried to build
out a semi-quantitative method of risk analysis that generates provisional
recommendations regarding proportionate risk management actions andstakeholder engagement. Coming up with a metric to support the determinationof risk-based project governance was a thorny endeavour, and we ended up
integrating a very straightforward and user-accessible approach to risk analysis(the calculation of a risk index number [RIN]) with (1) the risk management
priorities articulated in the UNGP, the UNHROHC, and the CoE’s Committee of
Ministers and (2) the calibration mechanism of the precautionary principle. We
also considered using sums and combinations of the risk factors identified in
Section 1 of the PCRA for this metric, and this may still be an attractive option,
should it be decided that the method used in Section 2 is not fit-for-purpose. In10any case, we ask that you keep an eye on this part of the framework as it breaks
ground into unexplored territory.
We would, finally, just like to apologize, in advance, to readers for the length ofthe present work and thank them for taking the time to look over the materials
contained herein. We have had to cover much territory to animate the aspirations
outlined by the PDG, and we are grateful to the reader for accompanying us on
this journey to advance the CAHAI’s historical mission.

11

Preliminary Context-Based Risk Analysis(PCRA)

Introduction
The Preliminary Context-Based Risk Analysis (PCRA) provides an initial indicationof the context-based risks that an AI system could pose to human rights,
fundamental freedoms, and elements of democracy and the rule of law. The main
purpose of the PCRA is to help project teams establish a proportionate approach torisk management and assurance practices and to the level of stakeholder
engagement that is needed across the project lifecycle. The PCRA process begins
with a set of project scoping and planning activities in which members of the
project team (a) consolidate information about the project-, use-, domain-, anddata-contexts of the prospective system, (b) identify relevant stakeholders, (c)begin to scope the potential impacts of their systems on human rights, democracy,
and the rule of law by identifying salient rights and freedoms, and (d) map thegovernance workflow.
After these activities are recorded in a Project Summary Report, the project teamis ready to complete its PCRA. The PCRA is a set of questions and prompts that
help project teams to pinpoint risk factors and potential adverse impacts of theirsystems on human rights, fundamental freedoms, and elements of democracy andthe rule of law. When the PCRA is finished, it automatically generates a report(through triggers built on simple conditional rules) that directs project teams tospecific actions that they then need to take in their impact assessment process(HUDERIA) so that sufficient consideration can be given to each of the risk factors
detected and to salient rights and freedoms that could be impacted by theprospective AI system. It also directs them to specific goals, properties, and areas

12

that they should focus on in their subsequent risk management and assurance
processes (HUDERAC) to reduce and mitigate associated risks. The report
additionally assists project teams to understand, in a provisional way, the risk
level of their system, so that they can take a proportionate approach to risk
management actions and stakeholder engagement.

Preparing a Project Summary Report in the Planning and Scoping
Phase
Deliberate and reflective reporting practices are a crucial component to assuring
an AI system’s alignment with human rights, democracy, and the rule of law.
Project Summary (PS) reporting supports this goal by generating an ongoingsynopsis of an AI project’s contextual reflexivity, stakeholder awareness,governance resolve, and human rights diligence across its entire lifecycle. The first
iteration of writing a PS Report serves, during the planning and scoping phase ofa project, as preparation for answering the questions contained within the PCRA.
It provides a reference point for your responses to the context-based questions
about risk factors and the potential adverse impacts of AI on the human rights
and fundamental freedoms of affected individuals and communities.
The first time that the PS Report is completed (at the very beginning of your
HUDERAF), it should be informed by available organisational scoping and planningdocuments and by desk-based research pertaining to the system’s use context,
stakeholder contexts, and human rights, democracy, and the rule of law. It is later
revised and updated at each iteration of the Stakeholder Engagement Process(drawing from stakeholder input). Changes to this document are recorded andregistered over time, providing timestamped project summaries that are intended
to help build public trust by operationalizing goals of accountability andtransparency. The PS Report enables a dimension of traceability, illustrating how
a project has been configured through time by the use of a Human Rights,
Democracy, and Rule of Law Assurance Framework.
A Project Summary Report is comprised of four components: (1) a project
overview, (2) a list of identified stakeholders, (3) a scoping of potential impacts
on human rights, democracy, and rule of law, and (4) a project governanceworkflow. These components are developed through the following activities:131. Outlining
project-,use-,domain-,and datacontexts•Outline a high-level description pertaining to theproject to build the prospective system, the domain
in which it will operate, the contexts in which it will
be used, and the data on which it will be trained.

2.
Identifying
stakeholders•Build on this contextual understanding to identify
project stakeholders and get a sense of the social
environment and human factors that may be
impacted by the system. Pay close attention tovulnerable rights-holders.
3. Scoping
Potential
Impacts toFoundamental
Rights andFreedoms•Consider the system's context, stakeholders,
human rights, democracy, and the rule oflaw to get a sense of the human rights andfundamental freedoms that may be
impacted by the system.

4. Mapping
Governance
Workflow•Consider the scope of potential impacts tofundamental rights and freedoms to
distribute responsibility amidst project dutybearers and establish a logging protocol for
workflow activities.

Activity 1: Outline Project-, Use-, Domain-, and Data-Contexts
During the initial iteration of writing the PS Report, draw on organisational
documents (i.e., the project business case, proof of concept, or project charter),project team collaboration, and desk research (if necessary) to answer thefollowing questions 2:
Questions:
Project•

How would you describe your organisation and what sort of services or
products do you typically provide (beyond the system under consideration)?

These questions draw on research in both the study of sociotechnical dimensions of AI
governance and in human rights impact assessment. For exemplary resources in the firstgroup, see: Arnold, M., Bellamy, R. K., Hind, M., Houde, S., Mehta, S., Mojsilović, A., ...
& Varshney, K. R. (2019). FactSheets: Increasing trust in AI services through supplier's
declarations of conformity. IBM Journal of Research and Development, 63(4/5), 6-1;
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., ... & Gebru,
T. (2019, January). Model cards for model reporting. In Proceedings of the conference on
fairness, accountability, and transparency (pp. 220-229); Gebru, T., Morgenstern, J.,
Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2018).
Datasheets for datasets. arXiv preprint arXiv:1803.09010. For exemplary resources in thesecond group, see Danish Institute for Human Rights (2020). Human Rights Impact
Assessment: Guidance and Toolbox. DIHR; OECD (2018), OECD Due Diligence Guidancefor Responsible Business Conduct; Mantelero, A., and Esposito, S. (2021). An evidencebased methodology for human rights impact assessment (HRIA) in the development of AI
data-intensive systems. Computer Law & Security Review.
2

14

•
•
•
•
•
•

What AI system is being built and what type of product or service will it
offer?
What benefits will the system bring to its users and customers, and willthese benefits be widely accessible?
Which organisation(s)—yours, other suppliers, or other providers—are
responsible for building this AI system?
Which parts or elements of the AI system, if any, will be procured fromthird-party vendors, suppliers, sub-contractors, or external developers?
Which algorithms, techniques, and model types will be used in the AI
system? (Provide links to technical papers where appropriate)
In a scenario where your project optimally scales, how many people will it
impact, for how long, and in what geographic range (local, national, global)?
(Describe your rationale)

Use Context
•
•

•
•
•

What is the purpose of this AI system and in which contexts will it be used?
(Briefly describe a use-case that illustrates primary intended use)
Is the AI system’s processing output to be used in a fully automated way
or will there be some degree of human control, oversight, or input before
use? (Describe)
Will the AI system evolve or learn continuously in its use context, or will it
be static?
To what degree will the use of the AI system be time-critical, or will users
be able to evaluate outputs comfortably over time?
What sort of out-of-scope uses could users attempt to apply the AI system,
and what dangers may arise from this?

Domain••In what domain will this AI system operate?
Which, if any, domain experts have been or will be consulted in designingand developing the AI system?

Data
•
•
•
•

What datasets are being used to build this AI system?
Will any data being used in the production of the AI system be acquired
from a vendor or supplier? (Describe)
Will the data being used in the production of the AI system be collected forthat purpose, or will it be re-purposed from existing datasets? (Describe)
What quality assurance and bias mitigation processes do you have in place
for the data lifecycle—for both acquired and collected data?

Activity 2: Identify Stakeholders
Stakeholders include 3:

3 Danish Institute for Human Rights (2020). Human Rights Impact Assessment: Guidance

15

•

•

•

Rights-Holders: All individuals are human rights-holders. In the impact
assessment, the primary focus is on rights-holders who are, or may be,
adversely affected by the project.
Duty-Bearers: Duty-bearers are actors who have human rights duties or
responsibilities towards rights-holders. These include the company ororganization operating a project or conducting its activities, businesssuppliers and contractors, joint-venture and other business partners, state
actors such as local government authorities and regional and national
government departments and agencies.
Other Relevant Parties: These may include individuals or organisations
representing the interests of rights holders and official representations atinternational, national and local levels (e.g. the UN, national human rights
institutions, NGOs or civil society organisations).

During stakeholder identification it is particularly important to consider whichrights-holders are most vulnerable to potential impacts of the tool/system being
developed and which stakeholders’ views are underrepresented in development
and deployment processes. During the initial iteration of writing the PS Report,draw on desk research pertaining to stakeholders and their contexts to identify
stakeholders and to get a sense of the social environment and human factors that
may be impacted by the system. Answer the following questions:
Questions:•

Who are the rights-holders, duty bearers and other relevant parties that
may be impacted by, or may impact, the project?

To further identify groups of rights-holders, answer the following questions:Protected Characteristics:•

Do any of these rights-holders possess sensitive or protected characteristics
that could increase their vulnerability to abuse or discrimination, or forreason of which they may require additional protection or assistance withrespect to the impacts of the project? If so, what characteristics?

Contextual Vulnerability Characteristics:••

Could the outcomes of this project present significant concerns to specificgroups of rights-holders given vulnerabilities caused or precipitated by their
distinct circumstances?
If so, what vulnerability characteristics expose them to being jeopardized
by project outcomes?

For any characteristics identified in the previous answers, consider thequestions:••

What affected group or groups of rights-holders could these protected orcontextual vulnerability characteristics or qualities represent?
How could the distribution of power relations (i.e., the relative advantages
and disadvantages) between rights holders and duty bearers affect the way
the benefits and risks of the project are allocated?

16

Activity 3: Scope Impacts to Human Rights, Democracy, and Ruleof Law
The information contained below in the Table 1 (Principles and Priorities,
Corresponding Rights and Freedoms, and Elaborations) serves as backgroundmaterial to provide you with a means of accessing and understanding some of theexisting human rights and fundamental freedoms that could be impacted by theuse of AI technologies. A thorough review of this table and an engagement of thelinks to the relevant Charters, Conventions, Declarations, and elaborations it
contains is a critical first step that will help you identify the salient rights andfreedoms that could be affected by your project. You should also explore whether
your organisation has engaged in any previous Human Rights Impact Assessments
or other impact assessments (data protection impact assessment, equality impact
assessment, ethical and social impact assessment, environmental impact
assessment, etc.)—and review these where they are present.
As a starting point, the rights and freedoms delineated within this material can bepresented in summary form in the following principles and priorities, which areenumerated and elaborated upon in the CAHAI’s Feasibility Study and in the multistakeholder consultation that followed the plenary’s adoption of this text inDecember 2020 4:

As has been indicated within and at the bottom of Table 1, this list of rights and freedoms
also reflects important adjacent work done by the Council of Europe, UNESCO, theEuropean Commission, OECD, and the European Agency for Fundamental Rights among
others. For relevant sources, see: Council of Europe Commissioner for Human Rights(2019) – “Unboxing AI: 10 steps to protect Human Rights”; Council of Europe,“Recommendation CM/Rec (2020) of the Committee of Ministers to member States on thehuman rights impacts of algorithmic systems”; Muller, C. The Impact of Artificial
Intelligence on Human Rights, Democracy and the Rule of Law. Council of Europe,
CAHAI(2020)06-fin; Bergamini, D. (Rapporteur) (2020). Need for democratic governanceof artificial intelligence. Council of Europe, Committee on Political Affairs and Democracy,
Parliamentary Assembly; UNESCO (2021a/b) – Recommendations on the Ethics of Artificial
Intelligence (two drafts, 1-25, 26-134); High-level Expert Group on Artificial Intelligence,
European Commission (2020) – “The Assessment List for Trustworthy Artificial Intelligence(ALTAI) for self-assessment”; European Agency for Fundamental Rights (2020) – “Getting
the Future Right: Artificial Intelligence and Fundamental Rights”.
417Human Dignity
All individuals are inherently and inviolably worthy of respect by mere virtueof their status as human beings. Humans should be treated as moral subjects,
and not as objects to be algorithmically scored or manipulated.
Human Freedom and Autonomy
Humans should be empowered to determine in an informed and autonomous
manner if, when, and how AI systems are to be used. These systems should
not be employed to condition or control humans, but should rather enrich theircapabilities.
Prevention of Harm
The physical and mental integrity of humans and the sustainability of thebiosphere must be protected, and additional safeguards must be put in place
to protect the vulnerable. AI systems must not be permitted to adversely
impact human wellbeing or planetary health.
Fairness, Non-Discrimination, Equality, Diversity, and InclusivenessAll humans possess the right to non-discrimination and the right to equality
and equal treatment under the law. AI systems must be designed to be fair,equitable, and inclusive in their beneficial impacts and in the distribution of
their risks.
Data Protection and the Right to Privacy
The design and use of AI systems that rely on the processing of personal data
must secure a person’s right to respect for private and family life, including
the individual's right to control their own data. Informed, freely given, andunambiguous consent must play a role in this.
Democracy
Individuals should enjoy the ability to freely form bonds of social cohesion,
human connection, and solidarity through inclusive and regular democratic
participation, whether in political life, work life, or social life. This requires
informational plurality, the free and equitable flow of the legitimate and valid
forms of information, and the protection of freedoms of expression, assembly,
and association.
Rule of Law
AI systems must not undermine judicial independence, effective remedy, theright to a fair trial, due process, or impartiality. To ensure this, thetransparency, integrity, and fairness of the data, and data processing methods
must be secured.
Social and Economic Rights
Individuals must have access to the material means needed to participate
fully in work life, social life, and creative life, and in the conduct of publicaffairs, through the provision of proper education, adequate living andworking standards, health, safety, and social security. This means that AI
systems should not infringe upon individuals’ rights to work, to just, safe, andhealthy working conditions, to social security, to the protection of health, andto social and medical assistance.

18

Table 1. Principles and Priorities, Corresponding Rights and Freedoms, and Elaborations(Numbers in the first column indicate overlapping principles and priorities from other relevant sources listed at the bottom)

Principles andPriorities

Corresponding Rights and Freedoms with
Selected Elaborations

Respect for andprotection ofhuman dignity (2,3, 4, 5, 8)-The right to human dignity, the right to life (Art. 2
ECHR), and the right to physical and mental
integrity.
-The right to be informed of the fact that one isinteracting with an AI system rather than with ahuman being.
-The right to refuse interaction with an AI system
whenever this could adversely impact humandignity.

∼“Human dignity is the foundation of all human rights. It
recognises that all individuals are inherently worthy ofrespect by mere virtue of their status as human beings.
Human dignity, as an absolute right, is inviolable. Hence,even when a human right is restricted – for instance whena balance of rights and interests must be made – humandignity must always be safeguarded. In the context of AI,
this means that the design, development and use of AI
systems must respect the dignity of the human beingsinteracting therewith or impacted thereby. Humans
should be treated as moral subjects, and not as mere
objects that are categorised, scored, predicted ormanipulated.
AI applications can be used to foster human dignity andempower individuals, yet their use can also challenge itand (un)intentionally run counter to it. To safeguard

19

Resources for Principles andCorresponding Rights and Freedoms
European Convention on Human Rights (ECHR):-Article 2, European Convention on Human Rights – Rightto life-Article 2, ‘Guide on Article 2 of the European Convention
on Human Rights’, Council of Europe – Right to life
Charter of the Fundamental Rights of the EU:-Article 1, Charter of the Fundamental Rights of the EU –Human dignity-Article 1, Explanation on Article 1 — Human Dignity(Explanations Relating to the Charter of Fundamental
Rights) in the Official Journal of the European Union C
303/17 - 14.12.2007-Article 2, Charter of the Fundamental Rights of the EU –Right to life
-Article 2, Explanation on Article 2 — Right to life
(Explanations Relating to the Charter of Fundamental
Rights) in the Official Journal of the European Union C
303/17 - 14.12.2007
Universal Declaration of Human Rights:-Preamble, Universal Declaration of Human Rights –Dignityhuman dignity, it is essential that human beings areaware of the fact that they are interacting with an AI
system and are not misled in this regard. Moreover, theyshould in principle be able to choose not to interact withit, and to not be subject to a decision informed or madeby an AI system whenever this can significantly impact
their lives, especially when this can violate rights related
to their human dignity. Furthermore, the allocation ofcertain tasks may need to be reserved for humans rather
than machines given their potential impact on humandignity.” 5∼International Covenant on Civil and Political Rights:-Article 6, International Covenant on Civil and Political
Rights – Right to life
Council of Europe Resources:-Council of Europe, Convention for the Protection of
Human Rights and Dignity of the Human Being withregard to the Application of Biology and Medicine:
Convention on Human Rights and Biomedicine“As stated in the EU Charter’s official guiding explanations(Official Journal of the European Union, 2007: C 303/17):‘dignity of the human person is not only a fundamental
right in itself but constitutes the real basis of fundamental
rights.’ In this sense, dignity is a highly multifaceted
concept… One way to grasp the broad range of its
meanings is to explore what humans consider to beviolations of their dignity, as done by Halbertal (2015),
who points out the following three categories ofviolations: 1. Humiliation: being put in a state ofhelplessness, insignificance; losing autonomy over yourown representation. 2. Instrumentalization: treating an
individual as exchangeable and merely a means to an
end. 3. Rejection of one’s gift: making an individual
superfluous, unacknowledging one’s contribution,
aspiration, and potential. 6

5

CAHAI Feasibility Study, Council of Europe CAHAI (2020)23.

6 Aizenberg, E., & van den Hoven, J. (2020). Designing for human rights in AI. Big Data & Society, 7(2); Halbertal, M. (2015). Three

concepts of human dignity. 2015 Dewey Lecture, University of Chicago; Kaufmann, P., Kuch, H., Neuhaeuser, C., & Webster, E. (Eds.).
(2010). Humiliation, degradation, dehumanization: human dignity violated (Vol. 24). Springer Science & Business Media.
20

“The inviolable and inherent dignity of every humanconstitutes the foundation for the universal, indivisible,
inalienable, interdependent and interrelated system of
human rights and fundamental freedoms. Therefore,
respect, protection and promotion of human dignity andrights as established by international law, includinginternational human rights law, is essential throughout
the life cycle of AI systems. Human dignity relates to therecognition of the intrinsic and equal worth of eachindividual human being, regardless of race, colour,
descent, gender, age, language, religion, political opinion,
national origin, ethnic origin, social origin, economic orsocial condition of birth, or disability, and any othergrounds....Persons may interact with AI systems
throughout their life cycle and receive assistance fromthem such as care for vulnerable people or people invulnerable situations, including but not limited to children,
older persons, persons with disabilities or the ill. Withinsuch interactions, persons should never be objectified,
nor should their dignity be otherwise undermined, or
human rights and fundamental freedoms violated orabused.” 7

Protection ofhuman freedom
and autonomy (1,2, 4, 6, 7, 8)

7-The right to liberty and security (Art. 5 ECHR).

European Convention on Human Rights (ECHR):-The right to human autonomy and selfdetermination.

-Article 5, European Convention on Human Rights – Rightto liberty and security-The right not to be subject to a decision based
solely on automated processing when this produces
legal effects on groups or similarly significantly
affects individuals.

-Article 5, ‘Guide on Article 5 of the European Convention
on Human Rights’, Council of Europe – Right to liberty
and security

UNESCO (2021a/b) – Recommendations on the Ethics of Artificial Intelligence (two drafts, 1-25, 26-134).
21

-The right to effectively contest and challenge
decisions informed and/or made by an AI systemand to demand that such decisions be reviewed by aperson.
-The right to freely decide to be excluded from AIenabled manipulation, individualized profiling, andpredictions. This also applies to cases of nonpersonal data processing.
-The right to have the opportunity, when it is notoverridden by competing legitimate grounds, tochoose to have contact with a human being rather
than a robot.

∼-Article 9, European Convention on Human Rights– Freedom of thought, conscience, and religion-Article 9, ‘Guide on Article 9 of the European Convention
on Human Rights’, Council of Europe – Freedom ofthought, conscience, and religion-Article 10, European Convention on Human Rights– Freedom of expression-Article 10, ‘Guide on Article 10 of the European
Convention on Human Rights’, Council of Europe –Freedom of expression
Charter of the Fundamental Rights of the EU:“Individual self-determination and personal autonomy
necessarily entail the ability to freely take decisions andhave them respected by others. According to international
and European human rights jurisprudence, individual
personal autonomy also covers a further range of humanbehaviours, amongst which are the right to develop one’s
own personality and the right to establish and develop
relationships with other people, the right to pursue one’s
own aspirations and to control one’s own information.” 8“Promoting autonomy means enabling individuals to makedecisions about their lives on their own, which are notimposed upon them or manipulated by others. It requires
having a range of genuine options available from which
one can choose. A decision taken on the basis of-Article 6, Charter of the Fundamental Rights of the EU –Right to liberty and security-Article 6, Explanation on Article 6 — Right to liberty andsecurity (Explanations Relating to the Charter ofFundamental Rights) in the Official Journal of theEuropean Union C 303/17 - 14.12.2007-Article 10, Charter of the Fundamental Rights of the EU –Freedom of thought, conscience, and religion-Article 10, Explanation on Article 10 — Freedom ofthought, conscience, and religion (Explanations Relating
to the Charter of Fundamental Rights) in the Official
Journal of the European Union C 303/17 - 14.12.2007

Mantelero, A., and Esposito, S. (2021). An evidence-based methodology for human rights impact assessment (HRIA) in the developmentof AI data-intensive systems. Computer Law & Security Review.

8

22

inadequate information or deception is not consideredautonomous.” 9-Article 11, Charter of the Fundamental Rights of the EU –Freedom expression and information-Article 11, Explanation on Article 11 — Freedom of
expression and information (Explanations Relating to theCharter of Fundamental Rights) in the Official Journal of
the European Union C 303/17 - 14.12.2007
Universal Declaration of Human Rights:-Article 3, Universal Declaration of Human Rights – Rightto life, liberty, and the security of person-Article 18, Universal Declaration of Human Rights – Rightto freedom of thought, conscience, and religion-Article 19, Universal Declaration of Human Rights – Rightto freedom of opinion and expression
International Covenant on Civil and Political Rights:-Article 9, International Covenant on Civil and Political
Rights – Right to liberty and security of person-Article 18, International Covenant on Civil and Political
Rights – Right to freedom of thought, conscience, andreligion-Article 19, International Covenant on Civil and Political
Rights – Freedom of expression
Council of Europe Resources:

Loi, et al. (2021) Algorithm Watch – “Automated Decision-Making Systems in the Public Sector: An Impact Assessment Tool for Public
Authorities”

923

-Commissioner for Human Rights, Human Rights andDisability (20 October, 2008) ‘Equal rights for all’, para.
5.2. – Council of Europe-Bychawska-Siniarska, ‘Protecting the right to freedom of
expression under the European Convention on Human
Rights’ – Council of Europe, Strasbourg 2017

Prevention of harmand protection of
the right to life andphysical,
psychological, andmoral integrity (2,5, 6, 8)-The right to life (Art. 2 ECHR) and the right tophysical and mental integrity.
-The right to the protection of the environment.
-The right to sustainability of the community andbiosphere.

∼

European Convention on Human Rights (ECHR):-Article 2, European Convention on Human Rights – Rightto life-Article 2, ‘Guide on Article 2 of the European Convention
on Human Rights’, Council of Europe – Right to life
Charter of the Fundamental Rights of the EU:“Throughout the life cycle of AI systems, human beings
should not be harmed but rather their quality of life of
human beings should be maintained or enhanced, while
the definition of “quality of life” should be left open to
individuals or groups, while respecting relevant laws, aslong as there is no violation or abuse of human rights andfundamental freedoms, or the dignity of humans in termsof this definition....Environmental and ecosystemflourishing should be recognized, protected and promoted
through the life cycle of AI systems. Furthermore,environment and ecosystems are the existential necessity
for humanity and other living beings to be able to enjoy
the benefits of advances in AI....All actors involved in thelife cycle of AI systems must comply with applicableinternational law and domestic legislation, standards andpractices, such as precaution, designed for environmental
and ecosystem protection and restoration, andsustainable development. They should reduce the24-Article 2, Charter of the Fundamental Rights of the EU –Right to life
-Article 2, Explanation on Article 2 — Right to life
(Explanations Relating to the Charter of Fundamental
Rights) in the Official Journal of the European Union C
303/17 - 14.12.2007-Article 3, Charter of the Fundamental Rights of the EU –Right to integrity of the person-Article 3, Explanation on Article 3 — Right to theintegrity of the person (Explanations Relating to theCharter of Fundamental Rights) in the Official Journal of
the European Union C 303/17 - 14.12.2007

environmental impact of AI systems, including but notlimited to, its carbon footprint, to ensure the minimizationof climate change and environmental risk factors, andprevent the unsustainable exploitation, use andtransformation of natural resources contributing to thedeterioration of the environment and the degradation of
ecosystems.” 10

Non-discrimination,
fairness, andequality (1, 2, 3, 4,
5, 6, 7, 8)-The right to non-discrimination (on the basis of theprotected grounds set out in Article 14 of the ECHR
and Protocol 12 to the ECHR), includingintersectional discrimination.

-The right to non-discrimination and the right toequal treatment. This right must be ensured inrelation to the entire lifecycle of an AI system(design, development, implementation, and use), as
well as to the human choices concerning AI design,
adoption, and use, whether used in the public or
private sector.

∼-Protocol No. 12, European Convention on Human Rights-Article 14, European Convention on Human Rights –Prohibition of discrimination-Article 14 and Article 12 of Protocol No. 12, ‘Guide on
Article 14 of the European Convention on Human Right
and on Article 1 of Protocol No. 12 to the Convention’,Council of Europe – Prohibition of discrimination
Charter of the Fundamental Rights of the EU:“In all circumstances, discrimination risks must be
prevented and mitigated with special attention for groups
that have an increased risk of their rights being
disproportionately impacted by AI. This includes women,children, older people, economically disadvantaged
persons, members of the LGBTI community, persons withdisabilities, and “racial”, ethnic or religious groups.
Member states must refrain from using AI systems that
discriminate or lead to discriminatory outcomes and,

10

European Convention on Human Rights (ECHR):-Articles 20 and 21, Charter of the Fundamental Rights ofthe EU – Equality before the law and non-discrimination-Article 20, Explanation on Article 20 — Equality before
the law (Explanations Relating to the Charter ofFundamental Rights) in the Official Journal of theEuropean Union C 303/17 - 14.12.2007-Article 21, Explanation on Article 21 — Nondiscrimination (Explanations Relating to the Charter of

UNESCO (2021a/b) – Recommendations on the Ethics of Artificial Intelligence (two drafts, 1-25, 26-134).

25

within their jurisdiction, protect individuals from theconsequences of use of such AI systems by third parties.
The active participation of and meaningful consultation
with a diverse community that includes effective
representation from these groups in all stages of the AI
lifecycle is an important component of prevention andmitigation of adverse human rights impacts. In addition,
special attention needs to be paid to transparency andaccessibility of information on the training data used forthe development of an AI system.” 11

Universal Declaration of Human Rights:-Article 7, Universal Declaration of Human Rights –Equality before the law
International Covenant on Civil and Political Rights:-Article 6, International Covenant on Civil and Political
Rights – Right to life∼
“AI actors should promote social justice, by safeguarding
fairness and non-discrimination of any kind in compliancewith international law. Fairness implies sharing benefits of
AI technologies at local, national and international levels,
while taking into consideration the specific needs ofdifferent age groups, cultural systems, different language
groups, persons with disabilities, girls and women, anddisadvantaged, marginalized and vulnerable populations.
At the local level, it is a matter of working to givecommunities access to AI systems in the languages of
their choice and respecting different cultures. At thenational level, governments are obliged to demonstrate
equity between rural and urban areas, and among all
persons without distinction as to sex, gender, language,
religion, political or other opinion, national, ethnic,
indigenous or social origin, sexual orientation and gender
identity, property, birth, disability, age or other status, interms of access to and participation in the AI system life
cycle. At the international level, the most technologicallyadvanced countries have a responsibility of solidarity with
11

Fundamental Rights) in the Official Journal of theEuropean Union C 303/17 - 14.12.2007-Article 26, International Covenant on Civil and Political
Rights – Non-discrimination
Council of Europe Resources:-European Commission against Racism and Intolerance(ECRI) - Discrimination, artificial intelligence, andalgorithmic decision-making (2018)• AI applications have found ways to “escapecurrent laws.” The majority of non-discrimination
statutes relate only to specific protectedcharacteristics. There are other forms of
discrimination that are not correlated withprotected characteristics but can still reinforce
social inequality.
• The idea of sector-specific rules for the protectionof fairness and human rights in the area of AI is
proposed, as different sectors necessitate different
values and problems.

Council of Europe Commissioner for Human Rights (2019) – “Unboxing AI: 10 steps to protect Human Rights”

26

the least advanced to ensure that the benefits of AI
technologies are shared such that access to andparticipation in the AI system life cycle for the latter
contributes to a fairer world order with regard to
information, communication, culture, education, research,and socio-economic and political stability...AI actors
should promote social justice and safeguard fairness andnon-discrimination of any kind in compliance withinternational law. This implies an inclusive approach toensuring the benefits of AI technologies are available andaccessible to all taking into consideration the specificneeds of different age groups, cultural systems, different
language groups, persons with disabilities, girls andwomen, and disadvantaged, marginalized and vulnerable
people or people in vulnerable situations.” 12∼“AI is widely expected to improve the productivity of
economies. However, these productivity gains are
expected to be distributed unequally with most benefits
accruing to the well-off. Similarly, data and design
choices, combined with a lack of transparency of blackbox algorithms have shown to lead to a perpetuating
unjust bias against already disadvantaged groups insociety, such as women and ethnic minorities.28 AI could
lead to inequality and segregation and thus threaten thenecessary level of economic and social equality required
for a thriving democracy.” 13

Data protection
and the right torespect of private
12
13-The right to respect for private and family life andthe protection of personal data (Art. 8 ECHR).

-Unboxing artificial intelligence: 10 steps to protect

human rights (2019)• Recommendations are to be used to mitigate or
prevent negative impacts of AI systems on humanrights.
• Practical recommendations are given with 10 areasof action: human rights impact assessments;
public consultations; human rights standards inthe private sector; information and transparency;
independent monitoring; non-discrimination andequality; data protection and privacy; freedom of
expression, freedom of assembly and association,and the right to work; avenues for redress; andpromoting knowledge and understanding of AI.
Office of the United Nations High Commissioner for
Human Rights:-OHCHR, International Convention on the Elimination of
All Forms of Racial Discrimination-OHCHR, Convention on the Elimination of All Forms ofDiscrimination against Women

European Convention on Human Rights (ECHR):-Article 8, European Convention on Human Rights – Right
to respect for private and family life

UNESCO (2021a/b) – Recommendations on the Ethics of Artificial Intelligence (two drafts, 1-25, 26-134).
Muller, C. The Impact of Artificial Intelligence on Human Rights, Democracy and the Rule of Law. Council of Europe, CAHAI(2020)06-fin.
27

and family life (1,2, 3, 5, 7, 8)-The right to physical, psychological, and moral
integrity in light of AI-based profiling andemotion/personality recognition.
-All the rights enshrined in Convention 108+ and inits modernised version, and in particular withregard to AI-based profiling and location tracking.

∼“The development, training, testing and use of AI systems
that rely on the processing of personal data must fully
secure a person’s right to respect for private and family
life under Article 8 of the European Convention on Human
Rights, including the “right to a form of informational selfdetermination” in relation to their data.
Data processing in the context of AI systems shall be
proportionate in relation to the legitimate purpose
pursued through such processing, and should at all stagesof the processing reflect a fair balance between theinterests pursued through the development anddeployment of the AI system and the rights and freedoms
at stake.
Member states should effectively implement themodernised Council of Europe Convention for theProtection of Individuals with regard to Automatic
Processing of Personal Data (“Convention 108+”) as wellas any other international instrument on data protection
and privacy that is binding on the member state. The
processing of personal data at any stage of an AI system
lifecycle must be based on the principles set out under
the Convention 108+, in particular (i) there must be alegitimate basis laid down by law for the processing of thepersonal data at the relevant stages of the AI system
lifecycle; (ii) the personal data must be processedlawfully, fairly and in a transparent manner; (iii) the28-Article 8, ‘Guide on Article 8 of the European Convention
on Human Rights. Right to respect for private and family
life, home and correspondence’, Council of Europe – Right
to respect for private and family life
Charter of the Fundamental Rights of the EU:-Article 7, Charter of the Fundamental Rights of the EU –Right for private and family life
-Article 7, Explanation on Article 7 — Right for privateand family life (Explanations Relating to the Charter ofFundamental Rights) in the Official Journal of theEuropean Union C 303/17 - 14.12.2007-Article 8, Charter of the Fundamental Rights of the EU –Protection of personal data
-Article 8, Explanation on Article 8 — Protection of
personal data (Explanations Relating to the Charter ofFundamental Rights) in the Official Journal of theEuropean Union C 303/17 - 14.12.2007
Universal Declaration of Human Rights:-Article 12, Universal Declaration of Human Rights – Right
to respect for privacy, family, home, or correspondence
Council of Europe Resources:-Convention 108/108+ (1981/2018)• Processing of sensitive data can only be allowed
where appropriate guidelines are present• Every individual has the right to know the purposeof processing their data. Along with this, they havea right to rectification and obtainment ofpersonal data must be collected for explicit, specified andlegitimate purposes and not processed in a way
incompatible with those purposes; (iv) the personal data
must be adequate, relevant and not excessive in relation
to the purposes for which they are processed; (v) the
personal data must be accurate and, where necessary,
kept up to date; (vi) the personal data should bepreserved in a form which permits identification of data
subjects for no longer than is necessary for the purposes
for which those data are processed.” 14∼“The right to respect for private life and the protection of
personal data (Articles 7 and 8 of the EU Charter) are at
the core of fundamental rights discussions around the useof AI. While closely related, the rights to respect for
private life and the protection of personal data are
distinct, self-standing rights. They have been described asthe ‘classic’ right to the protection of privacy and a more“modern” right, the right to data protection. Both strive toprotect similar values, i.e. the autonomy and humandignity of individuals, by granting them a personal spherein which they can freely develop their personalities, thinkand shape their opinions. They thus form an essential
prerequisite for the exercise of other fundamental rights,
such as the freedom of thought, conscience and religion(Article 10 of the EU Charter), freedom of expression andinformation (Article 11 of the EU Charter), and freedom ofassembly and of association (Article 12 of the EU
Charter). The concept of ‘private life’ or ‘privacy’ iscomplex and broad, and not susceptible to an exhaustive
definition. It covers the physical and psychologicalintegrity of a person, and can, therefore, embrace
14

•

•
•

•

knowledge where data are processed contrary to
the Convention’s provisions
Transparency, proportionality, consent, data
quality, data security, purpose limitation,accountability, data minimisation, data protection
and privacy by design, fairness, and lawfulness are
principles introduced
Individuals should not be subjected to decisions
made solely by automated processing of data
without consideration of personal views“Legal framework built around Convention remains
fully applicable to AI technology, as soon as theprocessed data fall within the scope of theConvention.”
Respect for the rights of data subjects andconformity to the additional obligations of data
controllers and processors as set out in Articles 9
and 10 of Convention 108+

-Convention on Cybercrime (“Budapest
Convention”)(2001)• “Criminalising offences against and by the meansof computers, for procedural powers to investigate
cybercrime and secure electronic evidence.”• Crimes include but are not limited to infringementsof copyright, computer-related fraud, childpornography, and violations of a security network• Investigation includes a series of powers andprocedures including interception and the searchof computer networks• Primary objective is to “pursue a common criminal
policy aimed at the protection of society against
cybercrime, especially through appropriate
legislation and international co-operation.”

Council of Europe Commissioner for Human Rights (2019) – “Unboxing AI: 10 steps to protect Human Rights”
29multiple aspects of the person’s physical and social
identity. There is also a zone of interaction of a personwith others, even in a public context, which may fallwithin the scope of ‘privacy”.” 15∼“Many AI-systems and uses have a broad and deep
impact on the right to privacy. Privacy discussions aroundAI currently tend to focus primarily on data privacy andthe indiscriminate processing of personal (and nonpersonal) data. It should however be noted that, whiledata privacy is indeed an important element, the impact
of AI on our privacy goes well beyond our data. Art. 8 of
the ECHR encompasses the protection of a wide range ofelements of our private lives, that can be grouped into
three broad categories namely: (i) a person’s (general)
privacy, (ii) a person's physical, psychological or moral
integrity and (iii) a person's identity and autonomy.
Different applications and uses of AI can have an impact
on these categories, and have received little attention to
date.

-Directorate General of Human Rights and Rule of Law,
Guidelines on Artificial Intelligence and Data Protection,Council of Europe-Roagna, ‘Protecting the right to respect for private andfamily life under the European Convention on Human
Rights’ – Council of Europe, Strasbourg 2012

AI-driven (mass) surveillance, for example with facialrecognition, involves the capture, storage and processingof personal (biometric) data (our faces), but it also affects
our 'general' privacy, identity and autonomy in such a
way that it creates a situation where we are (constantly)
being watched, followed and identified. As a psychological‘chilling’ effect, people might feel inclined to adapt their
behaviour to a certain norm, which shifts the balance ofpower between the state or private organisation using
facial recognition and the individual. In legal doctrine andprecedent the chilling effect of surveillance can constitutea violation of the private space, which is necessary for
15

European Agency for Fundamental Rights (2020) – “Getting the Future Right: Artificial Intelligence and Fundamental Rights”.
30

personal development and democratic deliberation. Evenif our faces are immediately deleted after capturing, thetechnology still intrudes our psychological integrity.” 16

Economic andsocial rights (1, 7,
8)-The right to just working conditions, the right tosafe and healthy working conditions, the right toorganize, the right to social security, and the rights
to the protection of health and to social and medical
assistance (Art. 2, 3, 5, 11, 13 ESC).

∼“AI systems can have major benefits when used forhazardous, heavy, exhausting, unpleasant, repetitive or
boring work. However, the wide adoption of AI systems inall domains of our lives also creates new risks to socialand economic rights. AI systems are increasingly used tomonitor and track workers, distribute work without
human intervention and assess and predict worker
potential and performance in hiring and firing situations.
In some situations, this can also have detrimental
consequences for workers’ right to decent pay, as their
pay can be determined by algorithms in a way that isirregular, inconsistent and insufficient. Furthermore, AI
systems can also be used to detect and counter theunionisation of workers. These applications can jeopardise
the right to just, safe and healthy working conditions,
dignity at work as well as the right to organise. The
discrimination capacity of AI systems that assess andpredict the performance of job applications or workers
can also undermine equality, including gender equality, inmatters of employment and occupation.
In addition, AI systems can, for instance, be used in thecontext of social security decisions, in which case the16

European Social Charter:-Article 2, European Social Charter – Dignity of work andthe right to just working conditions-Article 3, European Social Charter – Right to safe andhealthy working conditions-Article 5, European Social Charter – Right to organize-Article 11, European Social Charter – Rights to theprotection of health-Article 12, European Social Charter – Right to social
security-Article 13, European Social Charter – Right to social andmedical assistanceInternational Covenant on Economic, Social andCultural Rights:-Article 6, International Covenant on Economic, Social,
and Cultural Rights – The right to work
-Article 7, International Covenant on Economic, Social,
and Cultural Rights – Right to just and favourable
conditions of work
-Article 8, International Covenant on Economic, Social,
and Cultural Rights – Right to organise

Muller, C. The Impact of Artificial Intelligence on Human Rights, Democracy and the Rule of Law. Council of Europe, CAHAI(2020)06-fin.
31

right guaranteed by Article 12 of the European Social
Charter – stating that all workers and their dependents
have the right to social security – can be impacted.
Indeed, AI systems are increasingly relied on in social
welfare administration, and the decisions taken in that
context can significantly impact individuals’ lives. Similar
issues arise where AI systems are deployed in the context
of education or housing allocation administrations.
Moreover, whenever AI systems are used to automate
decisions regarding the provision of healthcare andmedical assistances, such use can also impact the rights
enshrined in Articles 11 and 13 of the Charter, which
respectively state that everyone has the right to benefit
from measures that enable the enjoyment of the highest
possible standard of health attainable, and that anyone
without adequate resources has the right to social andmedical assistance. AI systems can, for instance, beutilised to determine patients’ access to health careservices by analysing patients’ personal data, such as
their health care records, lifestyle data and otherinformation. It is important that this occurs in line withnot only the right to privacy and personal data protection,
but also with all the social rights laid down in the
aforementioned Charter, the impact on which has so far
received less attention than the impact on civil andpolitical rights.” 17-Article 9, International Covenant on Economic, Social,
and Cultural Rights – Right to social security
Charter of the Fundamental Rights of the EU:-Article 15, Charter of the Fundamental Rights of the EU –Freedom to choose an occupation and right to engage inwork-Article 15, Explanation on Article 15 — Freedom tochoose an occupation and right to engage in work
(Explanations Relating to the Charter of Fundamental
Rights) in the Official Journal of the European Union C
303/17 - 14.12.2007-Article 16, Charter of the Fundamental Rights of the EU –Freedom to conduct a business-Article 16, Explanation on Article 16 — Freedom to
conduct a business (Explanations Relating to the Charterof Fundamental Rights) in the Official Journal of theEuropean Union C 303/17 - 14.12.2007-Article 17, Charter of the Fundamental Rights of the EU –Right to property-Article 17, Explanation on Article 17 — Right to property(Explanations Relating to the Charter of Fundamental
Rights) in the Official Journal of the European Union C
303/17 - 14.12.2007

17

CAHAI Feasibility Study, Council of Europe CAHAI (2020)23.
32

-Article 27, Charter of the Fundamental Rights of the EU –Workers’ rights to information and consultation with theundertaking-Article 27, Explanation on Article 27 — Workers’ rights to
information and consultation with the undertaking(Explanations Relating to the Charter of Fundamental
Rights) in the Official Journal of the European Union C
303/17 - 14.12.2007-Article 31, Charter of the Fundamental Rights of the EU –Fair and just working conditions-Article 31, Explanation on Article 31 — Fair and justworking conditions (Explanations Relating to the Charterof Fundamental Rights) in the Official Journal of theEuropean Union C 303/17 - 14.12.2007-Article 34, Charter of the Fundamental Rights of the EU –Social security and social assistance-Article 34, Explanation on Article 34 — Social security
and social assistance (Explanations Relating to theCharter of Fundamental Rights) in the Official Journal of
the European Union C 303/17 - 14.12.2007-Article 35, Charter of the Fundamental Rights of the EU –Healthcare-Article 35, Explanation on Article 35 — Healthcare(Explanations Relating to the Charter of Fundamental
Rights) in the Official Journal of the European Union C
303/17 - 14.12.2007
Universal Declaration of Human Rights:

33

-Article 3, Universal Declaration of Human Rights – Rightto life, liberty, and the security of person-Article 12, Universal Declaration of Human Rights – Rightto private home life
-Article 22, Universal Declaration of Human Rights – Right
to social security-Article 22, Universal Declaration of Human Rights –Workers’ rights

Accountability andeffective remedy(1, 2, 3, 5, 6, 7)-The right to an effective remedy for violation ofrights and freedoms (Art. 13 ECHR). This should
also include the right to effective and accessible
remedies whenever the development or use of AI
systems by private or public entities causes unjust
harm or breaches an individual’s legally protectedrights.

∼European Convention on Human Rights (ECHR):-Article 13, European Convention on Human Rights– Right to an effective remedy-Article 13, ‘Guide on Article 13 of the European
Convention on Human Rights.’, Council of Europe – Right
to an effective remedy
Charter of the Fundamental Rights of the EU:“AI systems must always remain under human control,
even in circumstances where machine learning or similar
techniques allow for the AI system to make decisions
independently of specific human intervention. Member
states must establish clear lines of responsibility forhuman rights violations that may arise at various phasesof an AI system lifecycle. Responsibility and accountability
for human rights violations that occur in the development,
deployment or use of AI Systems must always lie with anatural or legal person, even in cases where the measure
violating human rights was not directly ordered by a
responsible human commander or operator.

34-Article 47, Charter of the Fundamental Rights of the EU –Right to an effective remedy before a tribunal and to afair trial-Article 47, Explanation on Article 47 — Right to aneffective remedy and a fair trial (Explanations Relating to
the Charter of Fundamental Rights) in the Official Journalof the European Union C 303/17 - 14.12.2007
Universal Declaration of Human Rights:-Article 8, Universal Declaration of Human Rights – Right
to an effective remedy

Anyone who claims to be a victim of a human rights
violation arising from the development, deployment or
use by a public or private entity of an AI system should
be provided with an effective remedy before a national
authority. Moreover, member states should provide
access to an effective remedy to those who suspect thatthey have been subjected to a measure that has beensolely or significantly informed by the output of an AI
system in a non-transparent manner and without theirknowledge.
Effective remedies should involve prompt and adequate
reparation and redress for any harm suffered by thedevelopment, deployment or use of AI systems, and may
include measures under civil, administrative, or, whereappropriate, criminal law. NHRSs can be such a source ofredress, through rendering their own decisions inaccordance with their respective mandates.
Member states should provide individuals with the rightnot to be subject to a decision significantly affecting themthat is based on automated decision-making without
meaningful human intervention. At the very least, an
individual should be able to obtain human intervention insuch automated decision-making and have their views
taken into consideration before such a decision is
implemented.
Member states must ensure that individuals have accessto information in the possession of a defendant or a third
party that is relevant to substantiating their claim thatthey are the victim of a human rights violation caused by
an AI system, including, where relevant, training andtesting data, information on how the AI system was used,meaningful and understandable information on how the AI
system reached a recommendation, decision or

35

International Covenant on Civil and Political Rights:-Article 2, International Covenant on Civil and Political
Rights – Right to effective remedy
Council of Europe Resources:-MSI-AUT Responsibility and AI: A study of theimplications of advanced digital technologies (including AI
systems) for the concept of responsibility within a humanrights framework (2019)• This report outlines what AI is and how taskspecific technologies work, threats and harms
associated with advanced digital technologies, anda range of ‘responsibility models’ for the adverse
impacts of AI systems• The main recommendations from this report are“effective and legitimate mechanisms that will
prevent and forestall human rights violations”,
policy choices regarding responsibility models forAI systems, support of technical research involving
human rights protections and ‘algorithmic
auditing’, and the presence of legitimate
governance mechanisms for the protection ofhuman rights in the digital age

prediction, and details of how the AI system’s outputs
were interpreted and acted on.” 18

Democracy (1, 2, 4,
5, 7, 8)-The right to freedoms of expression, assembly, andassociation (Art. 10 and 11 ECHR).
-The right to vote and to be elected, the right tofree and fair elections, and in particular universal,
equal and free suffrage, including equality ofopportunities and the freedom of voters to form an
opinion. In this regard, individuals should not be
subjected to any deception or manipulation.
-The right to (diverse) information, free discourse,
and access to plurality of ideas and perspectives.
-The right to good governance.

-Article 3 of Protocol No.1, European Convention on
Human Rights – Right to free elections- Article 3 of Protocol No. 1, Guide on Article 3 of Protocol
No. 1 to the European Convention of Human Rights –Right to free elections-Article 10, European Convention on Human Rights– Freedom of expression-Article 10, ‘Guide on Article 10 of the European
Convention on Human Rights’, Council of Europe –Freedom of expression∼“Freedom of expression: Member states should be mindfulof the obligation to create a diverse and pluralistic
information environment and the adverse impact AIdriven content moderation and curation can have on theexercise of the right to freedom of expression, access to
information, and freedom of opinion. Member states arealso encouraged to consider taking appropriate measures
to regulate technology monopolies to prevent the adverse
effects of concentration of AI expertise and power on thefree flow of information.
Freedom of assembly and association: Special attention
should be paid to the impact the use of AI systems incontent moderation can have on the freedom of assembly
and association, especially in contexts where these
freedoms are difficult to exercise offline. The use of facial
18

European Convention on Human Rights (ECHR):-Article 11, European Convention on Human Rights– Freedom of assembly and association-Article 11, ‘Guide on Article 11 of the European
Convention on Human Rights’, Council of Europe –Freedom of assembly and association
Charter of the Fundamental Rights of the EU:-Article 11, Charter of the Fundamental Rights of the EU –Freedom of expression and information-Article 11, Explanation on Article 11 — Freedom of
expression and information (Explanations Relating to theCharter of Fundamental Rights) in the Official Journal of
the European Union C 303/17 - 14.12.2007

Council of Europe Commissioner for Human Rights (2019) – “Unboxing AI: 10 steps to protect Human Rights”
36

recognition technology should be strictly regulated bymember states, including through legislation setting out
clear limitations for its use, and public transparency toprotect the effective exercise of the right to freedom ofassembly.” 19∼“Well-functioning democracies require a well-informed
citizenry, an open social and political discourse andabsence of opaque voter influence.
This requires a well-informed citizenry. In informationsocieties citizens can only select to consume a small
amount of all the available information. Search engines,
social media feeds, recommender systems and many
news sites employ AI to determine which content iscreated and shown to users (information personalization).
If done well, this could help citizens to better navigate theflood of available information and improve theirdemocratic competences, for instance by allowing them toaccess resources in other languages through translation
tools. However, if AI determines which information isshown and consumed, what issues are suppressed in theflood of online information and which are virally amplified,
this also brings risks of bias and unequal representationof opinions and voices.
AI-driven information personalisation is enabled by theconstant monitoring and profiling of every individual.
Driven by commercial or political motives thistechnologically-enabled informational infrastructure of oursocieties could amplify hyper-partisan content one islikely to agree with and provide an unprecedented
19-Article 12, Charter of the Fundamental Rights of the EU –Freedom of assembly and association-Article 12, Explanation on Article 12 — Freedom ofassembly and of association (Explanations Relating to theCharter of Fundamental Rights) in the Official Journal of
the European Union C 303/17 - 14.12.2007
Universal Declaration of Human Rights:-Article 19, Universal Declaration of Human Rights – Rightto freedom of opinion and expression-Article 20, Universal Declaration of Human Rights – Rightto freedom of peaceful assembly and association
International Covenant on Civil and Political Rights:-Article 19, International Covenant on Civil and Political
Rights – Freedom of expression-Article 21, International Covenant on Civil and Political
Rights – Freedom of assembly-Article 22, International Covenant on Civil and Political
Rights – Freedom of association-Article 25, International Covenant on Civil and Political
Rights – Right to participate in public affairs, good
governance, and elections
Council of Europe Resources:

Council of Europe Commissioner for Human Rights (2019) – “Unboxing AI: 10 steps to protect Human Rights”

37

powerful tool for individualised influence. As aconsequence it may undermine the shared understanding,mutual respect and social cohesion required for
democracy to thrive. If personal AI predictions become
very powerful and effective, they may even threaten toundermine the human agency and autonomy required formeaningful decisions by voters.
Thirdly, AI can undermine a fair electoral process. Political
campaigns or foreign actors can use (and have beenusing) personalised advertisements to send different
messages to distinct voter groups without public
accountability in the agora. However, it should be notedthat it remains uncertain exactly how influential microtargeted advertisement is. AI can also be used to createand spread misinformation and deep fakes, in the form of
text, pictures, audio or video. Since these are hard to
identify by citizens, journalists, or public institutions,misleading and manipulating the public becomes easierand the level of truthfulness and credibility of media anddemocratic discourse may deteriorate.” 20-Venice Commission: Principles for a fundamental rightscompliant use of digital technologies in electoral
processes (2020)• Emphasized the need for a human rightscompliant approach to eight principles involving
the use of digital technologies in elections-Parliamentary Assembly, Need for democratic
governance of artificial intelligence, Council of Europe∼“Democracy is government of the people by the people
for the people. It provides checks against theconcentration of power in the hands of a few and canfunction properly only if based on sound institutions whichenjoy confidence of an active, committed, and informed
citizenry and are able to provide for dynamic balance ofinterests of constituents. The crisis of modern
democracies touches almost all elements of democraticorder, including erosion of, and loss of confidence ininstitutions, mis- and disinformation of the public, breakup of cohesion and polarisation of society. Modern
20

Muller, C. The Impact of Artificial Intelligence on Human Rights, Democracy and the Rule of Law. Council of Europe, CAHAI(2020)06-fin.
38

technologies, including AI-based systems may both help
resolve and aggravate this crisis.
The use of AI by humans is not neutral. It can be used tostrengthen government accountability and can produce
many benefits for democratic action, participation, andpluralism, making democracy more direct and responsive.
However, it can also be used to strengthen repressive
capabilities and for manipulation purposes. Indeed, therapid integration of AI technologies into modern
communication tools and social media platforms provides
unique opportunities for targeted, personalised and often
unnoticed influence on individuals and social groups,
which different political actors may be tempted to use to
their own benefit.
The experience of the last few years helps to identify
some key areas where the use of AI-based technology
can threaten to undermine and destabilise democracy,
including, inter alia: (a)access to information(misinformation, “echo chambers” and erosion of criticalthinking); (b)targeted manipulation of citizens;(c)interference in electoral processes; (d) erosion of civilrights; (e) shifts of financial and political power in thedata economy.
Moreover, the broad use by States of AI-based
technologies to control citizens such as automatedfiltering of information amounting to censorship, andmass surveillance using smartphones and closed-circuit
television coupled with vast integrated databases, may
lead to the erosion of political freedoms and theemergence of digital authoritarianism – a new social ordercompeting with democracy.” 21

Bergamini, D. (Rapporteur) (2020). Need for democratic governance of artificial intelligence. Council of Europe, Committee on Political
Affairs and Democracy, Parliamentary Assembly.
21

39

Rule of law (2)-The right to a fair trial and due process (Art. 6
ECHR). This should also include the possibility ofreceiving insight into and challenging AI-informed
decisions in the context of law enforcement or
justice, including the right to review of suchdecisions by a human. The essential requirementsthat secure impacted individuals' access to the rightof a fair trial must also be met: equality of arms,
right to a natural judge established by law, theright to an independent and impartial tribunal, andrespect for the adversarial process.
-The right to judicial independence and impartiality,
and the right to legal assistance.
-The right to an effective remedy (Art. 13 ECHR),
also in cases of unlawful harm or breach anindividual’s human rights in the context of AI
systems.

∼“The rule of law requires respect for principles such aslegality, transparency, accountability, legal certainty, nondiscrimination, equality and effective judicial protection –which can be at risk when certain decisions are delegatedto AI systems. In addition, AI systems can also negatively
affect the process of law-making and the application of
the law by judges. Concerns have also been expressed on
the possible negative effects of some AI applications usedin judicial systems or connected areas. Such use could
pose a challenge to the right to a fair trial enshrined in
Article 6 of the ECHR, of which components such as theright to an independent and impartial judiciary, the rightto a lawyer or the principle of equality of arms in judicial

40

European Convention on Human Rights (ECHR):-Article 6, European Convention on Human Rights – Right
to a fair trial-Article 6, ‘Guide on Article 6 of the European Convention
on Human Rights.’, Council of Europe – Right to a fairtrial
-Article 13, European Convention on Human Rights– Right to an effective remedy-Article 13, ‘Guide on Article 13 of the European
Convention on Human Rights.’, Council of Europe – Right
to an effective remedy
Charter of the Fundamental Rights of the EU:-Article 47, Charter of the Fundamental Rights of the EU –Right to an effective remedy before a tribunal and to afair trial-Article 47, Explanation on Article 47 — Right to aneffective remedy and a fair trial (Explanations Relating to
the Charter of Fundamental Rights) in the Official Journalof the European Union C 303/17 - 14.12.2007
Universal Declaration of Human Rights:-Article 8, Universal Declaration of Human Rights – Right
to an effective remedy- Article 10, Universal Declaration of Human Rights –Right to a fair trialproceedings are key elements that are also essential forthe effective implementation of the rule of law.
Moreover, companies face increased pressure, including
through regulation, to take decisions on the legality ofcontent that is shown on their platform. Since social
media platforms have become the new “public square”,
their own terms of service essentially set the rules of how
freedom of expression manifests itself online, but withfewer safeguards than in more traditional public settings.
It is, however, essential that states can and do continue
to fulfil their responsibility for the protection of the rule oflaw.” 22∼
“The fact that AI can perpetuate or amplify existing
biases, is particularly pertinent when used in law
enforcement and the judiciary. In situations where
physical freedom or personal security is at stake, such aswith predictive policing, recidivism risk determination andsentencing, the right to liberty and security combinedwith the right to a fair trial are vulnerable. When an AIsystem is used for recidivism prediction or sentencing it
can have biased outcomes. When it is a black box, it
becomes impossible for legal professionals, such as
judges, lawyers and district attorneys to understand thereasoning behind the outcomes of the system and thus
complicate the motivation and appeal of the judgement.
…Public institutions are held to a higher standard when it
comes to their behaviour towards individuals and society,which is reflected in principles such as justification,
proportionality and equality. AI can increase the efficiency
22

CAHAI Feasibility Study, Council of Europe CAHAI (2020)23.
41

International Covenant on Civil and Political Rights:-Article 2, International Covenant on Civil and Political
Rights – Right to effective remedy-Article 14, International Covenant on Civil and Political
Rights – Right to fair trial
Council of Europe Resources:-European Commission for the Efficiency of Justice,‘European ethical charter on the use of Artificial
Intelligence in judicial systems and their environment’ –Council of Europe

of institutions, yet on the other it can also erode theprocedural legitimacy of and trust in democratic
institutions and the authority of the law.
Courts, law enforcement and public administrations could
become more efficient, yet at the cost of being moreopaque and less human agency, autonomy and oversight.
Similarly, whereas previously courts were the only ones
to determine what counts as illegal hate speech, today
mostly private AI systems determine whether speech istaken down by social media platforms. These AI systems
de facto compete for authority with judges and the law
and In general, AI can contribute to developing judicial
systems that operate outside the boundaries andprotections of the rule of law.
Automated online dispute resolutions provided by privatecompanies are governed by the terms of service rather
than the law that do not award consumers the samerights and procedural protections in public courts.” 23
Related enumerations of human rights, fundamental freedoms, and elements of democracy and the rule of law:
1: Council of Europe Commissioner for Human Rights (2019) – “Unboxing AI: 10 steps to protect Human Rights”
2. OECD AI Principles and Classification Framework
3. European Agency for Fundamental Rights (2020) – “Getting the Future Right: Artificial Intelligence and Fundamental Rights”
4. Aizenberg & van den Hoven (2020) – “Designing for human rights in AI.”
5. UNESCO (2021a/b) – Recommendations on the Ethics of Artificial Intelligence (two drafts, 1-25, 26-134)
6. Loi, et al. (2021) Algorithm Watch – “Automated Decision-Making Systems in the Public Sector: An Impact Assessment Tool for Public
Authorities”
7. High-level Expert Group on Artificial Intelligence, European Commission (2020) – “The Assessment List for Trustworthy Artificial Intelligence(ALTAI) for self-assessment”
8. Mantelero & Esposito (2021) – An evidence-based methodology for human rights impact assessment (HRIA) in the development of AI dataintensive systems
23

Muller, C. The Impact of Artificial Intelligence on Human Rights, Democracy and the Rule of Law. Council of Europe, CAHAI(2020)06-fin.
42

Determining the salience of the rights, freedoms, and elements ofdemocracy and the rule of law to the AI system you are planning to build

After reviewing Table 1—and carrying out further desk research (or consulting
experts) to clarify questions pertaining to any of the rights or freedoms about
which you are unsure—you are in a good position to begin considering the humanrights, fundamental freedoms, and the elements of democracy and the rule of law
that could be impacted by your system.
Go through each principle and priority listed in Table 1 (and the corresponding
rights and freedoms enumerated in the adjacent columns) and ask:•••

How, if at all, are the rights, freedoms, or elements of democracy and therule of law that are associated with this principle/priority salient to the AI
system we are planning to build, given its intended purpose and thecontexts in which it will be used?
How could the rights, freedoms, or elements of democracy and the rule oflaw that are associated with this principle/priority be impacted by the AI
system we are planning to build?
If things go wrong in the implementation of our AI system or if it is used
out-of-the-scope of its intended purpose and function, what harms could bedone to impacted rights-holders in relation to the rights, freedoms, orelements of democracy and the rule of law that are associated with thisprinciple/priority?

Answering these questions as part of the PS Report will help you prepare for
conducting the PCRA, where you will further assess the extent to which the AI
system poses probable harms to specific rights and freedoms.

Activity 4: Map Governance Workflow
The last activity of the planning and scoping phase of your PS Report involves
mapping the governance workflow of your project. Governance is defined in the
recently-adopted standard ISO 370000 as ‘the system by which the wholeorganization is directed, controlled, and held accountable to achieve its core
purpose in the long run’. 24 The governance workflow is the foundation of an
anticipatory approach to your project that supports duty-bearers with ex-anteprocesses of accountability and auditing throughout the project lifecycle. 25
https://committee.iso.org/sites/tc309/home/projects/ongoing/ongoing-1.html
The algorithmic impact assessment literature describes the value and importance ofboth ex-post and ex-ante assessments, including as they pertain to different features oforganisational governance. However, recent work tends to place greater importance on
the effectiveness of ex-ante assessments. See, for example, Moss, E., Watkins, E., Metcalf,J., & Elish, M. C. (2020). Governing with Algorithmic Impact Assessments: Six
Observations. SSRN Electronic Journal. Leslie specifically argues that effective governanceincludes structures of accountability throughout the development and implementation of

24
25

43

Mapping a governance workflow, therefore, creates the baseline conditions forholding an organization actively accountable for the fulfilment of its goals andprinciples, including its duty to uphold human rights, democracy, and the rule oflaw.
Mapping the governance workflow is not only a process of articulating guiding
principles and assigning responsibility; it is an opportunity to ensure a sufficiencyof perspectives and understandings that extends beyond the organization to thesocieties and material conditions potentially impacted by an AI system. For
example, as technical systems can significantly affect the social and economic
conditions of human lives, systems of governance should be inclusive of a
sufficiently diverse set of voices to increase the degree of critical reflection andthe mitigation of institutional and individual biases that can lead to thereproduction and amplification of unjust social structures. 26
Establishing a governance workflow is essential where the proposed system
operates in an incomplete or uncertain regulatory environment, but an establishedgovernance workflow remains important even within tightly regulated contexts asa feature of collaborative governance in which structures of internal accountability
complement regulatory structures and vice-versa. 27
Questions:•••

What roles are involved in each of the project’s phases?
o Design
o Developmento Deployment
What are the responsibilities associated with each of these roles?
How are each of the duty-bearers involved in the project assigned
responsibility for the system’s potential impacts to human rights,
democracy, and the rule of law?
o Does this distribution establish a continuous chain of humanaccountability throughout the design, development, anddeployment of this project? If so, how?

AI systems rather than retrospectively. Leslie, D. (2019). Understanding artificial
intelligence ethics and safety. The Alan Turing Institute. And McGregor et al. argues thathuman rights doctrine provides guidance for the governance of AI systems andorganisations, including decision structures to guide the project lifecycle. McGregor, L.,
Murray, D., & Ng, V. (2019). International human rights law as a framework for algorithmic
accountability. International and Comparative Law Quarterly, 68(2), 309–343.
26
For further discussion on the components of an effective structure of internal governancefor AI system development, including the requirements for critically assessing a system’ssocial impact, see Raji et al. (2020). Closing the AI Accountability Gap: Defining an Endto-End Framework for Internal Algorithmic Auditing. Proceedings of the 2020 conference
on fairness, accountability, and transparency. Similarly, Kemper & Kolkman argue for theimportance of a critical audience to ensure that algorithmic systems are truly accountablebeyond the firm. Kemper, J., & Kolkman, D. (2019). Transparent to whom? No algorithmic
accountability without a critical audience. Information, Communication & Society, 22(14),2081–2096.
27
Complementary governance is described by Kaminski (2018). Binary Governance:
Lessons from the GDPR’s approach to algorithmic accountability. S. Cal. L. Rev., 92, 1529.
44

Do all duty-bearers understand the responsibility that has been
assigned to them?
What logging protocol will be established for documenting workflow
activities?
o Does this protocol enable external auditing and oversight of thedesign, development, and deployment of this project? If so, how?
Can responsible duty bearers be traced in the event that the human rights
or fundamental freedoms of rights-holders are harmed by this system? If
so, how do the project’s distribution of responsibilities and logging
protocol enable this?
If you are procuring parts or elements of the system from third-party
vendors, suppliers, sub-contractors, or external developers, how are you
instituting appropriate governance controls that will establish end-to-end
accountability, traceability, and auditability?
If any data being used in the production of the AI system will be acquired
from a vendor or supplier, how are you instituting appropriate governancecontrols that will establish end-to-end accountability, traceability, andauditability across the data lifecycle?
o

•

•

•

•

45

Project Summary Report Template
Project Phases and Questions

Responses

ACTIVITY 1: Outline Project, Use Context, Domain, and Data
PROJECTHow would you describe your organisation andwhat sort of services or products do you typicallyprovide (beyond the system under consideration)?
What AI system is being built and what type of
product or service will it offer?
What benefits will the system bring to its users
and customers, and will these benefits be widely
accessible?
Which organisation(s)—yours, other suppliers, orother providers—are responsible for building thisAI system?
Which parts or elements of the AI system, if any,
will be procured from third-party vendors,
suppliers, sub-contractors, or external
developers?

46

Which algorithms, techniques, and model types
will be used in the AI system? (Provide links totechnical papers where appropriate)
In a scenario where your project optimally scales,
how many people will it impact, for how long, andin what geographic range (local, national, global)?
(Describe your rationale)
USE CONTEXTWhat is the purpose of this AI system and inwhich contexts will it be used? (Briefly describe a
use-case that illustrates primary intended use)
Is the AI system’s processing output to be used ina fully automated way or will there be somedegree of human control, oversight, or input
before use? (Describe)
Will the AI system evolve or learn continuously in
its use context, or will it be static?
To what degree will the use of the AI system betime-critical, or will users be able to evaluate
outputs comfortably over time?
What sort of out-of-scope uses could users
attempt to apply the AI system, and what dangers
may arise from this?
DOMAIN47

In what domain will this AI system operate?
Which, if any, domain experts have been or will
be consulted in designing and developing the AI
system?
DATA
What datasets are being used to build this AI
system?
Will any data being used in the production of the
AI system be acquired from a vendor or supplier?
(Describe)
Will the data being used in the production of the
AI system be collected for that purpose, or will it
be re-purposed from existing datasets? (Describe)
What quality assurance and bias mitigation
processes do you have in place for the data
lifecycle—for both acquired and collected data?

ACTIVITY 2: Identify Stakeholders
Who are the rights-holders, duty bearers andother relevant parties that may be impacted by,or may impact, the project?
Do any of these rights-holders possess sensitiveor protected characteristics that could increase
their vulnerability to abuse or discrimination, or
for reason of which they may require48

additional protection or assistance with respect tothe impacts of the project? If so, whatcharacteristics?
Could the outcomes of this project present
significant concerns to specific groups of rightsholders given vulnerabilities caused or
precipitated by their distinct circumstances?
If so, what vulnerability characteristics expose
them to being jeopardized by project outcomes?
What affected group or groups of rights-holders
could these protected or contextual vulnerability
characteristics or qualities represent?
How could the distribution of power relations (i.e.,
the relative advantages and disadvantages)
between rights-holders and duty-bearers affectthe way the benefits and risks of the project
are allocated?

ACTIVITY 3: Scope Impacts to Human Rights, Democracy, and Rule of Law
Instructions: Go through each principle and priority listed in Table 1 (and the corresponding rights and freedoms enumerated
in the adjacent columns) and ask and respond to the following questions:
1. How, if at all, are salient to the AI system we are planning to build, given its intended purpose and the contexts in which
it will be used?
2. How could the rights, freedoms, or elements of democracy and the rule of law that are associated with this principle/priority
be impacted by the AI system we are planning to build?
3. If things go wrong in the implementation of our AI system or if it is used out-of-the-scope of its intended purpose andfunction, what harms could be done to impacted persons in relation to the rights, freedoms, or elements of democracy andthe rule of law that are associated with this principle/priority?

49

1.

Respect for and Protection of Human
Dignity

2.

3.

1.

Protection of Human Freedom andAutonomy

2.

3.

50

1.

Prevention of Harm and Protection of theRight to Life and Physical, Psychological,
and Moral Integrity

2.

3.

1.

Non-discrimination, Fairness, andEquality

2.

3.

Data Protection and the Right to Respectof Private and Family Life

1.

51

2.

3.

1.

2.

Economic and Social Rights
3.

1.

Accountability and Effective Remedy
2.

52

3.

1.

2.

Democracy
3.

1.

Rule of Law

2.

53

3.

ACTIVITY 4: Map Governance Workflow
What roles are involved in each of the project phases?
Design Phase:

Development Phase:

Deployment Phase:
What are the responsibilities of each of these roles?
Design Roles:

Development Roles:

Deployment Roles:

54

How are each of these duty bearers assigned
responsibility for the system’s potential impacts to
Human Rights, Democracy, and the Rule of Law?
oDoes this distribution establish a continuous
chain of human accountability throughout
the design, development, and deploymentof this project? If so, how?

What logging protocol is established for
documenting workflow activities?
o

Does this protocol enable auditing andoversight of the design, development, anddeployment of this project? If so, how?

Can responsible duty bearers be traced in theevent that the human rights or fundamental
freedoms of rights-holders are harmed by thissystem? If so, how do the project’s distribution of
responsibilities and logging protocol enable this?
If you are procuring parts or elements of thesystem from third-party vendors, suppliers, subcontractors, or external developers, how are you
instituting appropriate governance controls thatwill establish end-to-end accountability,
traceability, and auditability?

55

If any data being used in the production of the AI
system will be acquired from a vendor or supplier,
how are you instituting appropriate governancecontrols that will establish end-to-end
accountability, traceability, and auditability acrossthe data lifecycle?

56

PCRA: Document Overview
Introduction
After you have done your preparatory desk research and completed the initial
iteration of your PS Report, you will be ready to carry out your Preliminary
Context-Based Risk Analysis (PCRA). The PCRA serves several important
purposes:
1. It operates as an indicator of the pre-emptive actions needed to ensure
non-harm for projects that pose unacceptable risk to impacted rightsholders. In this way, it facilitates the operationalization of the risk
assessment component of the precautionary principle. 28
2. It helps you identify the risk factors that should be considered as you
endeavour to reduce and mitigate the potential adverse impacts of yourprospective project on the fundamental freedoms and human rights ofaffected rights-holders.
3. It directs you to specific actions that you then need to take in your impact
assessment process (HUDERIA) so that sufficient consideration can begiven to each of the risk factors detected and to the salient rights andfreedoms that could be impacted by the prospective AI system. It also
directs you to specific goals, properties, and areas that you should focus
on in your subsequent risk management and assurance processes(HUDERAC) to reduce and mitigate associated risks.
4. Finally, it helps you establish a proportionate approach to your impact
assessment and assurance practices, and to the level of stakeholder
engagement that you will include across your project lifecycle. The
proportionality principle counsels that efforts of impact assessment, risk
management, and stakeholder involvement should be proportionate to thelikelihood and extent of risks for adverse impacts on human rights,
democracy, and the rule of law associated with an AI system. The reportthat is generated when you complete your PCRA will help you understand,in a provisional way, the risk level of your system, so that you can takeappropriate risk management actions and engage stakeholders in a
proportionate way.

The European Commission guidance on the precautionary principles states that it“should be considered within a structured approach to the analysis of risk which comprises
three elements: risk assessment, risk management, risk communication. The PCRA
supports the risk assessment aspect, whereas—as is discussed below—the Human Rights,
Democracy, and the Rule of Law Assurance Case (HUDERAC) primarily supports the risk
management and risk communication elements. See: EU (2000). Communication from thecommission on the precautionary principle, Communication. Brussels: Commission of theEuropean Communities; Bourguignon, D. (2016) European Parliament, and DirectorateGeneral for Parliamentary Research Services, The precautionary principle: definitions,applications and governance: in-depth analysis. Luxembourg: Publications Office.
28

57

PCRA Section One: Identifying Risk Factors
In the first section of the PCRA, you will be asked to answer a series of questions
that are intended to help you identify both modifiable and circumstantial risk
factors to consider as you embark on the impact assessment and risk
management and assurance elements of your HUDERAF. The term “risk factor” ishere taken to mean the characteristics of an AI innovation context at the technical,
sociotechnical, historical, legal, economic, political, or practical levels that precedeand are associated with a higher likelihood of outcomes that negatively impact
human rights, democracy, and the rule of law. 29 These factors are not necessarily
to be treated as causes of adverse impacts but rather as conditions that are
correlated with an increased chance of harm and that need to be anticipated andconsidered in risk mitigation efforts.
Those risk factors that emerge externally from the technical, sociotechnicalhistorical, legal, economic, or political environments within which the design,development, and deployment of AI systems occur and that are thereby lesscontrollable are called “circumstantial risk factors.” Those risk factors that emerge
internally from the actual practices of producing and using AI technologies, andthat are thus more controllable, are called “modifiable risk factors”.
To remain as “algorithm neutral” 30 as possible, the PCRA focuses on the spectrumof risk factors that surround the practical contexts of designing, developing, anddeploying AI systems rather than on the technical details or underlying
specifications of AI applications. Building on OECD’s AI classification scheme 31 andrelated work, the questions in the first section focus on risk factors arising fromboth the use contexts and the design, development, and deployment contexts of
the AI project lifecycle. In addition to this, the final fifteen questions focus on risk
factors that arise in the immediate context human rights and fundamental
freedoms. A summary of the question headings and sub-headings shows how thisspectrum of concerns stretches from use contexts and design, development, anddeployment contexts to rights and freedoms contexts:

The concept of “risk factors,” as used here, was first fully developed in medical andpublic health contexts but has since seen wide application in public policy and the social
sciences. For background discussions, see Mercy, J.A., and O’Carroll, P.W. 1998. New
directions in violence prevention: The public health arena. Violence and Victims 3(4):285–
301.; Mrazek, P.J., and Haggerty, R.J., eds. 1994. Reducing Risks for Mental Disorders:
Frontiers for Preventative Intervention Research. Washington, DC: National Academy
Press.; Shader, M. (2001). Risk factors for delinquency: An overview. Washington, DC:
US Department of Justice, Office of Justice Programs, Office of Juvenile Justice andDelinquency Prevention.; Sorooshian, S., & Mun, S. Y. (2020). Literature review: Criticalrisk factors affecting information-technology projects. Calitatea, 21(175), 157-161.
30
Following both the CAHAI Feasibility Study, Council of Europe CAHAI (2020)23 (p. 4)
and CAHAI-PDG (2021)05rev.
31
OECD (2021). Framework to Classify AI Systems. Public Consultation.
29

58

Use context (Questions 1-20)
•
•
•
•
•
•
•
•
•
•

Sector or domain in which the system is being built
Existing law and regulatory environment of the sector or domain
Impact-level of the system
Prohibited systems and uses
Scope of deployment (breadth and temporality)
Technological maturity
Existing system (human or technological) that the application is replacing
Bias and discrimination in sector or domain context
Environmental context
Cybersecurity context

Data Lifecycle Context (Questions 21-40)
•
•
•
•
•

Data quality, integrity, and provenance
Means and methods of data collection
Data types
Dataset linkage
Data labelling and annotating practices

Goal Setting and Problem Formulation Context (Questions 41-42)••

Decision to design
Definition of outcome

Model Design & Development Context (Questions 43-46)•••

AI model characteristics
Pre-processing and feature engineering
Model selection

Model Output & Implementation Context (Questions 47-52)•••

Model inference
Model verification and validation
Model accuracy and performance metrics

System-User Interface and Human Factors Context (Questions 53-55)••

Implementers or users of the system
Level of automation/level of human involvement and choice

Rights & Freedoms Context (Questions 56-71)
•
•
•
•
•
•

Respect for and protection of human dignity
Protection of human freedom and autonomy
Non-discrimination, fairness, and equality
Data protection and privacy context
Accountability and access to justiceSocial and economic rights

59

Depending on the form that the section one questions take, certain responses toeach question will trigger one of three classes of risk factors, prohibitive, major,
or moderate:

Prohibitive risk factor

Major risk factor

Moderate risk factor

Prohibitive risk factors indicate the presence ofdeterminants of potential harms that trigger the
precautionary principle and precipitate preemptive measures to prevent adverse impacts
on the human rights and fundamental freedoms
of affected persons, democracy, and the rule oflaw. Pre-emptive measures are appropriate
where the severity, scale, and irremediablenessof the potential harm outweigh levels of risk
reduction and mitigation.
Major risk factors indicate the presence ofdeterminants of potential harms that are
directly or indirectly associated with significant
risks of adverse impacts on the human rights
and fundamental freedoms of affected persons,democracy, and the rule of law but that provide
opportunities for risk reduction and mitigationthat make the risks posed tolerable.
Moderate risk factors indicate the presence ofdeterminants of potential harms that are
directly or indirectly associated with risks ofadverse impacts on the human rights andfundamental freedoms of affected persons,democracy, and the rule of law but that provide
opportunities for risk reduction and mitigationthat make the risks posed broadly acceptable.

Upon completion of the entire PCRA, a summary report will automatically begenerated (See PCRA Output Example) which compiles and organizes the answers
that flagged up risk factors into the three risk factor categories. The reportprovides prompts for each of these answers that direct you to specific actions totake in your impact assessment process and to specific goals, properties, andareas that you should focus on in your subsequent risk management andassurance processes to reduce and mitigate associated risks. For example, if you
answered “yes” to Question 2, “Does the sector or domain in which the AI system
will operate include vulnerable groups (or groups with protected characteristics)
who may be significantly or disproportionately impacted by the design and use of
the system?”, the following message will be displayed in the summary report:

60

Because you answered Yes to Question 2:

Major circumstantial risk factor
Where the sector or domain in which the AI system will operate includes vulnerable
groups (or groups with protected characteristics) who may be significantly or
disproportionately impacted by the design and use of the system, this presents amajor circumstantial risk factor for adverse impacts on the human rights andfundamental freedom of persons• Actions to take for your HUDERIA: Make sure to focus upon
considerations surrounding the potential impacts of your AI system on
fairness, non-discrimination, equality, diversity, and inclusiveness• Goals, properties, and areas to focus on in your HUDERAC: Fairness(non-discrimination, equality, bias mitigation, diversity, and inclusiveness)

The enumeration of risk factors and corresponding practical recommendations thatare presented in the PCRA summary report is intended to support a deliberate,
anticipatory, and reflective approach to your impact assessment, risk
management, and assurance practices. It is meant to bolster safe and responsible
AI innovation processes that respect human rights, democracy, and the rule oflaw.

PCRA Section Two: Risks of Adverse Impacts on the Human Rights
and Fundamental Freedoms of Persons, Democracy, and the Rule ofLaw
In the second section of your PCRA, you will be asked to provide a preliminary
estimation of the likelihood of the potential adverse impacts that your system
could have on human rights, democracy, and the rule of law. This will involve
drawing on the stakeholder analysis and the determination of salient rights andfreedoms you carried out as part of your initial PS reporting to think through therelevance and likelihood of the harms indicated in the prompts. Each prompt will
be formed as a statement with likelihood/probability options below it.
These forty prompts encompass the categories of principles and priorities includedin Table 1 of the Project Planning and Scoping Summary section of the PCRA. The
range of potential adverse impacts covered in section two of the PCRA reflect theconcerns raised in the CAHAI’s Feasibility Study and in the multi-stakeholder
consultation that followed the plenary’s adoption of this text in December 2020.

61

It also reflects adjacent work done by the Council of Europe, UNESCO, theEuropean Commission, OECD, and the European Agency for Fundamental Rights. 32
The likelihood options that are provided for each of the section two prompts should
be understood as follows 33:

Likelihood level

Description

Score

Unlikely

The risk of adverse impact is low, improbable, or
highly improbable

1

Possible

The risk of adverse impact is moderate; the harmis possible and may occur

2

LikelyThe risk of adverse impact is high; it is probablethat the harm will occur

3

Very Likely

The risk of adverse impact is very high; it is highly
probable that the harm will occur

4

Not Applicable

It can be claimed with certainty that the risk ofadverse impact indicated in the prompt does not
apply to the AI system.

0

The likelihood score that is indicated in the far-right column is applied in thesummary report as part of the calculation of a risk index number (RIN) that will
be used to generate preliminary recommendations for proportionate risk
management and assurance practices and stakeholder engagement given the risk
level of the prospective AI system. After you have completed all the PCRAFor other relevant sources, see: Council of Europe Commissioner for Human Rights(2019) – “Unboxing AI: 10 steps to protect Human Rights”; Council of Europe,“Recommendation CM/Rec (2020) of the Committee of Ministers to member States on thehuman rights impacts of algorithmic systems”; Muller, C. The Impact of Artificial
Intelligence on Human Rights, Democracy and the Rule of Law. Council of Europe,
CAHAI(2020)06-fin; Bergamini, D. (Rapporteur) (2020). Need for democratic governanceof artificial intelligence. Council of Europe, Committee on Political Affairs and Democracy,
Parliamentary Assembly; UNESCO (2021a/b) – Recommendations on the Ethics of Artificial
Intelligence (two drafts, 1-25, 26-134); High-level Expert Group on Artificial Intelligence,
European Commission (2020) – “The Assessment List for Trustworthy Artificial Intelligence(ALTAI) for self-assessment”; European Agency for Fundamental Rights (2020) – “Getting
the Future Right: Artificial Intelligence and Fundamental Rights”.
33
Likelihood level descriptions are drawn partly from the probability table presented in(Mantelero & Esposito, 2021, p. 19), but whereas the latter includes the category of
exposure in the description and calculation of likelihood levels, the PCRA counts only theprobability. In the PCRA, the degree of human exposure is covered in the calculation ofrisk level by the variable of “number of rights-holders affected,” the information for which
is drawn from answers given to Question 10 in section 1 of the PCRA. Mantelero, A., andEsposito, S. (2021). An evidence-based methodology for human rights impact assessment(HRIA) in the development of AI data-intensive systems. Computer Law & Security Review.
32

62

questions, the likelihood scores from the answers you have given to the section 2questions will be compiled in a table in the summary report that produces anestimated risk level for each of potential adverse impacts that have received alikelihood score of 1 or above:(Table taken from PCRA Output Example). Note that numbers in the “Gravity Potential” and “Affected RightsHolders Affected” columns reflect ordinal scales rather than real values and will be more fully described below)

The following formula is used to calculate the estimated RIN for potential harm
event 𝑖𝑖:
𝑅𝑅𝑅𝑅𝑅𝑅𝑖𝑖 = 𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝐿𝐿𝐿𝐿𝐿𝐿𝐿𝐿𝐿𝐿𝐿𝐿ℎ𝑜𝑜𝑜𝑜𝑜𝑜𝑖𝑖 34
In this equation, the risk index number of a potential harm event 𝑖𝑖 is determinedby an additive combination of the severity of the potential harm and its estimatedlikelihood. 35 This calculation works from the common risk analysis formulation ofNote that at this preliminary scoping phase, where risk analysis is helping to determine
the proportionality of stakeholder engagement and risk management strategies, thevariable of probability/likelihood figures in to impact estimation in a way that will differ
from the approach taken in the HUDERIA phase of impact assessment, where severity
assessment and impact management prioritization does not include probability
considerations (following UNGP guidance) or incorporates them only as a secondary
concern after severity assessment determines impact mitigation measures. As the DHRI
notes, at the impact assessment stage, “severity does not include consideration ofprobability; instead, it prioritises a focus on the human rights consequences of the impact.
This is not to say that consideration of probability is irrelevant. Consideration of probability
will necessarily be involved in initial issues scoping.” See See Figure 1 in Götzmann, N.,
Bansal, T., Wrzoncki, E., Veiberg, C. B., Tedaldi, J., & Høvsgaard, R. (2020). Human rights
impact assessment guidance and toolbox. The Danish Institute for Human Rights. p. 90.
35
The RIN calculation uses additive rather than multiplicative combination to reflect theapproximately logarithmic relation of categories and the nature of the potential harms,
following (Levine 2012), (Duijm 2015), and (Rausand and Haugen 2020). See: Levine, E.
34

63

risk level as the product of severity (or impact/consequence) and likelihood (orprobability/ occurrence), 36 but, in this case, severity is significantly upweighted
in accordance with its priority as a “predominant factor” in human rights risk. 37 In
the RIN formula used in the table, the Severity Potential number is determined by
the addition of the Gravity Potential of the harm and the Number of Affected
Rights-Holders 38:

𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 = 𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑖𝑖 + 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁 𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜𝑜 𝑅𝑅𝑅𝑅𝑅𝑅ℎ𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑖𝑖
Here, Gravity Potential is defined as the maximal or “worst-case scenario” degreeof the gravity or seriousness of a potential harm’s expected consequence, where
gravity is categorized according to the extent of a potential harm’s damage tohuman dignity or to the integrity of individual, collective, and planetary life. 39 Hereis a table that maps out different levels of impact gravity (these gradients are

S. (2012). Improving risk matrices: the advantages of logarithmically scaled axes. Journalof Risk Research, 15(2), 209-222; Duijm, N. J. (2015). Recommendations on the use anddesign of risk matrices. Safety science, 76, 21-31. Rausand, M., & Haugen, S. (2020). Risk
Assessment: Theory, Methods, and Applications. John Wiley & Sons.
𝑛𝑛
36
The quantification of risk is often represented by the general expression,�𝑖𝑖=1 𝑙𝑙(𝐶𝐶𝑖𝑖 )𝑝𝑝𝑖𝑖 ,where consequence 𝐶𝐶𝑖𝑖 (quantified in loss function 𝑙𝑙(𝐶𝐶𝑖𝑖 )) is multiplied by the probability ofthe event 𝑝𝑝𝑖𝑖 . However, where loss functions and probabilities are less quantifiable, ordinalapproaches to determining severity and probability levels lead to the use of semiquantitative methods. It should be noted, as well, that human rights risks cannot be
reduced to any sort of quantified cost-benefit analysis, so conventional, quantitative
approach to risk assessment are not applicable. For a discussion of the importance ofcombining quantitative and qualitative approaches in human rights impact assessments,
see Nordic Trust Fund (2013). Study on Human Rights Impact Assessments: A Review of
the Literature, Differences with other Forms of Assessments and Relevance forDevelopment.
37
UN Human Rights Office of the High Commissioner (UNHROHC) (2012) “The Corporate
Responsibility to Protect Human Rights: An Interpretive Guide”. p. 7.
38
These two dimensions, gravity and number of affected rights-holders, line up with what
the UN Human Rights Office of the High Commissioner term “scale” (how grave or serious
the impact is) and “scope” (numbers of people impacted). Scale and scope are seen by
the latter as constitutive elements of severity. N.B, the other crucial variable or“remediability” will be considered in the HUDERIA phase, where potentially affectedstakeholders are included in the assessment of human rights impacts and these impacts
are properly contextually specified. See UN Human Rights Office of the High Commissioner(2020) “Identifying and Assessing Human Rights Risks related to End-Use”.
39
It must be acknowledged that there is no predetermined ordering of human rights thatindicates a prioritization of one over another, and that severity is a relative andcontextually situated concept rather than an absolute one. The use of a notion of “gravity
potential” here is not intended to imply otherwise, but rather to provide a provisional way
to frame the human rights risks of AI through a precaution-centered approach so that
proportionate methods of stakeholder engagement and risk management can be initiated.
For guidance on characterizing severity, see: the UN Human Rights Office of the HighCommissioner (UNHROHC) (2012) “The Corporate Responsibility to Protect Human Rights:
An Interpretive Guide”. pp. 83-84.
64intended to orient you as you think through the harm potential of the system as
it relates to the adverse impact on the right or freedom under consideration):

Gravity Level

Description

Score

Catastrophic
1
Harm

Catastrophic prejudices or impairments in theexercise of fundamental rights and freedoms thatlead to the deprivation of the right to life;
irreversible injury to physical, psychological, or
moral integrity; deprivation of the welfare ofentire groups or communities; catastrophic harm
to democratic society, the rule of law, or to thepreconditions of democratic ways of life and justlegal order; deprivation of individual freedom andof the right to liberty and security; harm to thebiosphere.

4

Critical Harm

Critical prejudices or impairments in the exercise
of fundamental rights and freedoms that lead tothe significant and enduring degradation of
human
dignity,autonomy,
physical,
psychological, or moral integrity, or the integrityof communal life, democratic society, or just legal
order

3

Serious Harm

Serious prejudices or impairments in the exercise
of fundamental rights and freedoms that lead tothe temporary degradation of human dignity,
autonomy, physical, psychological, or moral
integrity, or the integrity of communal life,
democratic society, or just legal order or that
harm to the information and communication
environment

2

Moderate or
Minor Harm

Moderate or minor prejudices or impairments inthe exercise of fundamental rights and freedoms
that do not lead to any significant, enduring, or
temporary degradation of human dignity,
autonomy, physical, psychological, or moral
integrity, or the integrity of communal life,
democratic society, or just legal order1

For the purposes of a preliminary estimation of risk level, Gravity Potential shouldbe treated as a heuristic that is based on the contextual assumption of a maximal
degree of potential injury or damage for any given adverse impact—a heuristicthat prioritizes the protection of rights-holders, communities, and the biospherefrom potential irreversible, irreparable, catastrophic, critical, or serious harm. This
follows the logic of the precautionary principle in working from the assumption
that, in situations of uncertainty, where there are reasonable grounds for concern,65

levels of protection should prioritize human and environmental health and safety
to facilitate sufficient preventative and anticipatory action. 40 It is important tonote, however, that the Gravity Potential heuristic is used only for the purpose ofdetermining preliminary proportionality recommendations for risk management
and assurance practices and stakeholder engagement—and is properly re-visitedand re-evaluated in a contextually sensitive way (and with stakeholder input) in
the Stakeholder Engagement Process and in the HUDERIA. 41
The Affected Rights-Holders Number is derived from the answer to question 10
from the first section of the PCRA, which asks you to estimate the number ofaffected rights-holders who would be directly or indirectly affected by the AI
system over time in a scenario where your project optimally scales. In thecalculation of the RIN, the four ranks of Affected Rights-Holders Numbers are
assigned values that increase in intervals of .5, starting at .5:

Number of Affected Rights-Holders

Score

Between 1-10,000 Rights-Holders.5

Between 10,001-100,000 Rights-Holders

1

Between 100,001-1,000,000 Rights-Holders

1.5Over 1,000,000 Rights-Holders

2This calibration reflects an up-weighting of the Gravity Potential and Likelihoodscores in the RIN formula as they relate to the Affected Rights-Holders score, in
keeping with their principal importance in the determination of the risk level.
Whereas Gravity Potential and Likelihood are essential elements of the risk
calculus and hence double-weighted, the Number of Affected Rights-Holders is
40
Following positions taken in the Rio Declaration (1992), the Maastricht Treaty (1992),
the Wingspread Statement (1998), and Commission Communication (2000), but stressing
the importance of “moderate precaution” in accordance with the Commission
Communication (2000). For discussions, see: Garnett, K., & Parsons, D. J. (2017). Multi‐case review of the application of the precautionary principle in European Union law andcase law. Risk Analysis, 37(3), 502-516; Lofstedt, R. E. (2003). The precautionary
principle: Risk, regulation and politics. Process Safety and Environmental Protection,81(1), 36-43; Zander, J. (2010). The application of the precautionary principle in practice:comparative dimensions. Cambridge University Press; Sachs, N. M. (2011). Rescuing thestrong precautionary principle from its critics. U. Ill. L. Rev., 1285.
41
Another significant reason for using the precautionary principle to inform this heuristic
is that it addresses, in part, problems surrounding the decision-making and “knowledge
aspects” of risk characterization under conditions of uncertainty as emphasized in (Aven
2017) and (Krisper 2021). See: Aven, T. (2017). Improving risk characterisations inpractical situations by highlighting knowledge aspects, with applications to risk matrices.
Reliability Engineering & System Safety, 167, 42-48; Krisper, M. (2021). Problems with
Risk Matrices Using Ordinal Scales. arXiv preprint arXiv:2103.05440.

66

added to the Severity Potential element of the calculation to provide a crucial but
supplementary weight that incorporates information about the scope of thepotential harm. The addition of the Affected Rights-Holders score to the Gravity
Potential score, however, also means an overall upweighting of Severity Potential
variable in keeping with its priority as a “predominant factor” in human rights risk.
The combination of Severity Potential (Gravity Potential + Number of RightsHolders Affected) and Likelihood yields RINs that range from 2.5 to 10. Risk levels
are established as intervals within this range 42:

Low Risk

Moderate Risk

𝑅𝑅𝑅𝑅𝑅𝑅 ≤ 5

5.5 ≤ 𝑅𝑅𝑅𝑅𝑅𝑅 ≤ 6High Risk
6.5 ≤ 𝑅𝑅𝑅𝑅𝑅𝑅 ≤ 7.5

Very High Risk
𝑅𝑅𝑅𝑅𝑅𝑅 ≥ 8

The distribution of RINs across these risk levels is reflected in the matrix below. 43
At the extremes, a minor or moderate harm from an AI system affecting between
1-10,000 rights-holders that is deemed unlikely yields a “low risk” output, whereasa catastrophic harm from an AI system affecting over 1,000,000 rights holders
that is deemed very likely yields a “very high risk” output. Risk levels for borderline
cases have been calibrated to accord with a precautionary approach that prioritizes
sufficient levels of protection in risk reduction, mitigation, and management where
substantial harm may occur. So, for instance, a serious harm from an AI systemaffecting 1 – 10,000 rights holders that is deemed likely yields a “high risk” output,
and a catastrophic harm from the same system that is deemed possible also
generates a “high risk” output.

To explore the way this formula for risk level determination works, see the “Harms Table
and RIN Distribution” Excel file that accompanies this proposal.
43
The semi-quantitative methods employed here have taken into account concerns aroundconsistency, resolution, input-output ambiguity, and aggregation long recognized in thecritical study of risk matrices by (Cox 2008), (Smith et al 2009), (Levine 2012), (Duijm
2015), (Baybutt 2017), and (Krisper 2021). The distribution of RINs has been stresstested and calibrated in accordance with the precautionary principle, but the provisional
character and intermediate place of the results in a wider, dialogical and right-holder
involving process must be stressed as a mitigating factor to possible coarseness in outputs.
See: Anthony (Tony) Cox Jr, L. (2008). What's wrong with risk matrices?. Risk Analysis:
An International Journal, 28(2), 497-512; Smith, E.D., Siefert, W.T., Drain, D., 2009. Risk
matrix input data biases. Syst. Eng. 12 (4), 344–360. Levine, E. S. (2012). Improving risk
matrices: the advantages of logarithmically scaled axes. Journal of Risk Research, 15(2),209-222; Duijm, N. J. (2015). Recommendations on the use and design of risk matrices.
Safety science, 76, 21-31; Baybutt, P. (2018). Guidelines for designing risk matrices.
Process safety progress, 37(1), 49-55. Krisper, M. (2021). Problems with Risk Matrices
Using Ordinal Scales. arXiv preprint arXiv:2103.05440.
4267

In addition to providing a table which compiles the risks calculations for all theadverse impacts that have been identified in the PCRA (shown above), the PCRA
Output Report (as demonstrated in the PCRA Output Example) will direct you totake specific risk management actions for each of these potential impacts:

For any 𝑹𝑹𝑹𝑹𝑹𝑹 ≥ 𝟖𝟖 (veryhigh)

Further examination should be undertaken, throughexpert and stakeholder consultation, as to whether
sufficient risk reduction is possible to make very highrisks of harm tolerable or whether this is not feasible,
and the risks are unacceptable.

For any 𝟔𝟔. 𝟓𝟓 ≤ 𝑹𝑹𝑹𝑹𝑹𝑹 ≤𝟕𝟕. 𝟓𝟓 (high)Further examination should be undertaken, throughexpert and stakeholder consultation, as to whether
the risks of harm indicated to be high are tolerable
and can be appropriately reduced. Where likelihood= 1 (unlikely) and RPN indicates high risk,
confirmation of low risk probability should also be
undertaken through expert and stakeholder
consultation.

For any 𝟓𝟓. 𝟓𝟓 ≤ 𝑹𝑹𝑹𝑹𝑹𝑹 ≤ 𝟔𝟔
(moderate)

Further examination should be undertaken, throughexpert and stakeholder consultation, as to whether
the risks of harm indicated to be moderate arebroadly accepted as such, are tolerable, and can beappropriately reduced. Where likelihood = 1(unlikely) and RPN indicates moderate risk,
confirmation of low risk probability should also be
undertaken through expert and stakeholder
consultation.

For any 𝑹𝑹𝑹𝑹𝑹𝑹 ≤ 𝟓𝟓 (low)Further examination should be undertaken, throughexpert and stakeholder consultation, as to whether
the risks of harm indicated to be low are broadly
accepted as such.

The summary report will also produce more general recommendations for
proportionate risk management practices and stakeholder engagement.
These recommendations are based on a prioritization of a degree of risk
management and human right diligence that is proportionate to the highest levelof risk identified across an AI system’s potential impacts. Additional
recommendations for establishing sufficient public transparency andaccountability—as well as for ensuring adequate human rights diligence—are
included for large-scale projects that could have macro-scale, long-term impacts
on individuals and society. Here is table that includes all of the generalrecommendations:68Where the highest
𝑹𝑹𝑹𝑹𝑹𝑹 ≥ 𝟖𝟖 (very high)Where the highest
𝟔𝟔. 𝟓𝟓 ≤ 𝑹𝑹𝑹𝑹𝑹𝑹 ≤ 𝟕𝟕. 𝟓𝟓 (high)Where the highest
𝟓𝟓. 𝟓𝟓 ≤ 𝑹𝑹𝑹𝑹𝑹𝑹 ≤ 𝟔𝟔
(moderate)

Where the highest
𝑹𝑹𝑹𝑹𝑹𝑹 ≤ 𝟓𝟓 (low)Additional PCRA
triggered
recommendations forlarge-scale projects:

Full diligence in risk management and assurance
practices across all risks is recommended to prioritize
risk reduction and mitigation. This should be
informed by the risk factors identified in section 1 of
the PCRA and by the HUDERIA impact assessment
process; Comprehensive stakeholder engagement
across the project lifecycle (e.g. partnering with or
empowering rights-holders as determined by the
Stakeholder
Engagement
Process)isalso
recommended.
Full diligence in risk management and assurance
practices is recommended to prioritize risk reduction
and mitigation. This should be informed by the risk
factors identified in section 1 of the PCRA and by the
HUDERIA
impactassessment
process;
Comprehensive stakeholder engagement across the
project lifecycle (e.g. partnering with or empoweringrights-holders as determined by the Stakeholder
Engagement Process) is also recommended.
Full diligence in risk management and assurance
practices is recommended to prioritize risk reduction
and mitigation but more targeted approaches toaddressing specific risks may be acceptable, whereexperts and stakeholders have been appropriately
consulted as part of this determination. Risk
management and assurance practices should be
informed by the risk factors identified in section 1 of
the PCRA and by the HUDERIA impact assessment
process; Appropriate stakeholder engagement
across the project lifecycle is also recommended
following the engagement objectives established byyour Stakeholder Engagement Process.
Targeted approaches to addressing specific risks
may be acceptable, where your highest risk index
level is low and experts and stakeholders have been
appropriately consulted to confirm this. Diligence inrisk management and assurance practices across the
project lifecycle is, however, still recommended. Risk
management and assurance practices should be
informed by the risk factors identified in section 1 of
the PCRA and by the HUDERIA impact assessment
process; Appropriate stakeholder engagement
across the project lifecycle is also recommended
following the engagement objectives established byyour Stakeholder Engagement Process.
If the AI system could have generational (10 to 20
years), long-term (20 to 60 years), or crossgenerational (over 60 years) impacts on over
100,000 people and could pose risks to humanrights, democracy, and the rule of law, you should69If answer to PCRA
question #10 = c ord and answer to
PCRA question #11 =c, d, or e and answer
to any PCRA
questions #64-83 =likelihood 2, 3, or 4,
then:prioritise public transparency and accountability byforming an oversight board for your project, which
sufficiently represents the rights-holders andcommunities impacted by the system. You should
also carry out a complete Human Rights Impact
Assessment, following the recommendations of theDanish Institute for Human Rights or similar, togauge the wider effects of your business activities on
the communities and individuals they impact.

70

Preliminary Context Based Risk Analysis Template
Key:
ExampleExamples (hover cursor over text to display)

No trigger message

Moderate risk factor (considerable)

Major risk factor (elevated)

Prohibitive risk factor (triggers precautionary principle)

71

Context

Trigger messages for summary
report

Question

Section 1: Identifying risk factors
In this section, you will be asked a series of questions that are intended to help you identify risk
factors to consider as you embark on the impact assessment and risk management and assurance
elements of your HUDERAF. Draw on your Project Summary Report to answer the questions tothe best of your ability. If you are unsure about how to answer, choose that option and move
forward to the next question. As part of the report generated when you complete this PCRA, youwill receive an enumeration of the risk factors that have been identified.
Use Context
Sector or domain for which the system
is being built

1) Will the AI system serve an
essential or primary function in ahigh impact or safety critical sector(e.g., transport, social care,healthcare, other divisions of the
public sector)?

YES:
Major circumstantial risk factor
•

YES
NO

Where AI systems serve primary or
critical functions in high impact or safety
critical sectors, this presents a major
circumstantial risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:

UNSURE•

Example72

Make sure to focus upon considerations
surrounding the prevention of harm andthe respect of the right to life and tophysical,
psychological,
and
moral
integrity

Goals, properties, and areas to focus on inyour HUDERAC:••

Safety(accuracyandsystemperformance, security, reliability, androbustness)
Sustainability, (reflection on context andimpacts, change monitoring)

NO: No message
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system will serve a primary or
critical function in a high impact or safety criticalsector. When this information is ascertained, you
should return to your PCRA and revise it
accordingly.
2) Does the sector or domain in which
the AI system will operate include
vulnerable groups (or groups withprotected characteristics) who may
be significantly or
disproportionately impacted by thedesign and use of the system?
YES
NO
UNSURE

73

YES:
Major circumstantial risk factor•

Where the sector or domain in which the
AI system will operate includes
vulnerable groups (or groups withprotected characteristics) who may besignificantly or disproportionately
impacted by the design and use of thesystem, this presents a major
circumstantial risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Example

Actions to take for your HUDERIA:•

Make sure to focus upon considerations
surrounding the potential impacts of yourAI
systemonfairness,
nondiscrimination, equality, diversity, andinclusivenessGoals, properties, and areas to focus on inyour HUDERAC:•

Fairness (non-discrimination, equality,
bias mitigation, diversity, andinclusiveness)

NO: No message
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system will operate in a sector or
domain which includes vulnerable groups (orgroups with protected characteristics), who may
be significantly or disproportionately impacted by
the design and use of the system. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
Existinglaw
andregulatoryenvironment of the sector or domain

3) Have you assessed existing law
and regulation in the sector or
domain in which the AI system will
operate and determined that it has

74

YES: No message

NO: STOP.

a legal basis and can be developed
and deployed lawfully?
YES

Prohibitive circumstantial risk factor
If you have not been able to establish a legal
basis for the AI system, you should not proceed.
Where appropriate, you should consult withexperts to establish the lawfulness of yourprocessing objective and proposed project before
taking any next steps. If you would like tocontinue with the PCRA anyway, select yes. If
not, please reconsider your project and return
when you have ascertained that your AI system
will be lawful.

NO
UNSUREUNSURE: STOP. Before taking any further steps
in the proposed project, you should determine,through expert and stakeholder input whereappropriate, whether your project has a legal
basis. When this information is ascertained, you
should return to your PCRA and revise it
accordingly.

4) Is the sector or domain in which
the AI system will operate
historically highly regulated?
YES

YES:
Moderate circumstantial risk factor•

NO
UNSURE

Example75

Where the sector or domain in which the
AI system will operate is historicallyhighly regulated, this presents amoderate circumstantial risk factor
for adverse impacts on the human rights
and fundamental freedom of persons.

Actions to take for your HUDERIA:•

Make sure to consider the regulatory
context of your project and to reflect on
the expectations of compliant andreasonable practices that arise in that
context.

Goals, properties, and areas to focus on inyour HUDERAC:•

Accountability and process transparency,(traceability, accessibility, auditability,
and responsible governance)

NO: No message
UNSURE: Before taking any further steps in the
proposed project, you should examine, seeking
expert and stakeholder input where appropriate,
whether any relevant existing regulation in yoursector or domain applies to the prospective AI
system. When this information is ascertained,
you should return to your PCRA and revise it
accordingly.

5) Does statute or regulation in the
sector or domain in which the AI
system will operate require anyother types of impact assessment
for the specific use-case of the AI
systems you are planning to

76

YES:
Moderate circumstantial risk factor•

Where statute or regulation in the sector
or domain in which the AI system will

develop (e.g., data protection
impact assessment, equality
impact assessment, human rights
impact assessment, etc.)?

operate requires other types of impact
assessment for the specific use-case of
the AI systems, this presents amoderate circumstantial risk factor
for adverse impacts on the human rights
and fundamental freedom of persons.

YES

Actions to take for your HUDERIA:NOUNSURE•

ExampleMake sure to integrate the completion ofother compulsory impact assessments as
supplementary material and evidentiary
support for your HUDERIA.

Goals, properties, and areas to focus on inyour HUDERAC:•

Accountability and process transparency(traceability, accessibility, auditability,
and responsible governance)

NO: No message

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether any other types of impact assessment
for the specific use-case of the AI systems you
are planning to develop are required by law or
regulation. When this information is ascertained,
you should return to your PCRA and revise it
accordingly.
YES:

77

Impact-level of the system

6) Will the AI system perform a safety
critical or high impact function
independent of the sector in which
it operates?

Major circumstantial risk factor

•

YES
NO
UNSURE

Where AI systems perform a safety
critical or high impact function
independent of the sector in which they
operate, this presents a major
circumstantial risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:

Example•Make sure to focus upon considerations
surrounding the prevention of harm andthe respect of the right to life and tophysical,
psychological,
and
moral
integrity

Goals, properties, and areas to focus on inyour HUDERAC:•

Safety(accuracyandsystemperformance, security, reliability, androbustness) and sustainability

NO: No message

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
performs a safety critical or high impact function.

78

When this information is ascertained, you should
return to your PCRA and revise it accordingly.
Prohibited AI systems and uses

7) Is the AI system on the list ofprohibited systems - EU AI Reg?

YES: STOP.
Prohibitive circumstantial risk factor
If the AI system you are planning to build is on
the list of prohibited systems, it poses too high arisk to impacted rights-holders and communities
for you to proceed. Prohibited systems that may
catastrophically, critically, or seriously harm thephysical, psychological, or moral integrity or thehuman rights and fundamental freedoms ofaffected persons should not be developed or
implemented in that form. If you would like tocontinue with the PCRA anyway at this time,
select yes. If not, please reconsider your projectand return when you have ascertained that yourAI system will be lawful. If you would like tocontinue with the PCRA anyway, select yes. If
not, please reconsider your project and return
when you have ascertained that your AI system
will be not fall under the list of prohibited
systems.

YES
NO
UNSURE

Example

NO: No message

UNSURE: STOP. Before taking any further steps
in the proposed project, you should determine,through expert and stakeholder input whereappropriate, whether the AI system you areplanning to build is on the list of prohibited
systems.

798) Could the AI system be repurposed
or used in ways that fall under thelist of prohibited systems?
YES

YES:
Major circumstantial risk factor•

NO
UNSURE

ExampleWhere AI systems could be repurposed
or used in ways that fall under the list ofprohibited systems, this presents amajor circumstantial risk factor foradverse impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:
No trigger message
Goal, properties, and areas to focus on inyour HUDERAC:•

You should take special precautions to put
in place protections, processes, andmechanisms that ensure the limitation of
the system’s use to the purpose for which
it was created. These protections,processes, and mechanisms should bedemonstrated and evidenced in your
HUDERAC.

NO: No message

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,80whether the AI system could be used or
repurposed in ways that fall under the list ofprohibited systems are considered to posesignificant risks. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
Scope of deployment (breadth andtemporality)9) In a scenario where your project
optimally scales, will the AI system
directly and/or indirectly affectrights-holders and groups within:a. Low risk factor
No message
b. Moderate circumstantial risk factor

a. Organisations•

b. Local populations
c. National populations
d. Global population?

Where AI systems directly or indirectly
affect rights-holders and groups within
local populations, this presents amoderate circumstantial risk factor
for adverse impacts on the human rights
and fundamental freedom of persons.

Actions to take for your HUDERIA:

Example a•

Example b
Example c
Example dIn assessing the potential impacts of thesystem, you should pay special attention
to the potential effects of the AI system
on local populations and on thecommunities and groups that comprise
them.

Goals, properties, and areas to focus on inyour HUDERAC:
No trigger message

81

c. Major circumstantial risk factor•

Where AI systems directly or indirectly
affect rights-holders and groups within
national populations, this presents amajor circumstantial risk factor foradverse impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:•In assessing the potential impacts of thesystem, make sure to pay special
attention to the potential effects of the AI
system on local and national populations
and on the communities and groups thatcomprise them.

Goal, properties, and areas to focus on inyour HUDERAC:
No trigger messaged. Major circumstantial risk factor•

Where AI systems directly or indirectly
affect rights-holders and groups within
national populations, this presents amajor circumstantial risk factor foradverse impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:82

•

In assessing the potential impacts of thesystem, make sure to pay special
attention to the potential effects of the AI
system on local, national, and global
populations and on the communities andgroups that comprise them.

Goal, properties, and areas to focus on inyour HUDERAC:
No trigger message10) In a scenario where your project
optimally scales, will the AI system
directly and/or indirectly affect:a. No message
b. Moderate circumstantial risk factor•

a. Between 1 – 10,000 rightsholders
b. Between 10,001 – 100,000 rights
holders

Where AI systems directly or indirectly
affect between 10,001-100,000 rightsholders, this presents a moderate
circumstantial risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:
c. Between 100,001 – 1,000,000
rights-holders
d. Over 1,000,000 rights-holders?

No trigger message
Goal, properties, and areas to focus on inyour HUDERAC:
No trigger message

Example a83

c. Major circumstantial risk factor

Example b•

Example c
Example dWhere AI systems directly or indirectly
affectbetween
100,001-1,000,000
rights-holders, this presents a major
circumstantial risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:•In assessing the potential impacts of thesystem, make sure to pay special
attention to the at scale or mass-level
effects of the use of the AI system onsocial and political processes andinstitutions—in particular, as these relate
to human rights, fundamental freedoms,
democracy, and the rule of law.

Goal, properties, and areas to focus on inyour HUDERAC:
No trigger message

84

d. Major circumstantial risk factor•

Where AI systems directly or indirectly
affect over 1,000,000 rights-holders, thispresents a major circumstantial risk
factor for adverse impacts on the humanrights and fundamental freedom of
persons.

Actions to take for your HUDERIA:•In assessing the potential impacts of thesystem, make sure to pay special
attention to the at scale or mass-level
effects of the use of the AI system onsocial and political processes andinstitutions—in particular, as these relate
to human rights, fundamental freedoms,
democracy, and the rule of law.

Goal, properties, and areas to focus on inyour HUDERAC:
No trigger message11) Considering the potential direct
and indirect impacts of yourproject on individuals,
communities, and theenvironment, which of thefollowing is the widest timescale
within which the AI system could
affect rights-holders and groups:

85

a. No message

b. Moderate circumstantial risk factor•

Where AI systems directly or indirectly
affect rights-holders medium term (1 to10 years), this presents a moderate
circumstantial risk factor for adverse

a. Short term (less than a year)

impacts on the human rights
fundamental freedom of persons.

b. Medium term (1 to 10 years)c. Generationally (10 to 20
years)
d. Long-term (20 to 60 years)

Actions to take for your HUDERIA:•

e. Over lifetimes and acrossfuture generations (over 60 years)
f. UNSURE

Example a
Example b

andIn assessing the potential impacts of thesystem, make sure to pay special
attention to the medium term (1-10
years) effects of the use of the AI systemon the individual identities of rights
holders as well as on social and politicalprocesses and institutions—in particular,
as these relate to human rights,
fundamental freedoms, democracy, andthe rule of law.

Goal, properties, and areas to focus on inyour HUDERAC:

Example c

No trigger message

Example d
Example e:1

c. Major circumstantial risk factor

Example e:2•

Where AI systems directly or indirectly
affect rights-holders generationally (10to 20 years), this presents a major
circumstantial risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:

86

•

In assessing the potential impacts of thesystem make sure to pay special attention
to the generational (10-20 years) effectsof the use of the AI system on theindividual identities of rights holders, onsocioeconomic relationships, and onsocial and political processes andinstitutions—in particular, as these relate
to human rights, fundamental freedoms,
democracy, and the rule of law.

Goal, properties, and areas to focus on inyour HUDERAC:
No trigger messaged. Major circumstantial risk factor•

Where AI systems directly or indirectly
affect rights-holders long-term (20 to 60
years), this presents a major
circumstantial risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:•

87

In assessing the potential impacts of thesystem, make sure to pay special
attention to the long-term (20 to 60
years) and generational (10-20 years)
effects of the use of the AI system on theindividual identities of rights holders, onsocioeconomic relationships, and onsocial and political processes andinstitutions—in particular, as these relate
to human rights, fundamental freedoms,
democracy, and the rule of law.
Goal, properties, and areas to focus on inyour HUDERAC:
No trigger messagee. Major circumstantial risk factor•

Where AI systems directly or indirectly
affect rights-holders over lifetimes andacross future generations (over 60
years), this presents a major
circumstantial risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:•

88

In assessing the potential impacts of thesystem, make sure to pay special
attention to the long-term (20 to 60
years), generational (10-20 years), andcross-generational (over 60 years) effectsof the use of the AI system on theindividual identities of rights holders, onsocioeconomic relationships, and onsocial and political processes andinstitutions—in particular, as these relate
to human rights, fundamental freedoms,
democracy, and the rule of law.
Goal, properties, and areas to focus on inyour HUDERAC:
No trigger message
Proportionality trigger: If the AI system could
have cross-generational (over 60 years) impacts
on over 100,000 people, you should prioritise
public transparency and accountability by forming
an oversight board for your project, which
sufficiently represents the rights-holders andcommunities impacted by the system. You should
also carry out a complete Human Rights Impact
Assessment to gauge the wider effects of yourbusiness activities on the communities andindividuals they impact.
UNSURE: Before taking any further steps in the
proposed project, you should determine, seeking
expert and stakeholder input where appropriate,
the timescale of the prospective AI system’s
impactsonaffectedrights-holders
andcommunities.
When
this
informationisascertained, you should return to your PCRA andrevise it accordingly.
Technological maturity

12) Will the AI system's design bebased on well-understoodtechniques that have previously
been in operation and externally89

YES: No message
NO:validated for a similar purpose andin the same sector?

Moderate modifiable risk factor
•

YES
NO
UNSURE

ExampleWhere AI systems are not designed
based on well-understood techniquesthat have previously been in operationand externally validated for a similar
purpose and in the same sector, thispresents a moderate modifiable risk
factor for adverse impacts on thehuman rights and fundamental freedomof persons.

Actions to take for your HUDERIA:
No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:•

Safety(accuracyandsystemperformance, security, reliability, androbustness) and sustainability (changemonitoring). In particular, you should
demonstrate in your assurance case that
diligent processes of testing, verifying,and externally validating the performanceof the system has occurred. You should
also establish system monitoring andperformance evaluation protocols that areproportionate
tothesystem’s
technological maturity.

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the techniques on which the AI system
will be based have been successfully used in other90externally validated systems that have had
similar purposes and were in the same sector.
When this information is ascertained, you should
return to your PCRA and revise it accordingly.
Existing
system(humanortechnological) that the application is
replacing13) If the AI system is replacing ahuman, technical, or hybrid systemthat serves the same or similarfunction, is a reason for
replacement that the existing
system is considered flawed or
harmful?

YES:
Moderate circumstantial risk factor
•

YES
NO
UNSURE

Where AI systems are replacing ahuman, technical, or hybrid system thatserves the same or similar function,
because the existing system isconsidered flawed or harmful, thispresents a moderate circumstantial
risk factor for adverse impacts on thehuman rights and fundamental freedomof persons.

Actions to take for your HUDERIA:NOT APPLICABLE•

ExampleMake sure to pay special attention toassessing how your AI system could
potentially repeat the harmful outcomesgenerated by the replaced system andhow to minimise any potential replicatedharms.

Goals, properties, and areas to focus on inyour HUDERAC:•

91

Safety(accuracyandsystemperformance, security, reliability, androbustness). Your assurance case should
demonstrate how you have ensured thatthe flaws or harms generated by theprevious system are redressed andrectified across the design, development,
and deployment lifecycle
NO: No message

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the prospective AI system is replacing ahuman, technical, or hybrid system that serves
the same or similar function, is a reason for
replacement that the existing system isconsidered flawed or harmful. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message14) If the human, technical, or hybrid
system being replaced isconsidered critical infrastructure orserves a safety-critical or highimpact function, has assessment
and planning been done and madepublic to transparently ensure thatits updating/replacement does notcause unacceptable outage or
harm?
YES

92

YES: No message
NO: Major modifiable risk factor•

Where assessment and planning is notdone and made public to transparently
ensure that the updating/replacement ofa human, technical, or hybrid systemthat is considered critical infrastructureor serves a safety-critical or high impact
function does not cause unacceptable
outage or harm, this presents a major
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

NO

Actions to take for your HUDERIA:

UNSURENOT APPLICABLE

ExampleNo trigger message
Goals, properties, and areas to focus on inyour HUDERAC:•

You
shouldincorporate
publiclytransparent assessment and planning into
your assurance case which demonstrates
that the updating/replacement of theprevious
system
will
not
cause
unacceptable outage or harm.

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
will replace a human, technical, or hybrid systemthat is considered critical infrastructure or servesa safety-critical or high impact function andwhether the new system may cause unacceptable
outage or harm. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message15) If the system is automating or
replacing human labour, willassessment and planning be done
and made public to transparently
ensure that any corresponding job

93

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT ARE

loss, labour displacement, orredeployment is publicly
acceptable, and any possible
corollary harms are managed andmitigated?
YES, WE HAVE PLANS IN PLACETO DO THISUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS

Where assessment and planning is notdone and made public to transparently
ensure that job loss, labour
displacement, or redeployment, which is
the result of an AI system automating or
replacing human labour, is publicly
acceptable, and any possible corollaryharms are managed and mitigated, thispresents a moderate modifiable risk
factor for adverse impacts on the humanrights and fundamental freedom of
persons.

Actions to take for your HUDERIA:

UNSURENOT APPLICABLE•

Example 1
Example 2

Make sure to pay special attention toassessing how your AI system could
potentially harm human dignity throughlabour displacement and infringe upon theright to just working conditions, the rightto safe and healthy working conditions,and the right to organize as set out in theEuropean Social Charter

Goals, properties, and areas to focus on inyour HUDERAC:•

94

Sustainability (reflection on context andimpacts and stakeholder engagement andinvolvement)
andfairness(nondiscrimination and equality). Make sure toincorporate
publiclytransparent
assessment and planning into yourassurance case to ensure that any
corresponding
job
loss,
labourdisplacement, or redeployment is publicly
acceptable and that any possible corollaryharms are managed and mitigated.
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the system you are planning to build willautomate or replace human labour in a way thatcreates publicly unacceptable job loss, labour
displacement, or redeployment. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message

Bias and discrimination in sector or
domain context16) Do the sector(s) or domain(s) inwhich the AI system will operate,
and from which the data used totrain it are drawn, containhistorical legacies and patterns of
discrimination, inequality, bias,
racism, or unfair treatment ofminority, marginalized, orvulnerable groups that could be
replicated or augmented in thefunctioning of the system or in its
outputs and short- and long-term
impacts?

95

YES:
Major circumstantial risk factor•

Where the sector(s) or domain(s) in which
the AI system will operate, and from which
the data used to train it are drawn, containhistorical legacies and patterns of
discrimination, inequality, bias, racism, or
unfairtreatmentofminority,
marginalized, or vulnerable groups that
could be replicated or augmented in thefunctioning of the system or in its outputs
and short- and long-term impacts, thispresents a major circumstantial risk

factor for adverse impacts on the humanrights and fundamental freedom of
persons.

YES
NO

Actions to take for your HUDERIA:

UNSURENOT APPLICABLE•

Example 1

Make sure to focus upon considerations
surrounding
thefairness,
nondiscrimination, equality, diversity, andinclusivenessGoals, properties, and areas to focus on inyour HUDERAC:

Example 2•

Fairness (non-discrimination, equality,
bias mitigation, diversity, andinclusiveness). Continued emphasisshould be placed on data
representativeness and bias mitigation
across design, development, anddeployment processes.

NO: No message

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the sector(s) or domain(s) in which the
AI system will operate, and from which the data
used to train it are drawn, contain historicallegaciesandpatternsofdiscrimination,
inequality, bias, racism, or unfair treatment ofminority, marginalized, or vulnerable groups that
could be replicated or augmented in thefunctioning of the system or in its outputs and96

short- and long-term impacts. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message

Environmental context17) If the design, development, anddeployment of the AI system will
have potentially significant impacts
on the environment, will sufficientand transparently reported
processes be implemented
throughout the project’s lifecycle toensure that the system, in both its
production and use, complies withapplicable environmental
protection standards and supports
the sustainability of the planet?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURE

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

Where the design, development, anddeployment of the AI system will havepotentially significant impacts on theenvironment
andsufficientandtransparently reported processes are not
implemented throughout the project’s
lifecycle to ensure that the system, in bothits production and use, complies withapplicable
environmental
protection
standards and supports the sustainabilityof the planet, this presents a major
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:

97

NOT APPLICABLE•

ExampleMake sure to focus upon considerations
surrounding the impacts of the AI systemon the biosphere and planetary health.
You should also explore in your HUDERIA,
which communities may be adversely
impacted by environmental harms—
paying close attention to effects onvulnerable and marginalized groups.

Goals, properties, and areas to focus on inyour HUDERAC:•

Sustainability (reflection on context andimpacts and stakeholder engagement
and involvement)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate sufficient and transparently
reported processes to ensure that the system, inboth its production and use, does not adversely
impact the environment or harm the biosphereand that the system complies with applicable
environmental protection standards. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message

Cybersecurity context18) Could the AI system present
motivations or opportunities formalicious parties to hack or corrupt
it to achieve substantial financial

98

YES:
Moderate modifiable risk factor

gains, political goals, or otherperceived benefits?

•

YES
NO
UNSURENOT APPLICABLE

ExampleWhere AI systems provide motivations oropportunities for malicious parties tohack or corrupt them to achieve
substantial financial gains, political goals,
or other perceived benefits, this presents
a moderate modifiable risk factor foradverse impacts on the human rights
and fundamental freedom of persons.

Actions to take for your HUDERIA:
No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:•

Safety (security and robustness)

NO: No message
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system present motivations oropportunities for malicious parties to hack or
corrupt it to achieve substantial financial gains,political goals, or other perceived benefits. When
this information is ascertained, you should return
to your PCRA and revise it accordingly.
NOT APPLICABLE: No message19) Will sufficient and transparently
reported processes be

99

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageimplemented throughout the
project’s lifecycle to ensure thatmeasures put in place to safeguard
the system's safety, security, androbustness are appropriately
proportional to potential risks ofhacking, adversarial attack, datapoisoning, model inversion, orother cybersecurity threats?

WE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISWhere
sufficientandtransparently
reported processes are not implemented
throughout project lifecycles to ensurethat measures put in place to safeguard AI
systems’ safety, security, and robustness
are appropriately proportional to potential
risks of hacking, adversarial attack, datapoisoning, model inversion, or othercybersecurity threats, this presents amajor modifiable risk factor foradverse impacts on the human rights andfundamental freedom of persons.

UNSURE

Actions to take for your HUDERIA:NOT APPLICABLE

No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:

Example•Safety (security and robustness)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate sufficient and transparently
reported processes throughout the project’s
lifecycle to ensure that measures put in place to100

safeguard the system's safety, security, androbustness are appropriately proportional topotential risks of hacking, adversarial attack, datapoisoning,model
inversion,orothercybersecurity threats. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message20) Will sufficient and transparently
reported processes be
implemented throughout the
project’s lifecycle to stress test theAI system for cybersecurity
vulnerabilities and resilience?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURE

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor

•

Where
sufficientandtransparently
reported processes are not implemented
throughout project lifecycles to stress testAI
systemsforcybersecurity
vulnerabilities and resilience, this presents
a moderate modifiable risk factor foradverse impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:NOT APPLICABLE

101

No trigger message

Goals, properties, and areas to focus on inyour HUDERAC:

Example•Safety (security and robustness)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate sufficient and transparently
reported processes throughout the project’s
lifecycle to stress test the AI system forcybersecurity vulnerabilities and resilience. When
this information is ascertained, you should return
to your PCRA and revise it accordingly.
NOT APPLICABLE: No message

Data Lifecycle Context
Data quality, integrity, and provenance21) Will sufficient and transparently
reported processes be
implemented throughout the
project’s lifecycle to ensure that alldata used in producing the system
are sufficiently balanced andrepresentative of the individual
rights-holders and groups it isaffecting?

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor

YES, WE HAVE PLANS IN PLACETO DO THIS

102•

Where sufficient and transparently
reported processes are not implemented
throughout the project’s lifecycle toensure that all data used in producing the

WE HAD NOT CONSIDERED IT,
BUT MAY DO THIS

system are sufficiently balanced andrepresentative of the individual rightsholders and groups it is affecting, thispresents a moderate modifiable risk
factor for adverse impacts on the humanrights and fundamental freedom of
persons.

WE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURE

Actions to take for your HUDERIA:
No trigger message

ExampleGoals, properties, and areas to focus on inyour HUDERAC:•

Data quality (dataset balance andrepresentativeness) and fairness (nondiscrimination, and bias-mitigation)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate into the design anddevelopment of your AI system the inclusion ofdatasets, which are sufficiently balanced andrepresentative of the individual rights-holders
and groups they are affecting. You should also
determine how to demonstrate this in yourassurance case. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
22) Will sufficient and transparently
reported processes be
implemented throughout the103

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageproject’s lifecycle to ensure that alldata used in producing the system
are accurate, reliable, relevant,appropriate, up-to-date, and of
adequate quantity and quality forthe use case, domain, function,and purpose of the system?

WE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURE

Where sufficient and transparently
reported processes are not implemented
throughout the project’s lifecycle toensure that all data used in producing thesystem are accurate, reliable, relevant,appropriate, up-to-date, and of adequate
quantity and quality for the use case,domain, function, and purpose of thesystem, this presents a moderate
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:
No trigger message

ExampleGoals, properties, and areas to focus on inyour HUDERAC:•

104

Data quality (measurement accuracy andsource integrity, data reliability, data
relevance and appropriateness, data
timeliness and recency, adequacy of data
quantity and quality for the use case,domain, function, and purpose of thesystem,andresponsible
datamanagement)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate into the design anddevelopment of your AI system the inclusion ofdata, which are accurate, reliable, relevant, upto-date, appropriate, and of adequate quantityand quality for the use case, domain, function,and purpose of the system. You should also
determine how to demonstrate this in yourassurance case. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
23) Will sufficient and transparently
reported processes be
implemented throughout the
project’s lifecycle to ensure that alldata used in producing the system
are attributable, consistent,
complete, and contemporaneouswith collection?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS105

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

Where sufficient and transparently
reported processes are not implemented
throughout the project’s lifecycle toensure that all data used in producing thesystem are attributable, consistent,
complete, and contemporaneous withcollection, this presents a moderate
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:UNSURENo trigger message

ExampleGoals, properties, and areas to focus on inyour HUDERAC:•

Data integrity (data attributability, dataconsistency, data completeness, data
contemporaneous, and responsible data
management)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate into the design anddevelopment of your AI system the inclusion ofdata, which are attributable, consistent,
complete, and contemporaneous with collection.
You should also determine how to demonstrate
this in your assurance case. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
24) Will sufficient and transparently
reported processes be
implemented throughout the
project’s lifecycle to ensure theproper recording, traceability, andauditability of the provenance andlineage of all data used in
producing the system, and anyother data involved in the dynamic
learning, tuning, or re-training of
the system across its lifecycle?

106

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor

•YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURE

Where sufficient and transparently
reported processes are not implemented
throughout the project’s lifecycle toensure the proper recording, traceability,
and auditability of the provenance andlineage of all data used in producing thesystem, and any other data involved in
the dynamic learning, tuning, or retraining of the system across its lifecycle,, this presents a moderate modifiablerisk factor for adverse impacts on thehuman rights and fundamental freedomof persons.

Actions to take for your HUDERIA:

ExampleNo trigger message
Goals, properties, and areas to focus on inyour HUDERAC:•

Accountability and process transparency,(accessibility, clear data provenance andlineage) and data integrity (responsible
data management)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate into the design anddevelopment of your AI system the inclusion
mechanisms and processes to ensure the proper
recording, traceability, and auditability of theprovenance and lineage of all data used to train,
test, and validate the system, and any other data107

involved in the dynamic learning, tuning, or retraining of the system across its lifecycle. You
should also determine how to demonstrate this inyour assurance case. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
Means and methods of data collection25) Where there is human involvementin the data lifecycle, willtransparent and publicly accessible
measures be implemented toensure mitigation of potential
measurement errors or biases incollection, measurement, andrecording processes?

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor

YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS•

Where transparent and publicly
accessible measures are not implemented
to ensure mitigation of potential
measurement errors or biases incollection, measurement, and recording
processes, where there is human
involvement in the data lifecycle, , thispresents a moderate modifiable risk
factor for adverse impacts on the humanrights and fundamental freedom of
persons.

UNSURENOT APPLICABLE

Actions to take for your HUDERIA:
No trigger message

Example108

Goals, properties, and areas to focus on inyour HUDERAC:•

Data quality, measurement accuracy andsource integrity, and responsible data
management

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate into the design anddevelopment of your AI system transparent andpublicly accessible measures to ensure mitigationof the potential for measurement errors or biases
in collection, measurement, and recording
processes where there is human involvement indata collection. You should also determine how todemonstrate this in your assurance case. When
this information is ascertained, you should return
to your PCRA and revise it accordingly.
Not Applicable: No message26) In the event that collected or
procured datasets have missing or
unusable data, will the methods
used for addressing these
deficiencies be transparent andmade accessible to relevant
stakeholders?

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor

109

YES, WE HAVE PLANS IN PLACETO DO THIS•WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS

Where the methods used for addressing
missing or unusable data in collected or
procured datasets is not transparent andmade accessible to relevant stakeholders,
this presents a moderate modifiablerisk factor for adverse impacts on thehuman rights and fundamental freedomof persons.

Actions to take for your HUDERIA:
No trigger message

UNSURENOT APPLICABLE

ExampleGoals, properties, and areas to focus on inyour HUDERAC:•

Data quality (data appropriateness), data
integrity(data
completeness,andresponsible data management), andaccountability and process transparency(accessibility, and clear data provenanceand lineage)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
the methods and measure you will put into place
to address issues arising from collected or
procured datasets, which have missing or
unusable data. You should also determine how todemonstrate this in your assurance case. When
this information is ascertained, you should return
to your PCRA and revise it accordingly.

110

NOT APPLICABLE: No message27) Where personal data are used in
the production of the AI system,
will information be made available
to impacted rights-holders andother relevant stakeholders about
the consent or legitimate basis to
use that data for the purpose of
the system?
YES, WE HAVE PLANS IN PLACETO DO THIS

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURENOT APPLICABLE

ExampleWhere information is not made available
to impacted rights-holders and otherrelevant stakeholders about the consent
or legitimate basis to use personal data
for the purpose of the system, thispresents a moderate modifiable risk
factor for adverse impacts on the humanrights and fundamental freedom of
persons.

Actions to take for your HUDERIA:
No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:•

111

Data protection and privacy (consent andaccountability) and process transparency

(accessibility, and clear data provenanceand lineage)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to make information available to impacted
rights-holders and other relevant stakeholders
about the consent or legitimate basis to use
personal data for the purpose of the system.
When this information is ascertained, you should
return to your PCRA and revise it accordingly.
NOT APPLICABLE: No message28) If consent or the legitimate basis
to use personal data is implied, willrights-holders and other relevant
stakeholders be consulted to
identify acceptability of the data
use or concerns that need to beaddressed?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS

112

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

Where rights-holders and other relevant
stakeholders are not consulted to identify
acceptability of the data use or concernsthat need to be addressed when consent
to use personal data is implied, thispresents a moderate modifiable risk
factor for adverse impacts on the humanrights and fundamental freedom of
persons.

NO, WE ARE NOT PLANNING TO
DO THISUNSURE

Actions to take for your HUDERIA:•NOT APPLICABLE

ExampleMake sure to incorporate considerations
surrounding the acceptability of thepersonal data use into your impact
assessment, integrating affected rightsholders
into
relevant
deliberations
through your Stakeholder Engagement
Process where appropriate.

Goals, properties, and areas to focus on inyour HUDERAC:•

Sustainability (reflection on context andimpacts, stakeholder engagement andinvolvement)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether your project involves the use of personal
data based on implied consent and, if so, how you
can consult impacted rights-holders and otherrelevant stakeholders to identify acceptability ofthe data use or concerns that need to beaddressed. When this information is ascertained,
you should return to your PCRA and revise it
accordingly.
NOT APPLICABLE: No message

Data types29) Will the AI system use dynamic
data, collected and processed in

113

YES:
Major modifiable risk factor

real time (or near real time), forcontinuous learning?

•

YES
NO
UNSURENOT APPLICABLE

ExampleWhere the AI system uses dynamic data,collected and processed in real time (ornear real time), for continuous learning,
this presents a major modifiable risk
factor for adverse impacts on the humanrights and fundamental freedom of
persons.

Actions to take for your HUDERIA:
No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:•

Safety(security,
reliability,
androbustness), data quality, and data
integrity as well as, where appropriate,
non-discrimination and bias mitigation.
You should also transparently report youruse of dynamic data, and the measures
you are taking to manage the risks
surrounding them, to impacted rightsholders and relevant stakeholders.

NO: No message
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether your project involves the use of use
dynamic data, collected and processed in realtime (or near real time), for continuous learning
and, if so, how to manage the risks surrounding
this. You should also determine how to114

demonstrate such risk management measures in
the relevant parts of your assurance case—inparticular, those involving the assurance of goals
and properties of safety, security, reliability,
robustness, data quality, and data integrity as
well as, where appropriate, non-discrimination
and bias mitigation. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message30) Do the domain in which the data
are collected or procured, and thetype of the data collected or
procured, pose risks of rapid or
unexpected distributional shifts ordrifts that could adversely impact
the accuracy and performance ofthe system?
YES
NO
UNSURE

YES:
Moderate circumstantial risk factor•

Where the domain in which the data arecollected or procured, and the type of thedata collected or procured, pose risks of
rapid or unexpected distributional shifts ordrifts that could adversely impact theaccuracy and performance of the system,
this presents a moderate modifiablerisk factor for adverse impacts on thehuman rights and fundamental freedom of
persons.

Actions to take for your HUDERIA:NOT APPLICABLE

ExampleNo trigger message
Goals, properties, and areas to focus on inyour HUDERAC:

115

•

Sustainability (change monitoring), safety(reliability and robustness), data quality,and data integrity as well as, whereappropriate, non-discrimination and bias
mitigation. You should also involve
domain experts to determine the potential
sources of distributional shifts or drifts andbuild processes of dynamic assessment,
re-assessment, external validation, andmonitoring into your project lifecycle.

NO: No message

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the collected or procured data you will
be using are subject to rapid or unexpecteddistributional shifts or drifts and, if so, how tomanage the risks surrounding this. You should
also determine how to demonstrate such risk
management measures in the relevant parts ofyour assurance case—in particular, those
involving the assurance of goals of safety,
security, reliability, robustness, data quality, anddata integrity as well as, where appropriate, nondiscrimination and bias mitigation. In addition tothis, you should also involve domain experts todetermine the potential sources of distributional
shifts or drifts and build processes of dynamic
assessment, re-assessment, external validation,and monitoring into your project lifecycle. When
this information is ascertained, you should return
to your PCRA and revise it accordingly.

116

NOT APPLICABLE: No message31) If the AI system will use
unstructured data or a combinationof structured and unstructured
data, will the project lifecycle
incorporate mechanisms andprocesses to ensure that theinferences generated from thatdata by the system are reasonable,
fair, and do not contain lurking
proxies or correlations that are
discriminatory or inequitable?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

Where the project lifecycle does notincorporate mechanisms and processes
to ensure that the inferences generated
from the use of unstructured data, or a
combination of structured andunstructured data, are reasonable, fair,and do not contain lurking proxies orcorrelations that are discriminatory or
inequitable, this presents a moderate
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:

UNSURENOT APPLICABLE

No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:

117

•

Fairness (non-discrimination, and bias
mitigation). You should also involve
domain experts and social scientists todetermine the potential sources of thelurking
discriminatory
proxiesorcorrelations that may influence theoutputs of the system.

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether your AI system will use unstructured
data or a combination of structured andunstructured data and, if so, how to manage therisks that these data may generate inferencesthat are unreasonable, inequitable, or containlurking discriminatory proxies or correlations.
When this information is ascertained, you should
return to your PCRA and revise it accordingly.
NOT APPLICABLE: No message

Dataset linkage32) Is there a possibility of
deanonymizing or identifying
rights-holders through data linkage
with existing data, publicly
available datasets, or data that
could be easily obtained?
YES
NO
UNSURE

118

YES:
Major circumstantial risk factor•

Where there is a possibility of
deanonymizing or identifying rightsholders through data linkage withexisting data, publicly available datasets,
or data that could be easily obtained, thispresents a major circumstantial risk
factor for adverse impacts on the humanrights and fundamental freedom of
persons.

NOT APPLICABLE

Actions to take for your HUDERIA:
No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:•

Data protection and privacy(data
security). You should also involve experts
to determine the potential sources ofdeanonymisation or identification throughdata linkage.

NO: No message

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether your AI system may deanonymize or
identify rights-holders through data linkage withexisting data, publicly available datasets, or datathat could be easily obtained and, if so, how tomanage
the
risks
of
this
potential
deanonymization or identification. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message

Data labelling and annotating practices33) Will processes of labelling andannotating the data used to
produce the AI system be

119

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messagetransparently reported and madeaccessible for audit, oversight, andreview by appropriate authorities
and relevant stakeholders?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS

Where processes of labelling andannotating the data used to produce theAI system are not transparently reported
and made accessible for audit, oversight,
and review by appropriate authorities andrelevant stakeholders, this presents amoderate modifiable risk factor foradverse impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:

UNSURENOT APPLICABLE

No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:

Example•Accountability and process transparency(traceability, auditability, accessibility,
and responsible governance)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to make the processes of labelling andannotating the data that will be used to produce
your AI system transparent and accessible for
audit, oversight, and review by appropriate

120authorities and relevant stakeholders. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message34) Where human labellers andannotators are involved, will
sufficient and transparently
reported processes be put into
place to mitigate potential labellingor annotation biases, especially incases where these activities
concern social and demographic
categories that can import patternsof discrimination and proxies forprotected characteristics?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURE

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

Where
sufficientandtransparently
reported processes are not put into placetomitigate
potentiallabellingor
annotation biases where human labellers
and annotators are involved, especially incases where these activities concern socialand demographic categories that canimport patterns of discrimination andproxies for protected characteristics, thispresents a moderate modifiable risk
factor for adverse impacts on the humanrights and fundamental freedom of
persons.

Actions to take for your HUDERIA:NOT APPLICABLE

121

No trigger message

Example

Goals, properties, and areas to focus on inyour HUDERAC:•

Fairness (non-discrimination and bias
mitigation), accountability and processtransparency (accessibility, traceability,
auditability, and responsible governance)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to mitigate any potential labelling or
annotation biases that may arise in the
production of your AI system. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message35) If data labelling or annotation ispartly or fully automated, will
sufficient and transparently
reported processes of humanoversight be implemented tomitigate the negative impact ofbiases generated by automatedlabelling or annotation, especiallyin cases where the dataset includes
social and demographic categories
that can import patterns of
discrimination and proxies forprotected characteristics?

122

Yes: No message
WE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

Where sufficient and transparently
reported processes of human oversightare not implemented to mitigate thenegative impact of biases generated byautomated labelling or annotation,

especially in cases where the dataset
includes social and demographic
categories that can import patterns of
discrimination and proxies for protectedcharacteristics, this presents a moderate
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURENOT APPLICABLE

Actions to take for your HUDERIA:
No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:•

ExampleFairness(non-discrimination,
bias
mitigation), accountability and processtransparency (traceability, auditability,
accessibility, and responsible governance)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to mitigate any potential labelling or
annotation biases generated by automated datalabelling or annotation. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message

Goal Setting and Problem Formulation Context
123

Decision to design36) Will an evaluation be carried out as
to whether building the AI systemis the right approach given
available resources and data,
existing technologies andprocesses, the complexity of theuse-contexts involved, and thenature of the policy or social
problem that needs to be solved?
YES, WE HAVE PLANS IN PLACETO DO THIS

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS

Where an evaluation is not carried out as
to whether building the AI system is theright approach given available resourcesand data, existing technologies andprocesses, the complexity of the usecontexts involved, and the nature of thepolicy or social problem that needs to besolved, this presents a moderate
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:

UNSURE•

ExampleYou should take steps to ensure that an
initial evaluation of this kind takes placeand incorporate its results into your
HUDERIA as part of your impact
assessment process.

Goals, properties, and areas to focus on inyour HUDERAC:•

124

Sustainability (reflection on context andimpacts)Unsure: Before taking any further steps in the
proposed project, you should evaluate, throughexpert and stakeholder input where appropriate,
whether building the AI system is the rightapproach given available resources and data,
existing technologies and processes, thecomplexity of the use-contexts involved, and thenature of the policy or social problem that needs
to be solved. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
Definition of outcome37) Will processes of formulating the
problem to be solved by the AI
system and of defining its target
variable (or measurable proxy) be
opened to input from stakeholder
engagement and public scrutiny?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THIS

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

WE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURE

Where processes of formulating the
problem to be solved by the AI systemand of defining its target variable (ormeasurable proxy) are not opened to
input from stakeholder engagement andpublic scrutiny, this presents amoderate modifiable risk factor foradverse impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:

Example•125

You should determine, based on thispreliminary risk analysis and yourstakeholder engagement process, theproportionate
levelofstakeholder
involvement, and seek public input
accordingly. This public input should
include(1)
determining
thereasonableness, fairness, equity, andjustifiability of the translation of the
project's objective into the statistical andmathematical frame and (2) determining
the alignment of that translation with thepotential impacts of the system on humanrights,fundamental
freedoms,
democracy, and the rule of law. The way
you integrate this public input into thesystem’s
problemformulation
andoutcome definition should be incorporated
into your HUDERIA as part of your impact
assessment process.
Goals, properties, and areas to focus on inyour HUDERAC:•

Sustainability (reflection on context andimpacts and stakeholder engagement andinvolvement)

UNSURE: Before taking any further steps in the
proposed project, you should evaluate the extent
to which you need to incorporate public input into
your process of ascertaining the AI system’s
problem formulation and outcome definition. To
attain this objective, you should determine,based on this preliminary risk analysis and yourstakeholder
engagement
process,
theproportionate level of stakeholder involvement,
and seek appropriate public input. When this126

information is ascertained, you should return toyour PCRA and revise it accordingly.

Model Design & Development Context
AI model characteristics38) If the algorithmic model(s) or
technique(s) used by the AI
system have a non-deterministic,
probabilistic, evolving, or dynamic
character that prevents or hinders
the system's intended functionality
from being formalized into specificand checkable design-time
requirements (or that impairscommonly accepted methods offormal verification and validation),
will the system interact with rightsholders in ways that could
adversely impact their humanrights and fundamental freedoms?

YES:
Major modifiable risk factor

•

YES
NO
UNSURENOT APPLICABLE

ExampleWhere the AI system directly interacts
with rights-holders in ways that could
adversely impact their human rights andfundamental freedoms where thealgorithmic model(s) or technique(s)used by the system have a nondeterministic, probabilistic, evolving, ordynamic character that prevents orhinders the system's intended
functionality from being formalized into
specific and checkable design-time
requirements (or that impairs commonly
accepted methods of formal verificationand validation), this presents a major
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:
No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:•

127

Safety(security,
reliability,
robustness), data quality, and

and
data

integrity as well as, where appropriate,
fairness (non-discrimination and bias
mitigation)

NO: No message
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,(1) whether the algorithmic model(s) or
technique(s) used by the AI system will have anon-deterministic, probabilistic, evolving, ordynamic character that prevents or hinders thesystem's intended functionality from being
formalized into specific and checkable designtime requirements (or that impairs commonly
accepted methods of formal verification andvalidation), and (2) whether the system willinteract with rights-holders in ways that could
adversely impact their their human rights andfundamental freedoms. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.

128

NOT APPLICABLE: No message39) If the algorithmic model(s) or
technique(s) used by the AI
system have a complex, highdimensional, or non-linear
character that impairs or prevents
the interpretability andexplainability of the system, will
the system interact with rightsholders in ways that could
adversely impact their humanrights and fundamental freedoms?

YES:
Major modifiable risk factor

•

YES
NO
UNSURENOT APPLICABLE

Where the AI system directly interacts
with rights-holders in ways that could
adversely impact their human rights andfundamental freedoms where thealgorithmic model(s) or technique(s)used by the AI system have a complex,
high-dimensional, or non-linear character
that impairs or prevents theinterpretability and explainability of thesystem, this presents a major
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:
No trigger message

ExampleGoals, properties, and areas to focus on inyour HUDERAC:•

Safety(security,
reliability,
androbustness) and explainability as well as,
whereappropriate,
fairness(nondiscrimination and bias mitigation)

NO: No message

129

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,(1) whether the algorithmic model(s) or
technique(s) used by the AI system have acomplex,high-dimensional,or
non-linear
character that impairs or prevents theinterpretability and explainability of the system,and (2) whether the system will interact withrights-holders in ways that could adversely
impact their physical, psychological, or moral
integrity or their human rights and fundamental
freedoms. When this information is ascertained,
you should return to your PCRA and revise it
accordingly.
NOT APPLICABLE: No message

Pre-processing
engineering

andfeature40) Where feature engineering,whether automated or carried out
by humans, involves the grouping,disaggregating, or excluding of
input features related to protectedor potentially sensitive
characteristics (e.g. decisions
about combining or separating
categories of gender or ethnic
groups) or proxies for these, will
the production of the AI system
incorporate processes to mitigate
emergent forms of bias and to
make the rationale behind thesedecisions transparent andaccessible to impacted rights-

130

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

Where feature engineering, whether
automated or carried out by humans,
involves the grouping, disaggregating, or
excluding of input features related toprotected or potentially sensitive
characteristics (e.g. decisions about
combining or separating categories ofholders and other relevant
stakeholders?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURENOT APPLICABLE

Examplegender or ethnic groups) or proxies forthese, the production of the AI systemdoes not incorporate processes tomitigate any emergent forms of bias andto make the rationale behind thesedecisions transparent and accessible toimpacted rights-holders and otherrelevant stakeholders
Actions to take for your HUDERIA:
No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:
Fairness (non-discrimination and bias
mitigation) and accountability and processtransparency (traceability, auditability,
accessibility,andresponsible
governance). In particular, you should
ensure that the feature engineering stageof the AI system’s lifecycle incorporates
processes to mitigate emergent forms ofbias and to make the rationale behindthese decisions transparent and accessible
to impacted rights-holders
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate measures to mitigate any
emergent forms of bias in the event that thefeature engineering stage of an AI system’s
production, whether automated or carried out by131

humans, involves the grouping, disaggregating,or excluding of input features related to protectedor potentially sensitive characteristics or proxies
for these. You should also determine how toincorporate and demonstrate such bias mitigation
measures in your HUDERAC. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message

Model selection41) Where complex or potentiallyopaque models are underconsideration, will processes ofmodel selection include appropriate
and transparent considerations of
the AI system's explainability bytaking into account:a. The normal expectations ofintelligibility andaccessibility that
accompany the function thesystem will fulfil in the
sector or domain in which it
will operateb. The availability of moreinterpretable algorithmic
models or techniques incases where the selection ofan opaque model posesrisks to the physical,
psychological, or moral
integrity of rights-holders or

132

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor• Where complex or potentially opaquemodels are under consideration andprocesses of model selection do notinclude appropriate and transparent
considerations of the AI system's
explainability, this presents a moderate
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.
Actions to take for your HUDERIA:
No trigger message

to their human rights andfundamental freedomsc. The availability of theresources and capacity thatwill be needed toresponsibly provide
supplementary methods of
explanation (e.g. simpler
surrogate models,sensitivity analysis, or
relative feature important)in cases where an opaquemodel is deemed
appropriate and selected?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS

Goals, properties, and areas to focus on inyour HUDERAC:•

Explainability(responsiblemodel
selection). Model selection considerations
shouldinclude(a)thenormalexpectationsofintelligibility
andaccessibility that accompany the function
the system will fulfil in the sector or
domain in which it will operate; (b) The
availabilityofmoreinterpretablealgorithmic models or techniques in cases
where the selection of an opaque model
poses risks to the physical, psychological,
or moral integrity of rights-holders or totheir human rights and fundamental
freedoms; (c) The availability of theresources and capacity that will be needed
to responsibly provide supplementary
methods of explanation (e.g. simpler
surrogate models, sensitivity analysis, or
relative feature important) in cases wherean opaque model is deemed appropriate
and selected.

Example 1

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate measures to include
explainability considerations in your model
selection process. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.

Example 2

NOT APPLICABLE: No message

UNSURENOT APPLICABLE

133

Model Output & Implementation Context
Model inference42) Will sufficient and transparently
reported processes be
implemented throughout the
project’s lifecycle to ensure thatthe inferences generated from themodel’s learning mechanisms arereasonable, fair, equitable, and do
not contain discriminatory
correlations or influences of lurking
or hidden proxies fordiscriminatory features that may
act as significant factors in thegeneration of its output?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

Sufficient and transparently reported
processes are not implemented
throughout the project’s lifecycle toensure that the inferences generated
from the model’s learning mechanisms
are reasonable, fair, equitable, and do
not contain discriminatory correlations or
influences of lurking or hidden proxies fordiscriminatory features that may act assignificant factors in the generation of its
outputActions to take for your HUDERIA:
No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:

UNSURE134

NOT APPLICABLE

Example•Fairness (non-discrimination and bias
mitigation) and accountability and processtransparency (traceability, auditability,
accessibility,andresponsible
governance). You should involve domain
experts and social scientists to determine
the potential sources of the hidden
discriminatory proxies or correlations that
may influence the outputs of the system,
and carry out comprehensive assessmentsof algorithmic bias and of the differential
performance of the system as its outputs
relate to vulnerable and marginalizedgroups.

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to ensure that the inferences generated fromthe model’s learning mechanisms are reasonable,
fair, equitable, and do not contain discriminatory
correlations or influences of lurking or hidden
proxies for discriminatory features that may act
as significant factors in the generation of its
output. You should also determine how toincorporate and demonstrate such measures inyour HUDERAC. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message

Model verification and validation

43) Will sufficient and transparently
reported processes of external peer
review and evaluation by

135

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageindependent domain and technicalexperts be included in theevaluation, verification, andvalidation of the AI model?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS

Sufficient and transparently reported
processes of external peer review andevaluation by independent domain andtechnical experts are not included in theevaluation, verification, and validation of
the AI model

Actions to take for your HUDERIA:
No trigger message

UNSURENOT APPLICABLE

ExampleGoals, properties, and areas to focus on inyour HUDERAC:•

Accountability and process transparency,
traceability, auditability, accessibility,
reproducibility,andresponsible
governance. You should make sure toincorporate peer review protocols from theearliest stage of the project lifecycle, so
that
your
workflow
documentation
processes deliberately enable seamlessexternal evaluation and reproducibility.

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to ensure that processes of external peer136review and evaluation by independent domain
and technical experts be included in theevaluation, verification, and validation of the AI
model. You should also determine how toincorporate and demonstrate such measures inyour HUDERAC. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message44) Will processes for evaluating thetrained AI model include
transparently reported external
validation?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURE

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

Where processes for evaluating thetrained AI model do not include
transparently reported external
validation, this presents a moderate
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:NOT APPLICABLE
No trigger message

137

Example
Goals, properties, and areas to focus on inyour HUDERAC:•

Accountability and process transparency(traceability, auditability, reproducibility,
accessibility,andresponsible
governance). You should make sure to
consult domain experts (and, whereappropriate,
relevantusers
andpractitioners)
duringtheexternal
validation process, so that you have arealistic understanding of your system’s
operating environment and of the realworld conditions that may hamper the
performance of the trained model.

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to ensure that processes of external
validation be included in the evaluation of the
trained AI model. You should also determine how
to incorporate and demonstrate such measures inyour HUDERAC. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message45) Will processes of monitoring the AI
system during its operation involve
regular re-evaluations of138

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageperformance that are sufficient tokeep pace with real world changes
that may cause concept drifts andshifts in underlying data
distributions?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS

Actions to take for your HUDERIA:•

UNSURENOT APPLICABLE

ExampleProcesses of monitoring the AI system
during its operation do not involve
regular re-evaluations of performancethat are sufficient to keep pace with realworld changes that may cause concept
drifts and shifts in underlying data
distributions

Make sure to incorporate into your impact
assessment process—especially in its
post-development stage iteration—an
exploration and provisional determinationof the timeframes that are appropriate for
performance re-evaluation and impact reassessment.

Goals, properties, and areas to focus on inyour HUDERAC:•

139

Safety,accuracy
andperformance
metrics,
reliability,
robustness,sustainability, change monitoring, andreflection on context and impacts. You
should involve domain experts todetermine the potential sources of
distributional shifts or drifts and build
transparently processes of dynamic

assessment,
re-assessment,
external
validation, and monitoring into yourproject lifecycle.
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to ensure that processes of monitoring yourAI system during its operation involve regular reevaluations of performance that are sufficient tokeep pace with real world changes that may
cause concept drifts and shifts in underlying data
distributions. You should also determine how toincorporate and demonstrate such measures inyour HUDERAC. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message

Model accuracy
metrics

andperformance46) When performance metrics for the
AI system are considered,determined, and reported, will theprioritization of error types (e.g.
false positives/negatives) be
a. Informed by the specificcontext of the use case andby the potential effects ofdifferential error rates on
affected sub-populations (inparticular, on vulnerable orprotected groups)

140

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

When performance metrics for the AI
system are considered, determined, andreported, the prioritization and reportingof error types (e.g. false
positives/negatives) are (a) not informed

b. Clearly and accessiblypresented, so that therationale behind the chosen
metrics is made explicit andunderstandable in plain,
non-technical language?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS

by the specific context of the use caseand the potential effects of differential
error rates on affected sub-populations(in particular, on vulnerable or protectedgroups) and (b) not clearly andaccessibly presented, so that therationale behind the chosen metrics ismade explicit and understandable inplain, non-technical language
Actions to take for your HUDERIA:
No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:•

UNSURENOT APPLICABLE

ExampleSafety,accuracy
andperformance
metrics, accessibility, explainability. You
should ensure that the selection andreporting of error types are informed by
the specific context of the use case andthe potential effects of differential error
rates on affected sub-populations.

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to ensure that the selection and reporting oferror types are informed by the specific context
of the use case and the potential effects ofdifferential error rates on affected subpopulations. You should also determine how toincorporate and demonstrate such measures inyour HUDERAC. When this information is141

ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message47) When performance metrics for the
AI system are considered,determined, and presented, will
the prioritization and reporting of
metrics beyond accuracy (e.g.
sensitivity, precision, specificity) be
informed by the specific context of
the use case and its performanceneeds (e.g. a system whose
effective identification of rare
events is more critical than its
overall accuracy rate)?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Moderate modifiable risk factor•

When performance metrics for the AI
system are considered, determined, andreported, the prioritization and reportingof metrics beyond accuracy (e.g.
sensitivity, precision, specificity) is not
informed by the specific context of theuse case and its performance needs (e.g.
a system whose effective identification ofrare events is more critical than its
overall accuracy rate)

Actions to take for your HUDERIA:
No trigger message

UNSURENOT APPLICABLE

142

Goals, properties, and areas to focus on inyour HUDERAC:•

ExampleSafety,accuracy
andperformance
metrics, accessibility, explainability. You
should ensure that the selection andreporting of performance metrics for yourAI system beyond accuracy are informedby its specific context and its performanceneeds. You should also determine how toincorporate and demonstrate this in your
HUDERAC.

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to ensure that the selection and reporting of
performance metrics for your AI system beyond
accuracy (e.g. sensitivity, precision, specificity)
are informed by the specific context of the usecase and its performance needs. You should also
determine how to incorporate and demonstratesuch measures in your HUDERAC. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message

System-User Interface and Human Factors Context
Implementers or users of the system48) Will the project lifecycle
incorporate sufficient processes toensure that the deployment of thesystem does not harm thephysical, psychological, or moral

143

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREintegrity of implementers or
adversely impact their dignity,
autonomy, and ability to makefree, independent, and wellinformed judgements?

UNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISThe project lifecycle does not incorporate
sufficient processes to ensure that the
deployment of the system does not harm
the physical, psychological, or moral
integrity of implementers or adversely
impact their dignity, autonomy, and ability
to make free, independent, and wellinformed judgements

Actions to take for your HUDERIA:•

UNSURENOT APPLICABLE

ExampleMake sure to incorporate into your impact
assessment
processthoroughconsideration of the possible effects of the
AI system on the human rights andfundamental
freedoms
ofits
implementers. As affected rights-holders,
implementers should be included as
participants in the HUDERIA process. You
should also integrate any mitigation
measures for potential adverse impacts
identified into your HUDERAC.

Goals, properties, and areas to focus on inyour HUDERAC:•

Sustainability (reflection on context andimpacts, and responsible implementation
and user training)

UNSURE: Before taking any further steps in the
proposed project, you should determine, through144

expert and stakeholder input where appropriate,
how to ensure that the deployment of the systemdoes not harm the physical, psychological, or
moral integrity of implementers or adversely
impact their dignity, autonomy, and ability tomake free, independent, and well-informed
judgements.
When
this
informationisascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message

Level of automation/level of human
involvement and choice49) Will implementers of the AI system
be sufficiently trained so that they
are able to fully understanda. the strengths and limitationof the system and its
outputsb. the potential conditions ofsituational complexity,
uncertainty, anomaly, or
system failure that may
dictate the need for the
exercise of humanjudgment, common sense,and practical intervention?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THIS

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

Implementers of the AI system are notsufficiently trained so that they are able
to fully understand both the strengths
and limitation of the system (and its
outputs) and the potential conditions ofsituational complexity, uncertainty,
anomaly, or system failure that may
dictate the need for the exercise ofhuman judgment, common sense, andpractical intervention

Actions to take for your HUDERIA:

145

WE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURENOT APPLICABLE

No trigger message
Goals, properties, and areas to focus on inyour HUDERAC:•

Responsible
training.

implementation

and

user

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to ensure that implementers of the AI
system are sufficiently trained to fully understandboth the strengths and limitation of the system(and its outputs) and the potential conditions ofsituational complexity, uncertainty, anomaly, or
system failure that may dictate the need for the
exercise of human judgment, common sense, andpractical intervention. You should also determine
how to incorporate and demonstrate such training
measures in your HUDERAC. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.

ExampleNOT APPLICABLE: No message50) If the AI system has a high level ofautomation or operational'autonomy' and interacts withrights-holders in ways that could
adversely impact their physical,
psychological, or moral integrity or

146

YES: No message

NO: STOP.

harm their human rights andfundamental freedoms, willmechanisms of human control andintervention (e.g. human-in-theloop or human-on-the-loop) be
incorporated into theimplementation of the system?

YES
NO
UNSURENOT APPLICABLE

ExampleProhibitive modifiable risk factor
Where AI systems have high levels of automation
or operational ‘autonomy’ and interact withrights-holders in ways that could adversely
impact their physical, psychological, or moral
integrity or harm their human rights andfundamental
freedoms,
implementation
processes
must
incorporate
appropriate
mechanisms of human control and intervention.
Systems that fail to do this pose too high a risk
to impacted rights-holders and communities to be
developed or implemented in that form. In
revising your project, you should ensure thatprocesses of human control and intervention are
integrated into the system’s deployment anddetermine how to demonstrate this in your
HUDERAC. If you wish to continue with the PCRA
anyway at this time, select yes. If not, pleasereconsider your project and return when you haveascertained that your AI system will havesufficient mechanisms that enable human control
and intervention.
UNSURE: STOP. Where AI systems have highlevels of automation or operational ‘autonomy’and interact with rights-holders in ways that
could
adversely
impacttheirphysical,
psychological, or moral integrity or harm their
human rights and fundamental freedoms,
implementation processes must incorporate
appropriate mechanisms of human control andintervention. Systems that fail to do this pose toohigh a risk to impacted rights-holders andcommunities to be developed or implemented inthat form. Before taking any further steps in the147

proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to ensure that, if your AI system has a highlevel of automation and operational 'autonomy'
and interacts with rights-holders in ways that
could
adversely
impacttheirphysical,
psychological, or moral integrity or harm their
human rights and fundamental freedoms,
processes of human control and intervention areappropriately integrated into the system’s
deployment. If you wish to continue with the
PCRA anyway at this time, select yes. If not,please reconsider your project and return when
you have ascertained that your AI system will
have sufficient mechanisms that enable humancontrol and intervention.
NOT APPLICABLE: No message

Rights & Freedoms Context
Respect for and protection of humandignity51) If the AI system interacts withrights-holders (users or decisionsubjects) in ways that could
adversely impact their physical,
psychological, or moral integrity or
harm their dignity, will they beable to opt out of the interaction
and revert to human intervention?

YES: No message
NO: STOP
Prohibitive modifiable risk factor
Where AI systems interact with rights-holders(users or decision subjects) in ways that could
adversely impact their physical, psychological, or
moral integrity or harm their dignity but they are
not given the choice to opt out of the interaction

YES
NO

148

UNSURENOT APPLICABLE

Exampleand revert to human intervention, such systems
are considered to pose too high a risk to impacted
rights-holders and communities for you toproceed. In revising your project, you should pay
close attention to ensuring the availability andaccessibility of an opt out option for impacted
rights-holders, and to demonstrating this in your
HUDERAC. You should also pay close attention inyour HUDERIA to assessing the impact of thesystem on human dignity. If you wish to continue
with the PCRA anyway at this time, select yes. If
not, please reconsider your project and return
when you have ascertained that your AI system
will have sufficient mechanisms that ensure theavailability and accessibility of an opt out option
for impacted rights-holders.
UNSURE: STOP Before taking any further steps inthe proposed project, you should determine,through expert and stakeholder input whereappropriate, whether, in cases where the AI
system will interact with rights-holders (users or
decision subjects), they will be given the choice
to opt out of the interaction and revert to humanintervention if they feel that their dignity risks
being violated. If no option is given, the system
is considered to pose too high a risk to impacted
rights-holders and communities for you toproceed. In revising your project, you should pay
close attention to ensuring the availability andaccessibility of an opt out option for impacted
rights-holders, and to demonstrating this in your
HUDERAF. You should also pay close attention inyour HUDERIA to assessing the impacts of thesystem on human dignity. If you wish to continue
with the PCRA anyway at this time, select yes. If
not, please reconsider your project and return

149

when you have ascertained that your AI system
will have sufficient mechanisms that ensure theavailability and accessibility of an opt out option
for impacted rights-holders.
NOT APPLICABLE: No message52) If the AI system interacts directlywith rights-holders (users or
decision subjects) in ways that
could adversely impact their
physical, psychological, or moral
integrity or harm their dignity, will
they have pre-knowledge of thisinteraction and be able to provide
meaningful and informed consent
to participate in the interaction?
YES
NO
UNSURENOT APPLICABLE

Example150

YES: No message
NO: STOP
Prohibitive modifiable risk factor
Where AI systems interact directly with rightsholders (users or decision subjects) in ways that
could
adversely
impacttheirphysical,
psychological, or moral integrity or harm theirdignity but they do not have pre-knowledge ofthis interaction and are not able to provide
informed consent to participate in the interaction,such systems are considered to pose too high arisk to impacted rights-holders and communities
for you to proceed. In revising your project, you
should pay close attention to ensuring that
impacted rights-holders are provided clear
foreknowledge about interacting with the system
and are able to provide meaningful and informed
consent to participate in the interaction. Thisshould be demonstrated in your HUDERAC. You
should also pay close attention in your HUDERIA
to assessing the impacts of the system on humandignity and human freedom and autonomy. If you
wish to continue with the PCRA anyway at thistime, select yes. If not, please reconsider yourproject and return when you have ascertainedthat your AI system will have sufficientmechanisms to ensure that impacted rightsholders are provided clear foreknowledge about
interacting with the system and are able toprovide meaningful and informed consent toparticipate in the interaction.
UNSURE: STOP Before taking any further steps inthe proposed project, you should determine,through expert and stakeholder input whereappropriate, whether, in cases where the AI
system interacts directly with rights-holders(users or decision subjects) in ways that could
adversely impact their physical, psychological, or
moral integrity or harm their dignity, they will
have pre-knowledge of this interaction and beable to provide meaningful and informed consent
to participate in the interaction. If not, the system
is considered to pose too high a risk to impacted
rights-holders and communities for you toproceed. In revising your project, you should pay
close attention to ensuring that impacted rightsholders are provided clear foreknowledge about
interacting with the system and are able toprovide meaningful and informed consent toparticipate in the interaction. This should bedemonstrated in your HUDERAC. You should also
pay close attention in your HUDERIA to assessing
the impacts of the system on human dignity,
human freedom, and autonomy. If you wish tocontinue with the PCRA anyway at this time,
select yes. If not, please reconsider your projectand return when you have ascertained that yourAI system will have sufficient mechanisms toensure that impacted rights-holders are provided

151

clear foreknowledge about interacting with thesystem and are able to provide meaningful andinformed consent to participate in the interaction.
NOT APPLICABLE: No message
Protection
autonomy

of

human

freedomand53) Will the project lifecycle
incorporate sufficient andtransparently reported processes
to ensure the ability of rightsholders
a. to make free and wellinformed judgements about
the reasonableness andjustifiability of the outputsof the AI systemb. to effectively contest andchallenge decisions
informed and/or made bythat systemc. to demand that suchdecisions be reviewed by aperson?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS152

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

Where
project
lifecycles
donotincorporate sufficient processes to ensure
the ability of rights-holders to make free,independent,
andwell-informed
judgements about the reasonableness andjustifiability of the outputs of the AI
system, to effectively contest andchallenge decisions informed and/or madeby that system, and to demand that suchdecision be reviewed by a person, thispresents a major modifiable risk factor
for adverse impacts on the human rights
and fundamental freedom of persons.

Actions to take for your HUDERIA:

NO, WE ARE NOT PLANNING TO
DO THIS•UNSURENOT APPLICABLE

Example 1

Make sure to incorporate into your impact
assessment
processthoroughconsideration of the possible effects of the
AI system on the freedom and autonomy
of affected rights-holders and on theirabilities to make free, independent, andwell-informed decisions.

Goals, properties, and areas to focus on inyour HUDERAC:

Example 2•

Sustainability (reflection on context andimpacts) and accountability and processtransparency (traceability, auditability,
accessibility, and responsible governance)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how you can incorporate sufficient processes inyour project lifecycles to ensure the ability ofrights-holders to make free, independent, andwell-informed
judgements
about
the
reasonableness and justifiability of the outputs of
the AI system, to effectively contest andchallenge decisions informed and/or made bythat system, and to demand that such decision bereviewed by a person. You should also determine
how to incorporate and demonstrate suchmeasures in your HUDERAC. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message

153

Non-discrimination,equality

fairness,and54) Will sufficient and transparently
reported processes be
implemented throughout the
project’s lifecycle to ensure that
the AI system, in both its
production and use, mitigatespossible sources of bias anddiscriminatory patterns in each of
the following:a. The datasets used to train thesystemb. The decisions made to build thesystemc. The way the problem to which
the system responds isunderstood, formulated, andframedd. The way the target variable andits measurable proxy aredefinede. The way the system's
algorithmic model(s) is
selected, and its parameters
tuned and adjusted
f. The way that model is trained,
tested, and validated
g. The way the system isimplemented, and the way
users are trained to deploy it
h. The choices made about
monitoring, updating,repurposing, or deprovisioning
the system?

154

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

Where
project
lifecycles
donotincorporate sufficient and transparently
reported processes to ensure that AI
systems, in both their production and use,
mitigate possible sources of bias anddiscriminatory patterns, this presents amajor modifiable risk factor foradverse impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:•

Make sure to incorporate into your impact
assessment
processthoroughconsideration of the principles andpriorities of non-discrimination, fairness,and equality. You should also pay special
attention in your HUDERIA to assessing
the impacts of the system on historicallymarginalized and vulnerable groups andon groups with protected characteristics.

Goals, properties, and areas to focus on inyour HUDERAC:

YES, WE HAVE PLANS IN PLACETO DO THIS•WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURENOT APPLICABLE

ExampleFairness (non-discrimination and bias
mitigation) and accountability and processtransparency (traceability, auditability,
accessibility,andresponsible
governance).

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate sufficient and transparently
reported processes to ensure that your AI
systems, in both its production and use, mitigatespossible sources of bias and discriminatory
patterns. You should also determine how toincorporate and demonstrate such measures inyour HUDERAC. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message55) Will sufficient and transparently
reported processes be
implemented throughout the
project’s lifecycle to ensure thatthe system, in both its production
and use, promotes diversity andinclusivenessa. In the composition of
innovation team building
the system

155

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

Where
project
lifecycles
donotincorporate sufficient and transparently
reported processes to ensure that AI

b. In the expertise, insight,
and knowledge drawn upon
to develop itc. In the individuals andgroups able to access its
benefits?
YES, WE HAVE PLANS IN PLACETO DO THIS

systems, in both their production and use,
promote diversity and inclusiveness, thispresents a major modifiable risk factor
for adverse impacts on the human rights
and fundamental freedom of persons.
Actions to take for your HUDERIA:•WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURENOT APPLICABLE

ExampleMake sure to incorporate into your impact
assessment
processthoroughconsideration of the principles andpriorities of diversity, inclusiveness, nondiscrimination, fairness, and equality. You
should also pay special attention in your
Stakeholder Engagement Process toassessing team positionality and the needfor diversification of team composition,
expertise, insight, and knowledge.

Goals, properties, and areas to focus on inyour HUDERAC:•

Fairness (non-discrimination, bias
mitigation, diversity, and inclusiveness)and sustainability (reflection on context
and impacts and stakeholder
engagement and involvement)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to ensure that your AI system, in both its
production and use, promotes diversity andinclusiveness. You should also determine how toincorporate and demonstrate such measures inyour HUDERAC. When this information is156

ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message

Data protection and privacy context56) If the AI system will be processing
personal data, will the project
lifecycle incorporate transparently
reported mechanisms that
demonstrate its compliance withdata protection and privacy law
and the principles set out in theCouncil of Europe's Convention
108+?
YES, WE HAVE PLANS IN PLACETO DO THIS

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURE

Where project lifecycles for AI systems
that process personal data do notincorporatetransparently
reportedmechanisms that demonstrate theircompliance with data protection andprivacy law and the principles set out in
the Council of Europe's Convention 108+,
this presents a major modifiable risk
factor for adverse impacts on the humanrights and fundamental freedom of
persons.

Actions to take for your HUDERIA:NOT APPLICABLE

157•

Make sure to pay special attention in your
HUDERIA to assessing the impacts of thesystem on data protection and the respect
for private and family life, drawing on your

Example

Data Protection Impact(DPIA), if applicable.

Assessment

Goals, properties, and areas to focus on inyour HUDERAC:•

Data protection and privacy(transparency, proportionality, consent or
legitimate basis for processing, data
quality, data security, purpose limitation,accountability, data minimisation, dataprotection and privacy by design,
fairness, and lawfulness). You should
consult data protection experts to
examine and determine the measuresthat your project needs to take toprocess personal data lawfully and inaccordance with the principles set out inConvention 108+, including thecompletion of a Data Protection Impact
Assessment (DPIA)(undertaken either aspart of transparent reporting or as acompliance mechanism or as both). You
should include your DPIA in your
HUDERAC, if applicable, to evidence yourcompliance and best practices.

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughinput from data protection experts, the measuresthat your project needs to take to process
personal data lawfully and in accordance with theprinciples set out in Convention 108+, including
the completion of a Data Protection Impact
Assessment (DPIA)(undertaken either as part oftransparent reporting or as a compliancemechanism or as both). When this information is158

ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message57) If the AI system will be processing
personal data, will the project
lifecycle incorporate transparently
reported mechanisms that
demonstrate its respect for therights of data subjects andconformity to the additional
obligations of data controllers andprocessors as set out in Articles 9
and 10 of Convention 108+?
YES, WE HAVE PLANS IN PLACETO DO THIS

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURE

Where project lifecycles for AI systems
that process personal data do notincorporatetransparently
reportedmechanisms that demonstrate theirrespect for the rights of data subjects andconformity to the additional obligations ofdata controllers and processors as set out
in Articles 9 and 10 of Convention 108+,
this presents a major modifiable risk
factor for adverse impacts on the humanrights and fundamental freedom of
persons.

Actions to take for your HUDERIA:NOT APPLICABLE

159•

Make sure to pay special attention in your
HUDERIA to assessing the impacts of thesystem on data protection, the rights ofthe data subject, and the respect forExample

private and family life, drawing on your
DPIA, if applicable.
Goals, properties, and areas to focus on inyour HUDERAC:•

Data protection and privacy (respect for
the rights of data subjects andconformity to additional obligations—
Articles 9 and 10, Convention 108+) You
should consult data protection experts to
examine and determine the measuresthat your project needs to take toprocess personal data lawfully and inaccordance with the principles set out inConvention 108+, including thecompletion of a Data Protection Impact
Assessment (DPIA) (undertaken either aspart of transparent reporting or as acompliance mechanism or as both). You
should include your DPIA in your
HUDERAC, if applicable, to evidence yourcompliance and best practices.

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughinput from data protection experts, the measuresthat your project needs to take to process
personal data in accordance with obligations ofdata controllers and processors as set out in
Articles 9 and 10 of Convention 108+. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message

16058) If the AI system will be processing
sensitive data as defined inConvention 108+, will the project
lifecycle incorporate transparently
reported mechanisms that
demonstrate its compliance withappropriate safeguards?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THIS

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

WE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURENOT APPLICABLE

Where project lifecycles for AI systems
that process sensitive data as defined inConvention 108+ do not incorporatetransparently reported mechanisms that
demonstrate
theircompliancewithappropriate safeguards, this presents amajor modifiable risk factor foradverse impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:•

ExampleMake sure to pay special attention in your
HUDERIA to assessing the impacts of thesystem on data protection, the rights ofthe data subject, and the respect for
private and family life, drawing on your
DPIA, if applicable.

Goals, properties, and areas to focus on inyour HUDERAC:•

161

Data protection and privacy (responsible
handling of sensitive data). You should
consult data protection experts toexamine and determine the measuresthat your project needs to take toprocess personal data lawfully and inaccordance with the principles set out inConvention 108+, including thecompletion of a Data Protection Impact
Assessment (DPIA)(undertaken either aspart of transparent reporting or as acompliance mechanism or as both). You
should include your DPIA in your
HUDERAC, if applicable, to evidence yourcompliance and best practices.
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughinput from data protection experts, the measuresthat your project needs to take to process
sensitive data as defined in Convention 108+ incompliance with appropriate safeguards. When
this information is ascertained, you should return
to your PCRA and revise it accordingly.
NOT APPLICABLE: No message59) If the AI system is designed withthe purpose or function ofindividual-targeted curation,
profiling, prediction, or behaviouralsteering, will affected rightsholders be able to obtain from thedata controller sufficient
information concerning:

162

YES: No message

NO: STOP. AI systems that are designed with the
purpose or function of individual-targeted
curation, profiling, prediction, or behaviouralsteering and that do not make available toaffected rights-holders sufficient information
concerning data use, output rationale, purpose,and conveyance of processing results area. The use of their personal
data and the categories
used in the system's
processingb. An explanation of therationale behind the outputof the processing in plain,
non-technical languagec. The purpose of the curation,
profiling, prediction,
classification, or behaviouralsteering
d. The categories of persons or
bodies to whom personaldata, the profile or the
result of the processing may
be communicated?
YES
NO
UNSURENOT APPLICABLE

Example163

considered to pose too high a risk to impacted
rights-holders and communities for you toproceed. In revising your project, you should pay
close attention to ensuring that impacted rightsholders are provided clear, accessible, andsufficient information concerning data use, output
rationale, purpose, and conveyance of processing
results You should consult data protection experts
to examine and determine the measures thatyour project needs to take to accomplish this. If
you wish to continue with the PCRA anyway atthis time, select yes. If not, please reconsider
your project and return when you haveascertained that your AI system will havesufficient mechanisms to ensure that impacted
rights-holders are provided clear, accessible, andsufficient information concerning data use, output
rationale, purpose, and conveyance of processing
results.
UNSURE: STOP Before taking any further steps inthe proposed project, you should determine,through input from data protection experts,
whether, in the case that your AI system will be
designed with the purpose or function ofindividual-targeted curation, profiling, prediction,or behavioural steering, you will make available
to affected rights-holders sufficient information
concerning data use, output rationale, purpose,and conveyance of processing results. If not, thesystem is considered to pose too high a risk to
impacted rights-holders and communities for you
to proceed. In revising your project, you should
pay close attention to ensuring that impacted
rights-holders are provided clear, accessible, andsufficient information concerning data use, output
rationale, purpose, and conveyance of processing

results You should consult data protection experts
to examine and determine the measures thatyour project needs to take to accomplish this. If
you wish to continue with the PCRA anyway atthis time, select yes. If not, please reconsider
your project and return when you haveascertained that your AI system will havesufficient mechanisms to ensure that impacted
rights-holders are provided clear, accessible, andsufficient information concerning data use, output
rationale, purpose, and conveyance of processing
results.
NOT APPLICABLE: No message

Accountability and access to justice60) Will sufficient and transparently
reported processes be
implemented throughout the
project’s lifecycle to ensure end-toend accountability across the
production and use of the AI
system? Namely, will it ensure thatthe systema. Is auditable by design,
allowing for the end-to-end
traceability and oversight ofits processes of production
and useb. Establishes a continuous
chain of human
responsibility for all roles
involved in the project
lifecycle to allow for end-to-164

YES: No message
NO:
Major modifiable risk factor•

Where
project
lifecycles
donotincorporate sufficient and transparently
reported processes to ensure end-to-end
accountability across the production anduse of AI systems, this presents a major
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:
No trigger message

end answerability in theevent that the human rights
or fundamental freedoms ofaffected individuals havebeen negatively impacted
c. Enables designated publicauthorities and third
parties, where appropriate,
to assess its compliancewith existing legislation,
regulation, and standards
instruments across theentire project lifecycle?

Goals, properties, and areas to focus on inyour HUDERAC:•

Accountability and process transparency(traceability, auditability, accessibility,
and responsible governance).

UNSUREUNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate transparently reported
processes to ensure sufficient accountability
across the production and use of the AI system.
You should pay special attention to finding out
how to assure, in your HUDERAC, the goals of
accountability, transparency, and explainability
and how to demonstrate this in a clear andaccessible way. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.

NOT APPLICABLENOT APPLICABLE: No message

YES
NO

Example61) Will sufficient and transparently
reported processes be
implemented throughout the
project’s lifecycle to ensure that
affected persons whose humanrights or fundamental freedoms
have been adversely impacted by

165

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:

the AI system have actionable
redress and effective remedy bya. Providing sufficient andmeaningful information thatindicates when the system
is being used and how andwhere to complain in theevent of an adverse impact
on human rights andfundamental freedoms
b. Facilitating access ofaffected rights-holders to
sufficient and meaningful
information about theprocesses
behindthedesign, development, anddeployment of the system
and about the rationale
underlying the outcomes of
its processingc. Employing algorithmic
models that areappropriately interpretableor explainable (especiallyregarding discriminatory
proxies or inferences that
may be embedded intrained machine learning
systems) given the risks tohuman rights, fundamental
freedoms, democracy, andthe rule of law they may
pose?

166

Major modifiable risk factor•

Where project lifecycles do not implement
sufficient and transparently reported
processes to ensure that affectedindividuals have actionable redress andeffective remedy in cases where their
human rights or fundamental freedoms
have been adversely impacted, thispresents a major modifiable risk factor
for adverse impacts on the human rights
and fundamental freedom of persons.

Actions to take for your HUDERIA:•

Make sure to pay close attention toassessing the impacts of the system on
human freedom and autonomy and on
the ability of rights-holders to make free,independent, and well-informed
decisions.

Goals, properties, and areas to focus on inyour HUDERAC:•

Accountability and process transparency(traceability, auditability, accessibility,
and responsible governance),
explainability (interpretability andaccessible rationale explanation). Thisshould include incorporating into your
HUDERAC evidence that you will be ableto (a) provide sufficient and meaningful
information that indicates when thesystem is being used and how and where
to complain in the event of an adverse

YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURENOT APPLICABLE

Exampleimpact on human rights and fundamental
freedoms, (b) facilitate access of affectedrights-holders to sufficient andmeaningful information about theprocesses behind the design,development, and deployment of thesystem and about the rationale
underlying the outcomes of its
processing, and (c) employ algorithmic
models that are appropriatelyinterpretable or explainable (especiallyregarding discriminatory proxies orinferences that may be embedded intrained machine learning systems) given
the risks to human rights, fundamental
freedoms, democracy, and the rule of law
they may pose.
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate transparently reported
processes to ensure that affected individuals haveactionable redress and effective remedy in cases
where their human rights or fundamental
freedoms have been adversely impacted. You
should pay special attention to finding out how toassure, in your HUDERAC, the goals of
accountability, transparency, and explainability
and how to demonstrate this in a clear andaccessible way. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.
NOT APPLICABLE: No message

16762) If the AI system is used in the fieldof justice and law enforcement, will
meaningful information beprovided to affected rights-holders
about the existence and use of thesystem, its role within law
enforcement and the judicial
process, and the right to challenge
the decisions informed or madethereby?
YES, WE HAVE PLANS IN PLACETO DO THIS

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THIS

Where AI system are being used in thefield of justice and law enforcement andmeaningful information is not provided toaffectedrights-holders
about
the
existence and use of the system, its role
within law enforcement and the judicial
process, and the right to challenge thedecisions informed or made thereby, thispresents a major modifiable risk factor
for adverse impacts on the human rights
and fundamental freedom of persons.

Actions to take for your HUDERIA:

UNSURENOT APPLICABLE

Example•Make sure to pay special attention in your
HUDERIA to assessing the potential
impacts of the system on individual
dignity, freedom, and autonomy, and on
how effects on legal processes andinstitutions may damage the rule of law.

Goals, properties, and areas to focus on inyour HUDERAC:

168

•

Accountability and process transparency(traceability, auditability, accessibility,andresponsible
governance)
andexplainability(interpretability
andaccessible rationale explanation)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how meaningful information can be provided toaffected rights-holders about the existence anduse of the system, its role within law enforcement
and the judicial process, and the right tochallenge the decisions informed or madethereby. You should pay special attention tofinding out how to assure, in your HUDERAC, thegoals of accountability, transparency, andexplainability and how to demonstrate this in aclear and accessible way. When this information
is ascertained, you should return to your PCRA
and revise it accordingly.
NOT APPLICABLE: No message63) If the AI system is used in the fieldof justice and law enforcement, will
sufficiently and transparently
reported processes be
implemented throughout the
project’s lifecycle to ensure that its
deployment is in line with theessential requirements of impacted
individuals' right to a fair trial(equality of arms, right to a natural

169

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor

judge established by law, the rightto an independent and impartial
tribunal, and respect for theadversarial process)?

•

YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURENOT APPLICABLE

Where AI system are being used in thefield of justice and law enforcement andsufficiently and transparently reported
processes
arenotimplemented
throughout the project’s lifecycle toensure that its deployment is in line withthe essential requirements of impacted
individuals' right to a fair trial (equality ofarms, right to a natural judge establishedby law, the right to an independent andimpartial tribunal, and respect for theadversarial process), this presents amajor modifiable risk factor foradverse impacts on the human rights andfundamental freedom of persons.

Actions to take for your HUDERIA:•

ExampleMake sure to pay special attention in your
HUDERIA to assessing the potential
impacts of the system on individual
dignity, freedom, and autonomy, and on
how effects on legal processes andinstitutions may damage the rule of law.

Goals, properties, and areas to focus on inyour HUDERAC:•

170

Accountability and process transparency(traceability, auditability, accessibility,andresponsible
governance)
andexplainability(interpretability
andaccessible rationale explanation)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate transparently reported
processes to ensure that its deployment is in linewith the essential requirements of impacted
individuals' right to a fair trial. You should pay
special attention to finding out how to assure, inyour HUDERAC, the goals of accountability,transparency, and explainability and how todemonstrate this in a clear and accessible way.
When this information is ascertained, you should
return to your PCRA and revise it accordingly.
NOT APPLICABLE: No message

Social and economic rights64) If the AI system is being used in
the areas of employment decisions,
recruitment, worker management,or the distribution of work, will the
project lifecycle incorporatetransparently reported processes
to ensure that the system protects
a. The dignity of work and theright to just working
conditions as set out in
Article 2 of the European
Social Charterb. The right to safe andhealthy working conditions
as set out in Article 3 of theEuropeans Social Charter

171

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

Where AI systems are being used in theareasofemployment
decisions,
recruitment, worker management, or the
distribution of work and project lifecycles
do not incorporate transparently reported
processes to ensure that the system
protects the dignity of work, the right tojust working conditions, the right to safe

c. The right to organize as set
out in Article 5 of theEuropean Social Charter?
YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISand healthy working conditions, and theright to organize as set out in theEuropean Social Charter, this presents amajor modifiable risk factor foradverse impacts on the human rights andfundamental freedom of persons.
Actions to take for your HUDERIA:•WE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURENOT APPLICABLE

Example a
Example b

Make sure to pay close attention, in your
HUDERIA, to assessing the impacts of thesystem on social and economic rights,
equality, dignity, and the right to physical,
psychological, or moral integrity.

Goals, properties, and areas to focus on inyour HUDERAC:•

Fairness (equality, non-discrimination,
bias mitigation, diversity, andinclusiveness) and sustainability(reflection on context and impacts andstakeholder engagement andinvolvement)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate transparently reported
processes to ensure that the system protects the
dignity of work, the right to just working
conditions, the right to safe and healthy working
conditions, and the right to organize as set out inthe European Social Charter. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.

Example c

172

NOT APPLICABLE: No message65) If the AI system is being used in
the context of social security
decisions or social welfare
administration, will sufficient andtransparently reported processes
be implemented throughout the
project’s lifecycle to ensure thatthe system protects the right tosocial security as set out in Article
12 of the European Social Charter?

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

YES, WE HAVE PLANS IN PLACETO DO THISWE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURE

Actions to take for your HUDERIA:•NOT APPLICABLE

ExampleWhere AI systems are being used in thecontext of social security decisions orsocial welfare administration and project
lifecycles do not incorporate transparently
reported processes to ensure that thesystem protects the right to social security
as set out in Article 12 of the European
Social Charter, this presents a major
modifiable risk factor for adverse
impacts on the human rights andfundamental freedom of persons.

Make sure to pay close attention, in your
HUDERIA, to assessing the impacts of thesystem on social and economic rights,
equality, dignity, and the right to physical,
psychological, or moral integrity.

Goals, properties, and areas to focus on inyour HUDERAC:

173

•

Fairness (equality, non-discrimination,
bias mitigation, diversity, andinclusiveness) and sustainability(reflection on context and impacts andstakeholder engagement andinvolvement)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate transparently reported
processes to ensure that the system protects theright to social security as set out in Article 12 of
the European Social Charter. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message66) If the AI system is being used toautomate decisions regarding theprovision of healthcare and medical
assistance, will sufficient andtransparently reported processes
be implemented throughout the
project’s lifecycle to ensure thatthe system protects the rights tothe protection of health and tosocial and medical assistance as
set out in Articles 11 and 13 of theEuropean Social Charter?

174

YES, WE HAVE PLANS IN PLACE TO DO THIS: No
messageWE HAD NOT CONSIDERED IT, BUT MAY DO THIS
/ WE HAD NOT CONSIDERED IT, BUT AREUNLIKELY TO DO THIS / NO, WE ARE NOTPLANNING TO DO THIS:
Major modifiable risk factor•

Where AI systems are being used toautomate
decisionsregardingtheprovision of healthcare and medical
assistance and project lifecycles do notincorporatetransparently
reportedYES, WE HAVE PLANS IN PLACETO DO THIS

processes to ensure that the system
protects the rights to the protection ofhealth and to social and medical
assistance as set out in Articles 11 and 13of the European Social Charter, thispresents a major modifiable risk factor
for adverse impacts on the human rights
and fundamental freedom of persons.

WE HAD NOT CONSIDERED IT,
BUT MAY DO THISWE HAD NOT CONSIDERED IT,
BUT ARE UNLIKELY TO DO THIS
NO, WE ARE NOT PLANNING TO
DO THISUNSURE

Actions to take for your HUDERIA:•NOT APPLICABLE

ExampleMake sure to pay close attention, in your
HUDERIA, to assessing the impacts of thesystem on social and economic rights,
equality, dignity, and the right to physical,
psychological, or moral integrity.

Goals, properties, and areas to focus on inyour HUDERAC:•

Fairness (equality, non-discrimination,
bias mitigation, diversity, andinclusiveness) and sustainability(reflection on context and impacts andstakeholder engagement andinvolvement)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
how to incorporate transparently reported
processes to ensure that the system protects theright to the protection of health and to social andmedical assistance as set out in Articles 11 and
13 of the European Social Charter. When this175

information is ascertained, you should return toyour PCRA and revise it accordingly.
NOT APPLICABLE: No message

Section 2: Risks of adverse impacts on the human rights and fundamental freedom of
persons, democracy, and the rule of law
In the following section, you will be asked to provide a preliminary estimation of the likelihood andgravity potential of the adverse impacts that your system could have on human rights, democracy,
and the rule of law. Draw on the stakeholder analysis and the determination of salient rights you
carried out as part of your initial PS reporting to think through the relevance, likelihood, andgravity potential of the harms indicated in the prompts. Each prompt will be formed as a statementwith likelihood/probability options below.

Principles/Priorities

Statement

Respect for and protection of humandignity

67)

Trigger messages for summary
reportThe AI system could interact
with rights-holders in ways thatcreate confusion or uncertainty
about whether they are interactingwith a computational technology.
UNLIKELYPOSSIBLE
LIKELY176

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
will interact with rights-holders in ways that could
create confusion or uncertainty about whether
they are interacting with a computational
technology. When this information is ascertained,you should return to your PCRA and revise it
accordingly.

VERY LIKELYUNSURE
N/A

Example68) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM

17769) The AI system could interact withrights-holders (users or decisionsubjects) in ways that expose
them to humiliation (being put in astate of helplessness orinsignificance; being dehumanized
or losing a sense of individual
identity).
UNLIKELYUNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
could interact with rights-holders (users or
decision subjects) in ways that expose them tohumiliation (being put in a state of helplessness
or insignificance; being dehumanized or losing a
sense of individual identity). When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.

POSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

Example17870) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM

71) The AI system could interact withrights-holders (users or decisionsubjects) in ways that expose
them to instrumentalization or
objectification (treating an
individual solely as exchangeable,
as a statistical aggregate, as ameans to an end, or as an object
to be freely manipulated or
steered).
UNLIKELYPOSSIBLE
LIKELY179

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
could interact with rights-holders (users or
decision subjects) in ways that expose them to
instrumentalization or objectification (treating an
individual solely as exchangeable, as a statisticalaggregate, as a means to an end, or as an object
to be freely manipulated or steered). When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.

VERY LIKELYUNSURE
N/A

Example72) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM

73) The AI system could interact withrights-holders (users or decisionsubjects) in ways that expose

180

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)them to displacement, redundancy,
or a sense of worthlessness inregard to their participation inwork life, creative life, or the life of
the community.
UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

Example74) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).
CATASTROPHIC HARMCRITICAL HARM

181

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
could interact with rights-holders (users or
decision subjects) in ways that expose them todisplacement, redundancy, or a sense of
worthlessness in regard to their participation inwork life, creative life, or the life of thecommunity. When this information is ascertained,
you should return to your PCRA and revise it
accordingly.

SERIOUS HARM
MODERATE OR MINOR HARM

Protection
autonomy

of

human

freedomand75) The AI system could interact withrights-holders in ways thatadversely affect or hinder theirabilities to make free, independent,
and well-informed decisions about
their lives or about the system’s
outputs.
UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

Example182

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
Unsure: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
will interact with rights-holders in ways that could
adversely affect or hinder their abilities to makefree, independent, and well-informed decisions
about their lives or about the system’s outputs.
When this information is ascertained, you should
return to your PCRA and revise it accordingly.

76) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM

77) The AI system could interact withrights-holders in ways thatadversely affect or hinder theircapacities to flourish, to fully
develop themselves, and to pursue
their own freely determined life
plans.
UNLIKELYPOSSIBLE
LIKELYVERY LIKELY183

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
will interact with rights-holders in ways that could
adversely affect or hinder their capacities toflourish, to fully develop themselves, and to
pursue their own freely determined life plans.
When this information is ascertained, you should
return to your PCRA and revise it accordingly.

UNSURE
N/A

Example78) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM

79) The deployment of the system
could result in the arbitrary
deprivation of rights-holders'
physical freedom or personal
security, or the denial of theirfreedoms of expression, thought,
conscience, or assembly.

184

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,whether the AI system you are planning to build
will interact with rights-holders in ways that could
result in the arbitrary deprivation of rightsholders' physical freedom or personal security, or
the denial of their freedoms of expression,
thought, conscience, or assembly. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.

UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

Example80) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM

185

Prevention of harm and protection oftheright tolife andphysical,
psychological, and moral integrity81) The AI system could interact withrights-holders in ways that deprive
them of their right to life or their
physical, psychological, or moral
integrity.
UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

Example82) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).
CATASTROPHIC HARMCRITICAL HARM186

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
will interact with rights-holders in ways that could
deprive them of their right to life or their physical,
psychological, or moral integrity. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.

SERIOUS HARM
MODERATE OR MINOR HARM

83) The AI system could interact withthe environment (either in theprocesses of its production or in its
deployment) in ways that harm thebiosphere or adversely impact thehealth of the planet.
UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

Example187

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
will interact with the environment (either in theprocesses of its production or in its deployment)
in ways that could harm the biosphere or
adversely impact the health of the planet. When
this information is ascertained, you should return
to your PCRA and revise it accordingly.

84) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM
Non-discrimination,equality

fairness,and85) The AI system (either in theprocesses of its production or in its
deployment) could result in
discrimination, have discriminatory
effects on impacted rights-holders,or perform differentially for
different groups in discriminatoryor harmful ways—including forintersectional groups where
vulnerable or protectedcharacteristics converge.
UNLIKELYPOSSIBLE
LIKELY188

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI (either in the processes of its
production or in its deployment) could result in
discrimination, have discriminatory effects on
impacted rights-holders, or perform differentiallyfor different groups in discriminatory or harmful
ways—including for intersectional groups where
vulnerable or protected characteristics converge.
When this information is ascertained, you should
return to your PCRA and revise it accordingly.

VERY LIKELYUNSURE
N/A

Example86) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM

87) The use of the AI system could
expand existing inequalities in thecommunities it affects or augment
historical patterns of inequity and

189

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)

discrimination in thesecommunities.
UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

Example88) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM

190

UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
could expand existing inequalities in thecommunities they affect or augment historicalpatterns of inequity and discrimination in thesecommunities.
When
this
informationisascertained, you should return to your PCRA andrevise it accordingly.

MODERATE OR MINOR HARM
Data protection and the right to respect
for private and family life89) The design, development, anddeployment of the AI system could
harm the rights to data protection
enshrined in data protection andprivacy law and in the Council ofEurope's Convention 108+.
UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

Example90) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).

191

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whetherthedesign,development,
anddeployment of the AI system could harm the rightto data protection enshrined in data protection
and privacy law and in the Council of Europe's
Convention 108+. When this information is
ascertained, you should return to your PCRA andrevise it accordingly.

CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM
91) The AI system could intrude on orinterfere with the private andfamily life of rights-holders in waysthat prevent or impede them frommaintaining a personal sphere thatis independent from thetransformative effects of AI
technologies and in which they are
at liberty to freely think, form
opinions and beliefs, and develop
their personal identities andintimate relationships without the
influence of AI technologies.
UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

192

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
could intrude on or interfere with the private andfamily life of rights-holders in ways that preventor impede them from maintaining a personal
spherethatisindependentfromthetransformative effects of AI technologies and inwhich they are at liberty to freely think, form
opinions and beliefs, and develop their personal
identities and intimate relationships without the
influence of AI technologies. When thisinformation is ascertained, you should return toyour PCRA and revise it accordingly.

Example92) If you responded with any of theoptions besides N/A or Unsure in
the previous question, how would
you characterize the gravity
potential of the harm? (For details
on gravity potential, please refer tothe relevant section of the user’s
guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM

Social and economic rights93) The deployment of the AI system
could harm the social and economic
rights of affected persons, including
the right to just working conditions,the right to safe and healthy working
conditions, the right to organize, theright to social security, and the rights
to the protection of health and tosocial and medical assistance as set
out in the European Social Charter.
UNLIKELYPOSSIBLE

193

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the AI system you are planning to build
could harm the social and economic rights ofaffected persons, including the right to justworking conditions, the right to safe and healthy
working conditions, the right to organize, theright to social security, and the rights to theprotection of health and to social and medical
assistance as set out in the European Social

LIKELY

Charter. When this information is ascertained,
you should return to your PCRA and revise it
accordingly.

VERY LIKELYUNSURE
N/A

Example94) If you responded with any of theoptions besides N/A or Unsure in theprevious question, how would you
characterize the gravity potential of
the harm? (For details on gravity
potential, please refer to the relevant
section of the user’s guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM
Democracy95) The use or misuse of the AI
system could lead to interference withfree and fair election processes or withthe ability of impacted individuals toparticipate freely, fairly, and fully inthe political life of the community
through any of the following:

194

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the use or misuse of the AI system could
lead to interference with free and fair election

a. Mass deception, at local,
national, or global levels
caused by the deploymentof the system
b. Mass manipulation, at local,
national, or global levels
enabled by the deploymentof the system
c. Mass intimidation or
behavioural control, at
local, national, or global
levels enabled by the
deployment of the system
d. Mass personalized politicaltargeting or profiling, at
local, national, or global
levels enabled by the
deployment of the system
UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

Example195

processes or with the ability of impacted
individuals to participate freely, fairly, and fully inthe political life of the community (through mass
deceptions,mass
manipulation,
mass
intimidation or behavioural control, or mass
personalized political targeting or profiling).
When this information is ascertained, you should
return to your PCRA and revise it accordingly.

96) If you responded with any of theoptions besides N/A or Unsure in theprevious question, how would you
characterize the gravity potential of
the harm? (For details on gravity
potential, please refer to the relevant
section of the user’s guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM
97) The use or misuse of the AI
system could lead to the massdispersal, at local, national, or global
levels, of misinformation or
disinformation.
UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

196

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the use or misuse of the AI system could
lead to the mass dispersal, at local, national, or
global levels, of misinformation or disinformation.
When this information is ascertained, you should
return to your PCRA and revise it accordingly.

Example98) If you responded with any of theoptions besides N/A or Unsure in theprevious question, how would you
characterize the gravity potential of
the harm? (For details on gravity
potential, please refer to the relevant
section of the user’s guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM
99) The use or misuse of the AI
system could lead to obstruction of
informational plurality, at local,
national, or global levels.
UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE197

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the use or misuse of the AI system could
lead to obstruction of informational plurality, at
local, national, or global levels. If so, the system
is considered to pose significant risks to thephysical, psychological, or moral integrity or thehuman rights and fundamental freedoms ofaffected persons.
When this information isN/Aascertained, you should return to your PCRA andrevise it accordingly.

Example100) If you responded with any of theoptions besides N/A or Unsure in theprevious question, how would you
characterize the gravity potential of
the harm? (For details on gravity
potential, please refer to the relevant
section of the user’s guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM
101) The use or misuse of the AI
system (in particular, information
filtering models such as recommendersystems, search engines, or news
aggregators) could lead to an
obstruction of the free and equitableflow of the legitimate and valid forms
of information that are necessary forthe meaningful democratic
participation of impacted rights
holders and for their ability to engage
freely, fairly, and fully in collective
problem-solving.

198

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the use or misuse of the AI system could
lead to an obstruction of the free and equitableflow of the legitimate and valid forms of
information that are necessary for the meaningfuldemocratic participation of impacted rights
holders and for their ability to engage freely,fairly, and fully in collective problem-solving.
When this information is ascertained, you should
return to your PCRA and revise it accordingly.

UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

Example102) If you responded with any of theoptions besides N/A or Unsure in theprevious question, how would you
characterize the gravity potential of
the harm? (For details on gravity
potential, please refer to the relevant
section of the user’s guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM

199103) The use or misuse of the AI
system (in particular, models thattrack, identify, de-anonymize or
surveil rights-holders and socialgroups or enable the creation of social
graphs) could lead to interference withor obstruction of impacted rightsholders' abilities to exercise theirfreedoms of expression, assembly, or
association.
UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

Example104) If you responded with any of theoptions besides N/A or Unsure in theprevious question, how would you
characterize the gravity potential of
the harm? (For details on gravity
potential, please refer to the relevant
section of the user’s guide).

200

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the use or misuse of the AI system (inparticular, models that track, identify, deanonymize or surveil rights-holders and socialgroups or enable the creation of social graphs)
could lead to interference with or obstruction of
impacted rights-holders' abilities to exercise theirfreedoms of expression, assembly, or association.
When this information is ascertained, you should
return to your PCRA and revise it accordingly.

CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM
Rule of Law105) The deployment of the AI system
could harm impacted individuals' rightto effective remedy or right to a fairtrial (equality of arms, right to anatural judge established by law, theright to an independent and impartial
tribunal, and respect for theadversarial process).
UNLIKELYPOSSIBLE
LIKELYVERY LIKELYUNSURE
N/A

Example106) If you responded with any of theoptions besides N/A or Unsure in theprevious question, how would you

201

UNLIKELY, POSSIBLE, LIKELY, VERY LIKELY:(Probability answers to be tabulated in the risk
index table)
UNSURE: Before taking any further steps in the
proposed project, you should determine, throughexpert and stakeholder input where appropriate,
whether the deployment of the AI system could
harm impacted individuals' right to a fair trial(equality of arms, right to a natural judge
established by law, the right to an independentand impartial tribunal, and respect for theadversarial process). When this information is
ascertained, you should return to your PCRA andrevise it accordingly.

characterize the gravity potential of
the harm? (For details on gravity
potential, please refer to the relevant
section of the user’s guide).
CATASTROPHIC HARMCRITICAL HARMSERIOUS HARM
MODERATE OR MINOR HARM

202

PCRA Output Example

Human Rights, Democracy, andthe Rule of Law Preliminary
Context-Based Risk Assessment
Report203

Thank you for completing the Human Rights, Democracy, and Rule of Law
Preliminary Context-Based Risk Analysis (PCRA). This report has been generated
based on your responses.
The report is divided into two sections corresponding to the two sections in the
PCRA.
Responses from Section 1 are divided into three primary sections: Prohibitive
Risks Factors, Major Risks Factors, and Moderate Risk Factors. Each of these
sections detail best next steps and items to focus on in your impact assessment
process (HUDERIA), as well as in your risk management and assurance practices(HUDERAC).
Section 2 provides a table that compiles the provisional risk index numbers (RINs)
that have been calculated based upon your answers in the corresponding sectionof the PCRA. Each RIN falls with a numeric interval that establish a risk level forthe potential adverse impact on the human rights and fundamental freedom of
persons, democracy, and the rule of law. Under the table you will findrecommendations for risk management actions for each potential harm
identified as well as general recommendations for proportionate risk
management practices and stakeholder engagement.
The results of the PCRA are meant to be part of an iterative process which can be
returned to and revisiting as you process with the other stages of the HUDERAF
process.

SECTION 1: IDENTIFYING RISK FACTORS
Section 1 produces a summary which compiles and organizes the answers thatflagged up risk factors into the three risk factor categories: prohibitive, major, andmoderate. Additionally, these three risk factor categories are further divided into
modifiable or circumstantial. Those risk factors that emerge externally from thetechnical, sociotechnical historical, legal, economic, or political environments
within which the design, development, and deployment of AI systems occur andthat are thereby less controllable are called “circumstantial risk factors.” Thoserisk factors that emerge internally from the actual practices of producing and using
AI technologies, and that are thus more controllable, are called “modifiable risk
factors”.
The report provides prompts for each of these answers that direct you to specificactions to take in your impact assessment process and to specific goals,
properties, and areas that you should focus on in your subsequent risk
management and assurance processes to reduce and mitigate associated risks.

Prohibitive Risk Factors:
204

Stop to evaluate the items in your assessment that have indicated thepresence of prohibitive risk factors. These items must be prioritised andresponded to before you proceed with any other project tasks oractivities.

Because you answered No to Question 2:
STOP. Prohibitive circumstantial risk factor
If you have not been able to establish a legal basis for the AI system, you should
not proceed. Where appropriate, you should consult with experts to establish thelawfulness of your processing objective and proposed project before taking any
next steps.

Because you answered Unsure to Question 6:
STOP. Before taking any further steps in the proposed project, you should
determine, through expert and stakeholder input where appropriate, whether the
AI system you are planning to build is on the list of prohibited systems.

Major Risk Factors:
Because you answered Yes to Question 1:
Major circumstantial risk factor:
Where AI systems serve primary or critical functions in high impact or safety
critical sectors, this presents a major circumstantial risk factor for adverse
impacts on the human rights and fundamental freedom of persons.
• Actions to take for your HUDERIA: Make sure to focus upon
considerations surrounding the prevention of harm and the respect of theright to life and to physical, psychological, and moral integrity• Goals, properties, and areas to focus on in your HUDERAC: Safety(accuracy and system performance, security, reliability, and robustness),sustainability (reflection on context and impacts, change monitoring)205

Because you answered Yes to Question 5:
Major circumstantial risk factor
Where AI systems perform a safety critical or high impact function independent of
the sector in which they operate, this presents a major circumstantial risk
factor for adverse impacts on the human rights and fundamental freedom of
persons.
••

Actions to take for your HUDERIA: Make sure to focus upon
considerations surrounding the prevention of harm and the respect of theright to life and to physical, psychological, and moral integrity
Goals, properties, and areas to focus on in your HUDERAC: Safety(accuracy and system performance, security, reliability, and robustness)and sustainability (reflect on contexts and impact, change monitoring)

Because you answered Unsure to Question 7:
Before taking any further steps in the proposed project, you should determine,through expert and stakeholder input where appropriate, whether the AI system
could be used or repurposed in ways that fall under the list of prohibited systems
which are considered to pose significant risks. When this information is
ascertained, you should return to your PCRA and revise it accordingly.

Moderate Risk Factors:
Because you answered Yes to Question 3:
Moderate circumstantial risk factor
Where the sector or domain in which the AI system will operate is historicallyhighly regulated, this presents a moderate circumstantial risk factor foradverse impacts on the human rights and fundamental freedom of persons.
••Actions to take for your HUDERIA: Make sure to consider the
regulatory context of your project and to reflect on the expectations ofcompliant and reasonable practices that arise in that context.
Goals, properties, and areas to focus on in your HUDERAC:
Accountability and process transparency, (traceability, accessibility,
auditability, and responsible governance)

Because you answered Unsure to Question 4:
Before taking any further steps in the proposed project, you should determine,through expert and stakeholder input where appropriate, whether any other typesof impact assessment for the specific use-case of the AI systems you are planningto develop are required by law or regulation. When this information is ascertained,
you should return to your PCRA and revise it accordingly.

206

SECTION 2: RISKS OF ADVERSE IMPACTS ON
THE HUMAN RIGHTS AND FUNDAMENTAL
FREEDOM OF PERSONS, DEMOCRACY, AND
THE RULE OF LAW
Section 2 provides a table that compiles the provisional risk index numbers (RINs)
that have been calculated based upon your answers in the corresponding sectionof the PCRA. As you were asked to provide a preliminary estimation of thelikelihood of the potential adverse impacts that your system could have on humanrights, democracy, and the rule of law, this estimation of likelihood is incorporated
into the table and risk index calculation along with your responses to otherquestions such as the number of rights-holders affected.
The likelihood score that is indicated in the far-right column indicates a risk index
number (RIN). An RIN has been calculated for each of the questions in Section 2that were answered with likelihood estimates, but not for answers of “Unsure” or“Not Applicable”.
RINs are calculated by adding the Severity Level (Gravity Potential + Number ofRights-Holders Affected) and Likelihood Level for each potential harm. Details
about this calculation and its rationale can be found in the accompanying user’s
guide. Across all possible answers, RINs range from 2.5 to 10. Risk levels are
established as intervals within this range:

Low Risk

Moderate Risk

𝑅𝑅𝑅𝑅𝑅𝑅 ≤ 5

5.5 ≤ 𝑅𝑅𝑅𝑅𝑅𝑅 ≤ 6High Risk
6.5 ≤ 𝑅𝑅𝑅𝑅𝑅𝑅 ≤ 7.5

Very High Risk
𝑅𝑅𝑅𝑅𝑅𝑅 ≥ 8

These RINs correspond to preliminary recommendations for proportionate risk
management and assurance practices and stakeholder engagement given the risk
level of the prospective AI system. These recommendations can be found in thetables below the summary table generated from your responses to Section 2.
Risk management actions include recommendations on how to reduce the levelsof risk present through a variety of methods such as expert and stakeholder
consultations. These recommendations can be applied to your results by matching
the RIN to the corresponding risk management recommendations.
Additional recommendations for establishing sufficient public transparency andaccountability—as well as for ensuring adequate human rights diligence—are
included for large-scale projects that could have macro-scale, long-term impacts
on individuals and society. The second table includes recommendations for
proportionate risk management practices and stakeholder engagement that canbe applied to your results.

207Section 2 Output Table
Gravity
Potential

RightsHolders
Affected

Severity

Likelihood

Risk
Index
Number(RIN)

Anthropomorphic confusion over interaction with acomputational system (dignity)

3

1

4

2

6

Moderate

Exposure to humiliation (dignity)

3

1

4

1

5

Low

Deprivation of rights-holders’ abilities to make free,independent, and well-informed decisions about their lives
or about the system’s outputs, including the ability ofrights-holders to effectively challenge decisions informed
and/or made by that system and to demand that suchdecision be reviewed by a person (human freedom)

3

1

4

1

5

Low

Deprivation of the right to life or physical, psychological,
or moral integrity (Prevention of harm)

4

1

5

2

7

High

Discrimination, discriminatory effects on impacted rightsholders, or differential performance (Non-discrimination)

3

1

4

2

6

Moderate

Adverse Impacts:

208

Risk Level

See below a table of specific risk management actions for each of thesepotential impacts identified by your results:

For any 𝑹𝑹𝑹𝑹𝑹𝑹 ≥ 𝟖𝟖 (veryhigh)

Further examination should be undertaken, throughexpert and stakeholder consultation, as to whether
sufficient risk reduction is possible to make very highrisks of harm tolerable or whether this is not feasible,
and the risks are unacceptable.

For any 𝟔𝟔. 𝟓𝟓 ≤ 𝑹𝑹𝑹𝑹𝑹𝑹 ≤𝟕𝟕. 𝟓𝟓 (high)Further examination should be undertaken, throughexpert and stakeholder consultation, as to whether
the risks of harm indicated to be high are tolerable
and can be appropriately reduced. Where likelihood= 1 (unlikely) and RPN indicates high risk,
confirmation of low risk probability should also be
undertaken through expert and stakeholder
consultation.

For any 𝟓𝟓. 𝟓𝟓 ≤ 𝑹𝑹𝑹𝑹𝑹𝑹 ≤ 𝟔𝟔
(moderate)

Further examination should be undertaken, throughexpert and stakeholder consultation, as to whether
the risks of harm indicated to be moderate arebroadly accepted as such, are tolerable, and can beappropriately reduced. Where likelihood = 1(unlikely) and RPN indicates moderate risk,
confirmation of low risk probability should also be
undertaken through expert and stakeholder
consultation.

For any 𝑹𝑹𝑹𝑹𝑹𝑹 ≤ 𝟓𝟓 (low)Further examination should be undertaken, throughexpert and stakeholder consultation, as to whether
the risks of harm indicated to be low are broadly
accepted as such.

See below for general recommendations for proportionate risk management
practices and stakeholder engagement. These recommendations are based on aprioritization of a degree of risk management and human right diligence that is
proportionate to the highest level of risk identified across an AI system’s potential
impacts.
Because the highest
risk level in yourresults is 𝟔𝟔. 𝟓𝟓 ≤ 𝑹𝑹𝑹𝑹𝑹𝑹 ≤𝟕𝟕. 𝟓𝟓 (high)Full diligence in risk management and assurance
practices is recommended to prioritize risk reduction
and mitigation. This should be informed by the risk
factors identified in section 1 of the PCRA and by the
HUDERIA
impactassessment
process;
Comprehensive stakeholder engagement across the
project
lifecycle(e.g.,
partnering
withor
empowering rights-holders as determined by the209Stakeholder
Engagement
recommended.

210

Process)isalsoStakeholder Engagement Process

Stakeholder engagement is a core component of any human rights diligenceprocess. It is not a one-off activity, but rather should be seen as an ongoing setof activities that occur throughout the project lifecycle. 44 Stakeholder engagement
may take different forms at different points in a project. It is essential to ensurethat rights-holders’ views are incorporated at all stages and that any potential
risks or adverse impacts are identified and mitigated. 45 In this sense, an iterative
approach to stakeholder engagement is essential. 46 In particular, it is importantto revisit and revise your stakeholder analysis to ensure that approaches taken
continue to reflect the perspectives and interests of salient stakeholders, including
vulnerable and underrepresented rights-holders.
A Stakeholder Engagement Process (SEP) involves 5 key activities:

5 Key Activities within the Stakeholder
Engagement Process:
•
•
•
•
•

Stakeholder Analysis
Positionality Reflection
Establishing Engagement Objectives
Determining Engagement Method
Workflow Revisitation and Reporting

44 See Esteves, A. M., Factor, G., Vanclay, F., Götzmann, N., & Moreira, S. (2017).
Adapting social impact assessment to address a project's human rights impacts andrisks. Environmental Impact Assessment Review, 67, p. 77.
45 See Figure 1 in Götzmann, N., Bansal, T., Wrzoncki, E., Veiberg, C. B., Tedaldi, J., &Høvsgaard, R. (2020). Human rights impact assessment guidance and toolbox. The Danish
Institute for Human Rights. p. 7.
46 See page 77 in (Esteves, et al 2017).

211

The initial four steps of the SEP take place internally (i.e. within the organisationor team undertaking the impact assessment) during the design phase of the
project workflow. Throughout these steps, teams are to draw on the PCRA report,
updated PS Report, and further desk research to answer series of questions andprompts. Step 5 of this process (Workflow Revisitation and Reporting) is
undergone by conducting stakeholder engagement activities and providing project
reports. This step facilitates stakeholder participation and input to revise
approaches towards subsequent actions in the workflow. Note that, following
its initial iteration as Step 5 of the SEP, the Workflow Revisiting andReporting activity is repeated throughout the design, development, anddeployment phases of your project to ensure ongoing stakeholder
involvement across the workflow.
While stakeholder engagement is vital throughout the process, this can takevarious forms (outlined below) and involve varying degrees of engagement that
fit proportionality recommendations. In accordance with best practices in humanrights due diligence, you should always aim for the greatest depth of engagementthat is feasible within the scope and resource constraints of the project.

Step 1: Carry Out Stakeholder Analysis

Stakeholder Salience Analysis is the process of evaluating the salience of identified
stakeholders. It aims to help you understand the relevance of each identified
stakeholder to your project and to its use contexts. It does this by providing a
structured way for your team members to assess the relative interests, rights,
vulnerabilities, and advantages of identified stakeholders as these interests,
rights, vulnerabilities, and advantages may be impacted by, or may impact, themodel or tool your team is planning to develop and deploy.

Analysing Stakeholder Salience:
Use the PCRA, the updated Project Summary Report, and further desk research(if necessary) to inform answers to the following questions, assessing relative
stakeholder salience:
What stakeholder groups have existing interests in relation to thedomain or sector in which the system/tool will be deployed?
• What stakeholder groups are most likely to be impacted by the
deployment of the system or tool?
• What stakeholder groups have the greatest needs in relation to potential
benefits/applications of the system/tool or the domain or sector in which
it will be deployed?
• What stakeholder groups are most and least powerful?
• What stakeholder groups have existing influence within relevant
communities, political processes, or in relation to the domain in which
the system will be deployed?
• the
What
stakeholder
groups’ influence is limited?
Considerfollowing
diagram:•• What stakeholder groups’ rights may be impacted in relation to thedomain or sector in which the system/tool will be deployed?
•212Consider the following diagram:•

What stakeholder groups possess both protected and contextuallyvulnerable characteristics?

Contextual
Vulnerability
Characteristics

Protected
Characteristics•Do they possesscharacteristics thatmight increase theirvulnerability to abuseor discrimination, or
which may require
additional assistance?
If so, what
chatacteristics?

•Could the outcomesof this project present
significant concerns
given their specificcircumstances? If so,
what characteristics
expose them to being
jeopartised by project
outcomes?

Analysis of stakeholder salience aims to identify which stakeholders are likely to
be most impacted, vulnerable, and those that currently have least influence. 47
Special attention needs to be paid to engaging these stakeholders in subsequent
stages.

47 See page 124 and 125 in (Götzmann et al 2020)

213

Step 2: Engage in Positionality Reflection

When taking positionality into account, each of your team members should reflect
on their own positionality matrix by answers the questions contained in the graphic
below.

Reflecting on positionality:
After reflecting on their positionality matrices, team members shouldcollaboratively answer the following questions:a. How does the positionality of team members relate to that of affectedrights-holders and other stakeholders?
b. Are there ways that your position as a team could lead you to choose
one option over another when assessing the risks that the prospective
AI system poses to human rights, democracy and the rule of law?
c. What (if any) missing stakeholder viewpoints would strengthen yourteam’s assessment of this system’s potential impact on human rights
and fundamental freedoms?
214Step 3: Establish an Engagement Objective

Before selecting methods of engagement to use, you must first define objectivesof your stakeholder engagement. Through this process you should articulate:•••

Why you are engaging stakeholders
The purpose and expected outcomes of engagement activities
The level of influence stakeholders will have on engagement processes andoutcomes

There are many ways a project team can engage with the stakeholders they haveidentified, and the objectives and methods of engagement that your team chooses
will depend on several factors. These divide into three categories, which arepresented here with accompanying descriptions:

Factors determining the objectives of engagement
Team-based
assessments of risks

Team-based
assessments ofpositionality

Establishment of
stakeholder
engagement goals•

Assessment of how stakeholder involvement should be madeproportionate to the scope of a project’s potential risks andhazards to human rights, democracy, and the rule of law•

Evaluation of team positionality—for instance, cases where
the identity characteristics of team members do not
sufficiently reflect or represent significantly impacted groups.
How can the project team “fill the gaps” in knowledge, domain
expertise, and lived experience through stakeholder
participation?

•

Determination of engagement objectives that enable theappropriate degree of stakeholder engagement and coproduction in project evaluation and oversight processes•

Choosing participation goals from a spectrum engagement
options (informing, partnering, consulting, empowering) that
equip your project with a level of engagement which meets
team-based assessments of risk and positionality.

When weighing these three factors, your team should prioritise the establishmentof a clear and explicit stakeholder participation goal and document this. Thisis crucial, because all stakeholder engagement processes can run the risk eitherof being cosmetic tools employed to legitimate projects without substantial andmeaningful participation or of being insufficiently participative, i.e. of being oneway information flows or nudging exercises that serve as public relations
instruments. The purpose of stakeholder involvement in human rights diligence isjust the opposite: to amplify the participatory agency of affected individuals andorganisations in impact assessment, risk management, and assurance processes.
To avoid such hazards of superficiality, your team should shore up its
proportionate approach to stakeholder engagement with deliberate and precise
goal-setting. There are a range of engagement options that can help your project
obtain a level of citizen participation which meets team-based assessments ofimpact and positionality as well as practical considerations and stakeholder needs:215

Levels of Stakeholder Engagement

Questions:•••

Why are you engaging stakeholders?
What are the expected outcomes of engagement activities?
How will stakeholders be able to influence the engagement process andthe outcomes?

Step 4: Determine Engagement Method
Once you have established your engagement objective, you are in a better position
to assess which method or methods of stakeholder involvement are mostappropriate to achieve your goals. Determining appropriate engagement methods
necessitates that you (1) evaluate and accommodate of stakeholder needs, and(2) pay attention to practical considerations of resources, capacities, timeframes,
and logistics that could enable or constrain the realisation of your objective:

Factors determining engagement methods

216

Evaluation andaccommodation of
stakeholder needs••

Practical
considerations ofresources,
capacities,
timeframes, andlogistics••••

Identification of potential barriers to engagement such as
constraints on the capacity of vulnerable stakeholder groups toparticipate, difficulties in reaching marginalised, isolated, or
socially excluded groups, and challenges to participation thatare presented by digital divides or information andcommunication gaps between public sector organisations andimpacted communities
Identification of strategies to accommodate stakeholder needs
such as catering the location or media of engagement todifficult-to-reach groups, providing childcare, compensation,
or transport to secure equitable participation, tailoring theprovision of information and educational materials to theneeds of participants
the resources available for facilitating engagement activities
the timeframes set for project completionthe capacities of your organisation and team to properly
facilitate public engagement
the stages of project design, development, and implementation
at which stakeholders will be engaged

You and your team may face pitfalls when confronting any of these factors. For
example, limits on available resources and tight timelines could be at crosspurposes with the degree of stakeholder involvement that is recommended byteam-based assessments of potential hazards and positionality limitations.
Likewise, the chosen degree of appropriate citizen participation may be unrealisticor out-of-reach given the engagement barriers that arise from high levels of
stakeholder needs. In these instances, you should take a deliberate and reflective
approach to deciding on how to balance engagement goals with practicalconsiderations—always prioritizing the need for optimizing stakeholder input andrights-holder involvement in human rights diligence processes. And you shouldmake explicit the rationale behind your choices and document this.
Reflecting the varied levels of engagement set out above, there are varied
engagement methods which can be used. Stakeholder engagement is likely torequire more than a single method and may combine approaches aimed at
informing, consulting, partnering and empowering. However, stakeholder
engagement must always contain at least one method pertaining to the statedengagement objective (i.e. a method pertaining to a lower level of engagement
cannot account for the need for engagement aimed to be met through a higher
engagement objective):

Examples of Relevant Engagement Methods
Mode ofEngagement
Newsletters (email)Degree ofEngagement

Practical Strengths

InformCan reach many
people; can contain217

Practical
Weaknesses
Might not reach certain
portions of theRegular emails (e.g.:
fortnightly ormonthly) that contain
updates, relevant
news, and calls toaction in an inviting
format.

large amount ofrelevant information;
can be made accessible
and visually engaging.

Letters (post)
Regular letters (e.g.
monthly) that containthe latest updates,
relevant news andcalls to action.

Posts in local
community (e.g. local
newspapers,
community
newsletters andnoticeboards).
App notifications
Projects can rely on
the design of appsthat are pitched to
stakeholders who arenotified on theirphone with relevant
updates.

Can reach parts of the
population with nointernet or digitalaccess; can containlarge amount ofrelevant information;
can be made accessible
and visually engaging.

Might not engage
certain portions of the
population; Slow
delivery andinteraction times
hamper the effective
flow of information andthe organisation offurther engagement.

InformCan reach localisedcommunities; can reachpeople who do not have
digital access.

Limited to geographic
region; requires
physical access; lesscost-effective thandigital means.

InformEasy and cost-effective
to distributeinformation to large
numbers of people;
rapid information flows
bolster the provision of
relevant and timely
news and updates.

More significant initialinvestment indeveloping an app; will
not be available topeople without
smartphones.

InformCommunity fora
Events in which
panels of experts
share their knowledge
on issues and thenstakeholders can ask
questions.

InformCan inform people withmore relevant
information by
providing them with theopportunity to ask
questions; brings
community together ina shared space of
public communication.

Online surveys
Survey sent via email,embedded in a
website, shared via
social media
Phone interviews
Structured or semistructured interviews
held over the phone.

population; can be
demanding to design
and produce with someperiodicity; easily
forwarded to
spam/junk folders
without project teamknowing (leading tooverinflated readershipstats).

Consult

Consult

Cost-effective; simple
mass-distribution.

Opportunity forstakeholders to voice
concerns more openly.

218

More time-consuming
and resource intensive
to organise; might
attract smaller
numbers of people andself-selecting groups
rather thanrepresentative subsetsof the population;effectiveness isconstrained by forum
capacity.
Risk of pre-emptive
evaluative framework
when designingquestions; does notreach those without
internet connection orcomputer/smartphone
access.
Risk of pre-emptive
evaluative framework
when designingquestions; might
exclude portions of thepopulation withoutphone access or with
habits of infrequentphone use.
Door-to-doorinterviews

Consult

Structured or semistructured interviews
held in-person at
people’s houses.

Feedback Forms
Provided along withletters or posts orafter events.

In-personinterviews

Consult

Suggested forConsult

Short interviews
conducted in-person
in public spaces.

Opportunity forstakeholders to voice
concerns more openly;
can allow participants
the opportunity to form
connections throughempathy and face-toface communication.
Provides mechanisms
for stakeholders to
respond to information
provided; can be quick
and easy forstakeholders to
respond; can include
fixed choice or openended responses.

Consult andPartnerCitizen panel orassembly
Large groups ofpeople (dozens oreven thousands) who
are representative ofa town/region.

Inform, Partner
and Empower

Might limit responses
to pre-definedquestions; risk of preemptive evaluative
framework when
designing questions.

Can reach many people
and a representative
subset of the
population if
stakeholders areappropriately definedand sortition is used.

Less targeted;
pertinent stakeholders
must be identified byarea; little
time/interest to
engage withinterviewer; can beviewed by interviewees
as time-consuming andburdensome.

Can gather in-depth
information; can lead to
new insights anddirections that were notanticipated by the
project team.

Subject to hazards ofgroup think or peerpressure; complex tofacilitate; can besteered by dynamics ofdifferential power
among participants.

Provides an opportunity
for co-production of
outputs; can produce
insights and directions
that were notanticipated by the
project team; canprovide an informationbase for conducting
further outreach(surveys, interviews,
focus groups, etc.); canbe broadly
representative; canbolster a community’s
sense of democratic
agency and solidarity.

Participant rolls must
be continuously
updated to ensure
panels or assemblies
remains representativeof the population
throughout theirlifespan; resourceintensive forestablishment andmaintenance; subject
to hazards of groupthink or peer pressure;complex to facilitate;
can be steered bydynamics of differential
power among
participants.

Focus group
A group of
stakeholders brought
together and asked
their opinions on aparticular issue. Can
be more or lessformally structured.

Potential for limited
interest to engage withinterviewers; timeconsuming; can beseen by interviewees
as intrusive or
burdensome.

219

Citizen jury
A small group ofpeople (between 12
and 24),
representative of thedemographics of agiven area, cometogether to deliberate
on an issue (generally
one clearly framed setof questions), overthe period of 2 to 7
days (involve.org.uk).

Inform, Partner
and Empower

Can gather in-depth
information; canproduce insights anddirections that were notanticipated by the
project team; canbolster participants’sense of democratic
agency and solidarity.
Most effective when
there is a clearly
defined question which
the jury is tasked withanswering.

Subject to hazards ofgroup think; complex
to facilitate; Requires
independentfacilitation; Risk of preemptive evaluative
framework; Small
sample of citizens
involved risks low
representativeness ofwider range of public
opinions and beliefs.

Questions:Define an engagement method to be used for the Workflow Revisitation andReporting. Consider:
•
•
•
•
•
•

What resources are available and what constraints will limit
potential approaches?
What accessibility requirements might stakeholders have?
Will online or in-person methods (or a combination of both) be
most appropriate to engage salient stakeholders?
Which activities meet your team’s engagement objective?
How will your team make sure that the chosen method
accommodates different types of stakeholders?
How will each of the chosen activities feed useful information toconsider in your stakeholder impact assessment?

Outputs from the first four steps of the Stakeholder Engagement Process are used
to create an initial Stakeholder Engagement Process Report, which is to be revisedand updated at each subsequent iteration of Workflow Revisitation and Reporting.

Step 5: Workflow Revisitation and Reporting

As indicated above, the Workflow Reporting and Revisitation step is conducted
initially as a culmination of the Stakeholder Engagement Process, so that you canwork directly with stakeholders to re-assess the first four steps of the SEP, and it
provides an opportunity for you to revisit the PS Report and the project workflow
with them. This initial stakeholder engagement will also flow into your impact
assessment process; working through the PS Report and the project workflow with220

these stakeholders will help to prepare them for meaningful and well-informed
involvement in your HUDERIA. The later iterations of Workflow Reporting andRevisitation (in the development and deployment phases of your project) willsubsequently enable your project team to report to stakeholders the actions andevidence that are being compiled in the project assurance case (HUDERAC) in anaccessible manner that facilitates feedback and input.
Revisiting with Stakeholders as part of your SEP
During Step 5 of the Stakeholder Engagement Process, it is essential to ensurethat decisions and outcomes from steps 1 – 4 reflect the experiences andperspectives of salient stakeholders. Stakeholder engagement should involve
rights-holders, duty bearers and other relevant parties. In instances where initialstakeholder mapping and risk analysis has identified no potential adverse impacts
on rights-holders, a minimum level of stakeholder engagement should involve
consultation with relevant organisations (e.g. civil society and human rights
organisations) and rights-holders.
Revisiting Steps 1 and 2 (Stakeholder Analysis and Positionality) with
Stakeholders
A useful strategy for revisiting Stakeholder Analysis and Positionality Reflection
with rights-holders and other relevant parties is to merge these activities into
engagement exercises that focus on identifying gaps and drawing on the lived
experience of affected individuals and communities. This will allow you to:
•
•
•

•

More fully involve initially-identified salient stakeholder groups
Broaden identified stakeholders through a “snowball” approach
Discuss team positionality with stakeholders (while being mindful ofconfidentiality needs and not disclosing sensitive or personal information
about individuals).
Map stakeholder salience through participatory processes which reflect
stakeholder perspectives and interests.

Crucially, a comprehensive stakeholder analysis requires initial stakeholder
engagement to identify potentially salient stakeholders who may not have beenvisible to the project team (particularly underrepresented rights-holders).
Interviews or community engagement activities can be an effective mechanism toidentify additional relevant stakeholders to engage. This “snowballing” technique
is an effective mechanism to ensure that stakeholder analysis reflects varied
perspectives and interests, recognising that there may not be consensus aroundwhich stakeholders are most salient.
Revisiting Stakeholder Salience Questions, Engagement Objective andEngagement Method with Stakeholders
This step is undertaken through further discussion of stakeholder needs withstakeholders themselves. Particular attention should be paid here to vulnerable
rights-holders. At this stage, the stakeholder engagement should focus on
identifying existing mechanisms and constraints for engagement (i.e., networks,
community groups, barriers to engagement, difficulties). This should include an221assessment of any potential risks of participation (i.e. if taking part in an
engagement activity could put vulnerable individuals at risk). Where there arepotential risks associated with participation of vulnerable individuals,consideration must be given to alternative engagement methods or forms ofrepresentation (e.g. through civil society organisations).
Outputs from the above activities are used to update the SEP Report, informing
approaches for the following engagement activity in the workflow (HUDERIA or
Workflow Revisitation).
Reporting to Stakeholders
Communication and stakeholder evaluation of process outcomes at each phase isa vital component to build and sustain trust and to underpin ongoing stakeholder
engagement. During the initial iteration of the stakeholder engagement process,you are to use the PS Report as a baseline to convey the project status in anaccessible manner that facilitates input. After this initial iteration, you are to use
the assurance case as a basis for updating the PS Report. The most up-to-dateversions of the assurance case and Project Summary Report (which provides asynopsis of up-to-date understandings of the system, its domain and use contexts,
identified stakeholders, the scope of impacts on human rights, democracy, andrule of law, and the project governance workflow), will enable you to share project
reports that are relevant at each phase of the workflow.
Reporting to stakeholders must involve:••

Publication and dissemination of project outputs (taking account ofaccessibility requirements of diverse stakeholders);
Facilitation of stakeholder feedback (e.g. through feedback forms,
questionnaires or further engagement)’ and incorporation of feedback
documented through an update to the SEP report, in turn informing thesubsequent HUDERIA and HUDERAC processes.
Questions:
•
•
•

•

What medium of reporting is likely to most effectively engage
stakeholders?
What are the accessibility requirements of stakeholders (e.g. taking
account of physical and cognitive disabilities)?
What mechanisms will be put in place to facilitate feedback fromstakeholders, and how will feedback be addressed and/or
incorporated? Consider how these mechanisms will yield feedback
pertaining tostakeholders’ validation of the system considering its domain and use
contexts; the identified scope of impacts to human rights,
democracy, and rule of law; and the project governance workflow.

222

Stakeholder Engagement Process (SEP) Template
Questions

Responses

STEP 1: Stakeholder Analysis
ANALYSYING STAKEHOLDER SALIENCE
Use the PCRA, the updated Project Summary Report, and further desk research (if necessary) to inform answers to the following
questions, assessing relative stakeholder salience. (Refer to the Venn diagram in the related section as needed.):What stakeholder groups have existing interests inrelation to the domain or sector in which thesystem/tool will be deployed?

What stakeholder groups are most likely to be
impacted by the deployment of the system or
tool?
What stakeholder groups have the greatest needs
in relation to potential benefits/applications of thesystem/tool or the domain or sector in which it will
be deployed?
What stakeholder groups are most and least
powerful?
What stakeholder groups have existing influence
within relevant communities, political processes, orin relation to the domain in which the system will
be deployed?

223

What stakeholder groups’ influence is limited?
What stakeholder groups’ rights may be impacted
in relation to the domain or sector in which thesystem/tool will be deployed?

STEP 2: Positionality Reflection
ENGAGE IN POSITIONALITY REFLECTION
When taking positionality into account, each of your team members should reflect on their own positionality by considering theirindividual responses to the questions posed in the Positionality Matrix provided. After reflecting on their positionality matrices,
team members should collaboratively answer the following questions:
How does the positionality of team members relate
to that of affected rights-holders and otherstakeholders?
Are there ways that your position as a team could
lead you to choose one option over another whenassessing the risks that the prospective AI systemposes to human rights, democracy, and the rule oflaw?
What (if any) missing stakeholder viewpoints would
strengthen your team’s assessment of this system’s
potential impact on human rights and fundamental
freedoms?

STEP 3: Engagement Objective
ESTABLISH AN ENGAGEMENT OBJECTIVE

224

Refer to the Factors Determining the Objectives of Engagement and Levels of Stakeholder Engagement tables as needed. Be sure
to outline a clear and explicit stakeholder participation goal.
Why are you engaging stakeholders?
What is the purpose and what are the expected
outcomes of engagement activities?
How will stakeholders be able to influence theengagement process and the outcomes?
What engagement objective do you believe would
be appropriate for this project considering
challenges or limitations to assessments related topositionality, and proportionality to the project’s
potential degree of impact?

STEP 4: Engagement Method
DETERMINE AN ENGAGEMENT METHOD
Define an engagement method to be used for the Workflow Revisitation and Reporting.
What resources are available and what constraints
will limit potential approaches?
What accessibility requirements might stakeholders
have?
Will online or in-person methods (or a combinationof both) be most appropriate to engage salient
stakeholders?

225

Which activities meet your team’s engagement
objective?
How will your team make sure that the chosen
methodaccommodates
different
types
of
stakeholders?
How will each of the chosen activities feed useful
information to consider in your stakeholder impact
assessment?

STEP 5: Workflow Revisitation and Reporting
REPORTING TO STAKEHOLDERS
Outline a stakeholder communication plan that includes receiving and incorporating feedback and reporting back to stakeholders
throughout the Stakeholder Engagement Process.
What medium of reporting is likely to mosteffectively engage stakeholders?
What are the accessibility requirements ofstakeholders (e.g. taking account of physical andcognitive disabilities)?
What mechanisms will be put in place to facilitate
feedback from stakeholders, and how will feedback
be addressed and/or incorporated? Consider how
these mechanisms will yield feedback pertaining tostakeholders’ validation of the system considering
its domain and use contexts; the identified scope of
impacts to human rights, democracy, and rule oflaw; and the project governance workflow.

226

Human Rights, Democracy, and the Rule of
Law Impact Assessment

Your Human Rights, Democracy, and the Rule of Law Impact Assessment(HUDERIA) provides you with the opportunity to draw on the learning and insights
you have gained in your PCRA and SEP processes, and on the lived experience ofengaged rights-holders, to delve more deeply into the potential impacts of yourproject on human rights, fundamental freedoms, and elements of democracy andthe rule of law. Your HUDERIA should enable you
•
•
•

•

•
•

To re-examine and re-evaluate the potential harms you have alreadyidentified in your PS Report and your PCRA
To contextualize and corroborate these potential harms in dialogue withstakeholders
To identify and analyse further potential harms by engaging in extended
reflection and by giving stakeholders the chance to uncover new harms thathave not yet been explored and to pinpoint gaps in the completeness andcomprehensiveness of the previously enumerated harms
To explore, with stakeholders, the severity (scope, scale, and remediability)of the potential adverse impacts, so that the risks of these can be better
assessed, prioritized, managed, and mitigatedTo begin formulating impact mitigation measures (avoid, restore,
remediate) based upon the severity of the identified harms
To set up access to remedy for affected rights-holders and other relevant
parties.

227

Getting the HUDERIA Process Right
Before you launch into building the content of your HUDERIA, it will be importantthat you establish a responsible process for carrying out the impact assessmentwith the participation of affected rights-holders and other relevant stakeholders.
To do this, you will need to ensure that your HUDERIA process aligns with five key
criteria: participation, non-discrimination, empowerment, transparency,
and accountability. 48 Use the table below (adapted from the important work of
the Danish Institute for Human Rights), to orient yourself to these criteria and to
the guiding principles that will have to be operationalised to secure a responsible
HUDERIA process:Key criteria and guiding principles
for responsible HUDERIA processes
Meaningful participation ofaffected or potentially affectedrights-holders is integrated
during all stages of the impact
assessment process, including
planning and scoping; datacollection and context analysis;
impact analysis; impact
prevention, mitigation andremediation; and reporting andevaluation.

Participation•

Engage a broad range of rightsholders including vulnerable andmarginalised groups•

Consider the rights, freedoms, andinvolvementofrights-holders
throughout
theAI
innovation
ecosystem (designers, implementers,
users, those potentially impacted by
the product or service, those who
could be displaced or deskilled by the
product or service)•

Involve rights-holders or their proxies
throughout the impact assessment
process•

Involve rights-holders, duty-bearers
and other relevant parties in designingmeasures to address impacts (e.g.
through prevention, mitigation andremediation)andfollow-up
toevaluate the effectiveness of thesemeasures•

Include rights-holder representatives
or representative organisations, orrights-holder proxies in consultation
andengagement,
includingconsideration of the legitimacy of theirclaim to represent the relevant
individuals and/or groups•

Ensurethatengagement
andparticipation in the impact assessment

These criteria and the table in which they are described are directly adapted fromresearch by The Danish Institute for Human Rights (DIHR). In 2020, the DIHR produced
their “Guidance on Human Rights Impact Assessment of Digital Activities: Introduction.”In this paper, they present 10 key principles for a Human Rights Impact Assessment withfive key criteria for the HRIA process. The criteria used here and the table in which they
are elaborated have been drawn from their guidance. See Kernell, E. L., Veiberg, C. B., &Jacquot, C. (2020). “Guidance on Human Rights Impact Assessment of Digital Activities:
Introduction.” The Danish Institute for Human Rights.
48

228

is guided by the local context,
including through using the impacted
individuals’ preferred mechanisms(e.g. modes of communication) where
possible

Engagement and consultation
processes are inclusive, gendersensitive, and account for theneeds of individuals and groups
at risk of vulnerability or
marginalisation.

Nondiscrimination

Capacity building of individuals
and groups at risk ofvulnerability or marginalisationis undertaken to ensure their
meaningful participation.

•

Make sure the assessment process isbeing undertaken at particular times
to ensure equitable participation (e.g.,when women are not at work andyoung people are not at school)•

Facilitate ongoing dialogue between
rights-holders, duty-bearers and otherrelevant
parties(e.g.
throughcollaborative problem analysis anddesign of mitigation measures)•

To the extent digital and virtual meansof engagement are utilised (e.g.,
online consultations and surveys),assess and address accessibility
issues, particularly with regard to themost vulnerable rights-holders•

Involvewomen,
men,andtransgender people, including throughgender-identity-aware
engagement
methods as necessary•

Take steps to ensure that the modesof engagement and participation
address any barriers that may be
faced by vulnerable and marginalised
individuals (e.g., by offering transportor holding meetings in culturally
appropriate locations, and considering‘technology barriers’ for older persons,
persons with disabilities, or persons
who do not have access to the internetor to digital technologies)•

Identify the needs of vulnerable andmarginalised individuals and act toaddress these to ensure inclusive
rights-holder involvement•

Includesufficient
time
in
the
assessment process for capacity
building to allow individuals andgroups to be meaningfully involved(e.g., to first present the digitalproducts or services in a way that theaudience understands, and to followup later with the same groups when
they have had time to discuss andorganise, in order to receive feedback
and potential concerns)•

Make sure that rights-holders haveaccess to independent and competent
legal, technical, and other advice as

Empowerment229

necessary, and, if not, include
provisions for making such support
available

The impact assessment process
is as transparent as possible toadequately engage affected or
potentially affected rightsholders, without causing any risk
to security and well-being ofrights-holders or otherparticipants (such as NGOs andhuman rights defenders). Impact
assessment findings areappropriately publicly
communicated.

•

Provide for capacity building of rightsholders to know their rights (e.g., bythoroughly explaining the right toprivacy before explaining how the AI
product or service will be developed toensure respect for the same right), as
well as of duty-bearers to meet their
human rights duties•

Providefor
information
sharing
between stakeholders at relevant andregular intervals (such as throughWorkflow Revisitation and Reporting
processes)•

Make available information about the
AI project, product, or service toparticipating stakeholders that is
adequate for giving a comprehensive
understandingofpotential
implications and human rights impacts(such as through sharing the PS
Report)•Publicly
communicate
HUDERIAfindings and impact management
plans (action plans) to the greatest
extent possible (e.g., published, withany reservations based on risk torights-holders or other participants
clearly justified)•

Secure a firm top-level management
commitment
withregardtotransparency before the start of the
HUDERIA process•

Communicate the phases of the
impactassessment,
includingtimeframes, to relevant stakeholders
in a clear and timely manner•

Ensure that communication andreporting consider and respond to the
local context (e.g., by making surethat information is made available in
relevant languages and formats, innon-technical summaries and inphysical and/or web-based formats
that are accessible to stakeholders)•

Make sure that responsibility for the
implementation,
monitoring
andfollow-up of mitigation measures is
assigned to particular individuals orfunctions within the organisation(e.g., data engineers are tasked withTransparency

Accountability

The impact assessment team is
supported by human rights
expertise, and the roles andresponsibilities for impact
assessment, prevention,
mitigation and management are230

assigned and adequately
resourced. The impact
assessment identifies theentitlements of rights-holders
and the duties andresponsibilities of relevant dutybearers (e.g., developers,
companies buying digitalproducts or services, those using
or applying digital products andservices, and government
authorities).

231

changing the design to limit potential
misuse)•Make sure that sufficient resources are
dedicatedtoundertaking
the
HUDERIA, as well as implementing theimpact management plan (e.g.,adequate time, as well as financial andhuman resources)•

Meaningfully
andappropriately
engage relevant duty-bearers in theimpact assessment process, including
in impact prevention, mitigation andremediation (e.g., data protection
authorities are engaged since somesystemic impacts can best be dealt
with through data protection policies
and regulation)•

Ensure that project team members
who are undertaking the HUDERIA
have the relevant interdisciplinary
skills and expertise (including humanrights, technical, legal, domain,
language, and local knowledge) toundertake the impact assessment in
the given context and with regard to
the specific AI product or service (e.g.,
data
engineers
andsoftwaredevelopers might need to be involved)•

Prioritize that the HUDERIA is carriedout by a diverse and inclusive group ofteam members

Three Steps in the HUDERIA Process

Once you have set up your HUDERIA process in accordance with the criteria
outlined above, your project team will be ready to begin working on the content
of the impact assessment. Although the impact assessment process may takemany different shapes—depending on the needs and resources of yourorganisation and the results of your PCRA and SEP—there are a few steps thatshould be common to any full HUDERIA process:Step 1.
Project teamprepares forand begins
HUDERIA•Project team members prepare for the HUDERIA
by revisiting Table 1 (Principles and Priorities,
Corresponding Rights and Freedoms, andElaborations), reviewing the outcome of Activity 3of PS reporting, and revisiting PCRA and SEP
results; Project team completes first part of the
HUDERIA and the additional project teamquestions supplement.

Step 2.
Rights-holders
prepared forand begin
HUDERIA•Rights-holders
arepreparedtomeaningfully engage in the HUDERIA
through Workflow Revisiting and Reporting
and by reviewing PS Report and content ofTable 1 with facilitators; Rights-holders
provide input on the HUDERIA questions
from the first part through facilitators.

Step 3.
Integrate resultsof Steps 1 and 2
through rightsholder/dutybearer dialogue•Results of Steps 1 and 2 are comparedand integrated into rights-holder/ dutybearer dialogue; Impact mitigation
measures consonant with severity ofpotential impacts are explored; impact
management plan is co-designed byproject team and engaged rightsholders; access to remedy andoperational grievance mechanisms are
set up.

Building the Content of your HUDERIA
As noted above, the content of the HUDERIA is assembled to allow your team (1)to re-examine and re-evaluate the potential harms you have already identified inyour PS Report and your PCRA, (2) to contextualize and corroborate thesepotential harms in dialogue with stakeholders, (3) to identify and analyse further
potential harms through extended project team reflection and by giving
stakeholders the chance to uncover new harms and to pinpoint gaps in thecompleteness of the previously enumerated harms, (4) to explore, withstakeholders, the severity (scope, scale, and remediability) of the potential
adverse impacts, so that the risks of these can be better assessed, prioritized,
managed, and mitigated, and (5) to set up access to remedy for affected rightsholders and other relevant parties. In the interest of covering this spectrum ofdesiderata, the HUDERIA questions and prompts are organised around three
categories:2323 Categories of HUDERIA questions and prompts:
1. Identifying potential adverse impacts
2. Assessing the severity of potential impacts identified
3. Mitigating potential impacts and setting up access toremedyIn the first part of the HUDERIA, the first two of these question types are applied
to each of the 20 impact prompts from section 2 of the PCRA. Just as in the PCRA,
these prompts are grouped under the appropriate principle/priority from Table 1,
so that the potential harms to the rights, freedoms, or elements of democracy andthe rule of law that are associated with each principle/priority can be considered
together under the appropriate heading. This makes it easier for project teammembers and engaged rights-holders to identify potential gaps in thecompleteness of the previously enumerated adverse impacts (as these fall undereach principle/priority) and to unearth corresponding harms that have not yet
been recognized. It also allows considerations about the severity, mitigation, andremedy of contextually specific impacts to be based around the appropriate
principle/priority and thus to include wider reflections on how each impact issituated within the more general area of concern. For instance, particular harms
related to anthropomorphic deception, humiliation, and instrumentalization can bethought about as adverse impacts on the dignity of affected rights-holders, andassociated considerations about the severity, mitigation, and remedy can thus
occur, in part, through the wider-angled lens of concerns about protecting andrespecting human dignity.

Identifying Potential Adverse Impacts
For the HUDERIA questions that come under the category of Identifying
potential adverse impacts, each prompt from section 2 of the PCRA is now
reformulated as an explanation-seeking “how, if at all,” question that enables the
project team members and stakeholders to interrogate the ways in which theright, freedom, or element of democracy and the rule of law associated with theprinciple/priority under consideration could be adversely impacted. 49 This enables
more open-ended and flexible deliberation on the range of potential adverse
impacts at the same time as it allows participants in the impact assessment
process to zoom in on the specific contexts of the impacts to facilitate better, moregranular understandings of their scope, scale, and remediability.
49
N.B., for these questions, project team members should refer back to Activity 3, Scope
Impacts to Human Rights, Democracy, and Rule of Law, in the PS Report and workfrom the previous answers given there.

233

The Identifying potential adverse impact questions also include a final gapidentifying “in what other ways, if at all” question that encourages project teammembers and engaged rights-holders to think beyond the pre-identified adverse
impacts first introduced in the PCRA and to discern novel harms. Here is how thisappears in the section on the Protection of Human Freedom and Autonomy:

HUDERIA Questions

Responses

Protection of Human Freedom and Autonomy
IDENTIFYING POTENTIAL ADVERSE IMPACTS
How, if at all, could the system adversely affector hinder the abilities of rights-holders to makefree, independent, and well-informed decisions
about their lives or about the system’s
outputs?
How, if at all, could the system adversely affector hinder the capacities of rights-holders toflourish, to fully develop themselves, and to
pursue their own freely determined life plans?
How, if at all, could the deployment of thesystem result in the arbitrary deprivation ofrights-holders' physical freedom or personal
security, or the denial of their freedoms ofexpression, thought, conscience, or assembly
In what other ways, if any, could the use of thissystem adversely impact the freedom andautonomy of affected rights-holders?

Assessing the Severity of Potential Impacts
As explained in the United Nations Guiding Principles on Business and Human
Rights (UNGP), assessing the severity of potential human rights impacts involves
consideration of their scale, scope, and remediability, where scale is defined as“the gravity or seriousness of the impact,” scope as “how widespread the impact
is, or the numbers of people impacted,” and remediability, as the “ability to restore
those affected to a situation at least the same as, or equivalent to, their situation
before the impact”. 50 The UNGP further notes: “Where it is necessary to prioritize. See Principle 14 of (UNGP, 2011), but also Council of Europe’s “Recommendation
CM/Rec(2020)1 of the Committee of Ministers to member States on the human rights

50

234

actions to address actual and potential adverse human rights impacts, businessenterprises should first seek to prevent and mitigate those that are most severeor where delayed response would make them irremediable.” 51
The HUDERIA questions that appear in the Assessing the Severity of Potential
Impacts section are intended to bring clarity to the prioritisation of impact
mitigation and assurance actions by allowing the severity levels of an AI system’s
potential impacts to be differentiated, elucidated, and refined by relevant rightsholders or their proxies. The elements of differentiation and inclusive rightsholder participation are crucial here. Different human rights impacts may havevarying degrees of relative severity and must therefore be carefully distinguishedand understood both individually and in relation to each other. Likewise,
understanding the scale, scope, and remediability of potential human rights
impacts requires that the lived experience and contextual insights of a diverse andinclusive group of potentially affected rights-holders be included in the assessment
process. This is because, as the United Nations Human Rights Office of the HighCommissioner notes, “scale, scope and remediability may differ for different
individuals or groups at heightened risks of becoming vulnerable or marginalized,
and…there may be different risks faced by different groups, such as men andwomen.” 52 Including a wide range of perspectives in the impact assessment
process—especially from members of vulnerable and marginalized groups—allows
for essential first-hand knowledge about how different groups may experienceharms differently to be incorporated into assessments of potential severity.

Scale

Assessing the scale of a potential adverse impact on human rights, fundamental
freedoms and elements of democracy and the rule of law involves bothconsideration of the gravity or seriousness of the impact and a contextuallysensitive understanding of the way that different groups (especially vulnerable,
historically-discriminated-against, and marginalized groups) could suffer prejudice

impacts of algorithmic systems” and the UN Human Rights Office of the High Commissioner(UNHROHC) (2020) “Identifying and Assessing Human Rights Risks related to End-Use”.
51
See (UNGP 2011, p. 26). The commentary to this paragraph expands, “While businessenterprises should address all their adverse human rights impacts, it may not always be
possible to address them simultaneously. In the absence of specific legal guidance, ifprioritization is necessary business enterprises should begin with those human rights
impacts that would be most severe, recognizing that a delayed response may affectremediability. Severity is not an absolute concept in this context, but is relative to theother human rights impacts the business enterprise has identified.” See also (UNHROHC
2020).
52(UNHROHC 2020, p. 7). The interpretive guidance for the UNGP underlines this point:“Depending on the operational context, the most severe human rights impact may be
faced by persons belonging to groups that are at higher risk of vulnerability or
marginalization, such as children, women, indigenous peoples, or people belonging to
ethnic or other minorities. If the enterprise decides it needs to prioritize its responses tohuman rights impacts, it should take into account the vulnerability of such groups and therisk that a delayed response to certain impacts could affect them disproportionately.” See:
UNHROHC (2012) “The Corporate Responsibility to Protect Human Rights: An Interpretive
Guide”. p. 84.

235

in their exercise of the right or freedom under consideration in different ways andto different extents.
To orient your considerations of the gravity of potential adverse impacts, review
the following Gravity Table 53:
Gravity Level

Catastrophic
Harm

Descriptions
Catastrophic prejudices or impairments in the exercise
of fundamental rights and freedoms that lead to thedeprivation of the right to life; irreversible injury tophysical, psychological, or moral integrity; deprivationof the welfare of entire groups or communities;
catastrophic harm to democratic society, the rule of law,or to the preconditions of democratic ways of life andjust legal order; deprivation of individual freedom andof the right to liberty and security; harm to thebiosphere.

Critical Harm

Critical prejudices or impairments in the exercise offundamental rights and freedoms that lead to thesignificant and enduring degradation of human dignity,
autonomy, physical, psychological, or moral integrity,or the integrity of communal life, democratic society, orjust legal order

Serious Harm

Serious prejudices or impairments in the exercise offundamental rights and freedoms that lead to thetemporary degradation of human dignity, autonomy,
physical, psychological, or moral integrity, or the
integrity of communal life, democratic society, or justlegal order or that harm to the information andcommunication environment

Moderate or
Minor Harm

Moderate or minor prejudices or impairments in theexercise of fundamental rights and freedoms that do notlead to any significant, enduring, or temporary
degradation of human dignity, autonomy, physical,
psychological, or moral integrity, or the integrity ofcommunal life, democratic society, or just legal order

In the HUDERIA questions contained in this subsection, you will be prompted tocharacterize the gravity of each potential adverse impact that has been identified.
Note that the use of this table to help you think through the gravity level of a possible
adverse human rights impact differs from the use of the Gravity Table in the PCRA, which
asked that the potential gravity of a harm be estimated assuming its maximal impact. In
the HUDERIA, both project team members and potentially impacted rights-holders should
engage in contextually aware explorations of the actual gravity or seriousness of thepotential impact.
53

236

You will also be asked to take into account the different ways and different extents
to which individuals or groups who possess characteristics that could make themmore vulnerable to the adverse impact could suffer the harm:

ASSESSING THE SEVERITY OF POTENTIAL IMPACTS
SCALE: Before answering the questions below, review the descriptions of harmin Gravity Table contained in the accompanying user’s guide.
For each potential impact identified in the previous
section, would you characterize it as a catastrophic,
critical, serious, or minor harm?
For each potential impact identified in the previous
section, are there individuals or groups who
possess characteristics that could make them more
vulnerable to the impact? If so, what are thesecharacteristics and could those who possess themsuffer the harm more acutely or seriously thanothers?
For each potential impact identified in the previous
section, which individuals or groups could
encounter the gravest impact on the right or
freedom under consideration?
Your responses to these questions (and the responses of potentially impacted
rights-holders) will subsequently serve an important function during the impact
mitigation planning stage when the redress and prioritization of harms is underconsideration. At this point, however, your focus should be on (1) thinking throughthe varying contexts of the potential harms, (2) making sure that considerations
about the differing vulnerabilities of rights-holders to suffering harm is integrated
into the gravity assessment, and (3) ensuring that relevant rights-holders or theirproxies are appropriately and sufficiently represented in this assessment process.

Scope
In the Scope subsection, you will be asked to consider how extensive or
widespread each of the potential adverse impacts identified could be in terms ofthe numbers of people potentially impacted, the timescale of the impacts, and theexposure of particular groups of affected rights-holders to harm:
SCOPE: Before answering the questions below, review the answers to Activity
1 (specifically in the Project section) of the PS Report and questions 10 and 11
from the PCRA (on numbers of rights-holders affected by the AI system andtimescale of the system’s potential impacts).

237

For each potential impact identified in the previous
section, does the answer given to question 10 of the
PCRA seem to provide an accurate estimate of thescope of the impact? If not, how many rightsholders could encounter a deprivation of the right or
freedom under consideration?
For each potential impact identified in the previous
section,are
there
groupswho
possess
characteristics that could make them vulnerable tohigher levels of exposure to the impact? If so, howmuch exposure could these groups face?
For each potential impact identified in the previous
section, consider the overall timescale of the AI
system’s impacts on the right or freedom underconsideration. Are there cumulative or aggregate
impacts of the system on rights-holders and theirprogeny that could expand the effects of the systembeyond the scope of impact identified in the previous
answers?

The first question asks that you re-evaluate the answer given to question 10 of
the PCRA (about the number of rights-holders potentially impacted by the system)in the context of the specific right or freedom under consideration. This is intended
as a sense check. In some cases, the estimated total number of rights-holders
potentially affected by a system’s use may be greater than the number of those
who could suffer from a particular harm, as, for instance, where a prejudice in theexercise of a right or freedom is suffered disproportionately by a specific group(e.g. a commercial insurance risk model used in healthcare that disproportionately
mismeasures the level of chronic illness for members of a particular racial group
because they are systematically underrepresented in the training dataset). In
other instances, the estimated number of rights-holders potentially affected by asystem’s use may (if misidentified) be much less than the number of those who
could suffer from a particular harm, as, for instance, where a system has knockon effects over time that lead to wider legacy impacts (e.g. harms done tomembers of a particular racial group by a discriminatory predictive decisionsupport model used in medicine leads to the group’s long-term reluctance anddistrust in the medical system and much wider impacts on the welfare of its
members).
The second question asks you to consider whether there are groups who possesscharacteristics that could make them vulnerable to higher levels of exposure to
the impact under consideration. The term “level of exposure” here is understoodas the proportion of a group that is adversely impacted by an AI system, where,
in the case that a small fraction of the group is impacted, members have low levelsof exposure and in the case that a very large fraction of the group is impacted,
members have high levels of exposure. As an example, members of a group thatis characterised by low socioeconomic status may have a high level of exposure
to the potential adverse impacts of an AI model that is used to allocate benefits238

in a social-services setting, because a large portion of that groups needs to accessthe benefits of social services in order to survive.
The final question on scope asks you to consider the overall timescale of the AI
system’s impacts on the right or freedom under consideration, taking the answer
given to question 11 of the PCRA as a starting point. In particular, it asks you tothink through whether there may be “cumulative or aggregate impacts of thesystem on rights-holders and their progeny that could expand the effects of thesystem beyond the scope of impact identified in the previous answers”. Identifying
cumulative or aggregate impact can be much more difficult than identifying harms
directly or proximately caused by the use of an AI system, and providing a fullanswer to this question, for the project team, may involve additional research andconsultation with domain experts and other relevant stakeholders. This difficulty
results from the fact that cumulative impacts are often incremental and moredifficult to perceive, and they frequently involve complex contexts of multiple
actors or projects operating in the same area or sector or affecting the sameSome “big picture” questions to reflect on when assessing
populations. 54
cumulative or aggregate impacts include:•

Could the production and use of the system contribute to wider adverse
human rights impacts when its deployment is coordinated with (or occurs
in tandem with) other systems that serve similar functions or purposes? For
example, if the human rights impacts of a conversational AI system thatserves a customer service function are considered in combination with theproliferation of many other similar systems in a given sector, concerns
about wider cumulative impacts on labour displacement, the right to work,and other social and economic rights become relevant.

•Could the production and use of the system replicate, reinforce, or augment
socio-historically entrenched legacy harms that create knock-on effects in
impacted individuals and groups? For example, an AI system that uses facial
analysis technology to match photographs to other pictures contained indatabases scraped from social media websites could add to the legacy
harms of companies that have used data recklessly and eroded public trust
regarding the respect of privacy and data protection rights in the digital
sphere, thereby creating wider chilling effects on elements of open
communication, information sharing, and interpersonal connection that are
essential components for the sustainability of democratic forms of life.

•

Could the production and use of the system be understood to contribute towider aggregate adverse impacts on the biosphere and on planetary healthwhen its deployment is considered in combination with other systems that
may have similar environmental impacts? For example, a complex AI
system that involves moderate levels of energy consumption in training,
operating, or data storage may be seen to contribute to significant
environmental impact when considered alongside the energy consumptionof similar systems across AI innovation ecosystems.

This explanation of cumulative impacts draws heavily on Götzmann, N., Bansal, T.,
Wrzoncki, E., Veiberg, C. B., Tedaldi, J., & Høvsgaard, R. (2020). Human rights impact
assessment guidance and toolbox. The Danish Institute for Human Rights. pp. 86-88.

54

239

Remediability

Questions in the Remediability subsection seek information about the degree of
reparability or restoration that is possible for affected rights-holders as the resultof efforts to overcome the adverse impact under consideration. As noted above,
remediability involves the “ability to restore those affected to a situation at least
the same as, or equivalent to, their situation before the impact.” Much as withconsiderations surrounding the gravity of a potential impact, gaining an
understanding of how remediable or reversable a harm is will depend on
knowledge both about the specific context of the harm and about the affectedrights-holders who are subjected to it. Members of different groups may havedifferent levels of resilience, depending on their positions in society and thecircumstances of the harm (with vulnerable and marginalised groups often
possessing less resilience than other dominant, privileged, or majority groups).
Those seeking to understand the degree of remediability for any specific adverse
impact on fundamental rights and freedoms must therefore pay close attention to
who could be harmed and in what contexts.
Establishing the degree of remediability for a potential adverse impact involves
considerations about the effort needed to overcome and (potentially) reverse theharm 55:
Degree ofRemediability

EffortVery Low

Suffered harm may be irreversible and may not be
overcome (e.g., long-term psychological or physicalailments, death, etc.)

Low

Suffered harm can be overcome albeit with serious
difficulties and enduring effects (e.g., economic loss,
property damage, worsening health, loss of social trust,deterioration of confidence in the legal order, etc.)

MediumSuffered harm can be overcome despite somedifficulties(e.g.,
extra
costs,
fear,
lack
of
understanding, stress, minor physical ailments, etc.)

High

Suffered harm can be overcome without any problem(e.g., time spent amending information, annoyances,
irritations, etc.)

You will be prompted to use this Table as a starting point, to answer thesequestions:55
This Table is adapted from Table 6 in Mantelero & Esposito (2021) – An evidence-based
methodology for human rights impact assessment (HRIA) in the development of AI dataintensive systems.

240

REMEDIABILITY: Before answering the questions below, review thedescriptions of harm in the Remediability Table contained in the accompanying
user’s guide.
For each potential impact identified in
the previous section, how remediable
is the adverse effect on impacted
right-holders?
(Refertothe
Remediability Table for categories ofremediability to consider)
For each potential impact identified in
the previous section, are there groups
who possess characteristics that could
make their members vulnerable tolower levels of remediability for the
impact (i.e., more subject to difficultto-overcome or irreversible harm)? If
so, to what degree is the impact onmembers of each group irreversible orirremediable?

Additional Questions for the Project TeamAt the end of Part I of the HUDERIA, you’ll find a supplemental section whichcontains additional questions about human rights diligence to be answered by the
project team. The purpose of these questions is to establish that all human rights
impacts that the project is linked to across its value chain have been appropriately
considered and assessed 56:

ADDITIONAL QUESTIONS FOR THE PROJECT TEAM
These questions are meant to serve as a supplement to Part I of the HUDERIA.
They are intended to be answered by the project team and then to be revisited
during the dialogue between duty-bearers and rights-holders in Step 3 of the
HUDERIAHas a thorough assessment of thehuman rights compliant businesspractices of all businesses, parties andentities involved in the value chain of
the AI product or service been
undertaken? This would include allbusinesses, parties, and entities
directly linked to your business lifecycle
through supply chains, operations,56
For further guidance on due diligence processes for responsible organisational conduct
as it relates to the value chain see OECD (2018), Due Diligence Guidance for Responsible
Business Conduct.

241

contracting, sales, consulting, andpartnering. (If not, do you have plansto do this?)
Do you have processes in place andappropriate resources dedicated tomaintaining human rights diligencewith regard to all businesses, parties,
and entities directly linked to yourbusiness lifecycle?
Do you require your vendors andsuppliers to demonstrate compliancewith legal protections in place toprotect rights-holders from modern
slavery, human trafficking, and labour
exploitation?

Mitigating Potential Impacts and Setting Up Access to Remedy
Once sections on Identifying Potential Adverse Impacts, Assessing the
Severity of Potential Impacts Identified, and Additional Questions for the
Project Team have been completed, the project team and engaged rights-holders
should be brought together to discuss and compare results and to begin impact
prevention and mitigation prioritization and planning. 57
Diligent impact prevention and mitigation planning begins with a scoping andprioritization stage. Project team members and engaged rights-holders should go
through all of the identified potential adverse impacts and map out theinterrelations and interdependencies between them as well as surrounding social
factors (such as contextually specific rights-holder vulnerabilities andprecariousness) that could make impact mitigation more challenging. Where
prioritization of prevention and mitigation actions is necessary (for instance, where
delays in addressing a potential harm could reduce its remediability), decisionmaking should be steered by the relative severity of the impacts underconsideration. As a general rule, while impact prevention and mitigation planning
may involve prioritization of actions, all potential adverse impacts on humanrights, fundamental freedoms, and elements of democracy and the rule oflaw must be addressed.

The mitigation hierarchy

When deciding upon the range of available actions that can be taken to preventor mitigate potential adverse impacts, project team members and engaged rights
should refer to what many different types of impact assessment processes call the“mitigation hierarchy” (avoid, reduce, restore, compensate). During the initial
iteration of the HUDERIA—i.e at the early, design stage of the project lifecycle—
the impacts under consideration will be potential impacts, namely, impacts thathave not yet happened, so mitigation options of “avoid” and “reduce” will be more
57

The following material in this section draws heavily on (Götzmann et al 2020).

242

relevant, whereas in later iterations of HUDERIA monitoring, revisitation and reevaluation—i.e. during the deployment stage—actual adverse impacts may arise,making mitigation options of “restore” and “compensate” relevant alongside“avoid” and “reduce”. Descriptions of the elements of the mitigation hierarchy are
as follows:
Mitigation
Hierarchy

Description

Avoid

Making changes to the design, development, or
deployment processes behind the production and use of
the AI system (or to the AI system itself), at the outset,
to avoid the potential adverse impact.

ReduceImplementing actions in the design, development, or
deployment processes behind the production and use of
the AI system (or making changes to the AI systemitself) to minimise potential or actual impacts.

RestoreTaking actions to restore or rehabilitate affected rightsholders to a situation at least the same as, or equivalent
to, their situation before the impact

Compensate

Compensating in kind or by other means, where othermitigation approaches are neither possible nor effective.

The use of the term “mitigation hierarchy” is prescriptive and meaningful here
in the sense that it is directing actions to give precedence to avoiding potential
adverse impacts altogether, in the first instance, and then to reducing andremediating them. It is also notable that, at later stages of the project lifecycle,
where options of restoration and compensation become more relevant, more than
one of these mitigation options may be at play contemporaneously (as, forinstance, where an affected rights-holder needs to be restored simultaneously as
immediate actions to minimise further harms must be taken). At all events,
decisions about which mitigation action(s) to take should be guided, first andforemost, by human rights considerations, and choices made to avoid and reduce
adverse impacts should take first position and be kept distinct from choices tocompensate or remunerate impacted rights-holders for suffered harms.

Co-designing and implementing an impact management plan

Once potential adverse impacts have been mapped out and organised, andmitigation actions have been considered, project team members and engaged
rights-holders should begin co-designing an impact mitigation plan (IMP). The IMPwill become the part of your HUDERIA that specifies the actions and processes
needed to address the adverse impacts which have been identified and that243

assigns responsibility for the completions of these tasks and processes. 58 As such,
the IMP will serve a crucial documenting function, which, as part of the HUDERIA,
will then become an evidentiary element of the argument-based assurance
practices carried out in your HUDERAC. The IMP should include:•

A summary of combined impact findings from the project team andengaged rights-holders with a detailed mapping of potential adverse
impacts and any significant interrelationships between them•

A clear presentation of the measures and actions that will be taken and,
where needed, a presentation of the prioritization of actions accompanied
by an explanation of prioritization rationale (The project team and engaged
rights-holders should also draw on recommendations generated in the
results of Section 1 of the PCRA when setting out needed measures andactions. This will feed directly into the goal definition and property
determination aspects of your HUDERAC.)•

For any potential adverse impacts identified in direct links of the businessor organisation to businesses, parties and entities involved in the value
chain of the AI product or service, a presentation of plans to exercise
leverage or alter relationships to mitigate impacts•

A clarification of the roles and responsibilities of the various actors involved
in impact mitigation, management, and monitoring•(From the project team directly): a statement that indicates the allottedcommitment of time and resources that will be made available to carry out
the IMP•A plan for monitoring impact mitigation efforts and for re-assessing and reevaluating the HUDERIA during subsequent development and deployment
phases of the project lifecycle•

An accessible presentation of access to remedy and operational-levelgrievance mechanisms that will be available to impacted rights-holders ifthey suffer any harms identified in the HUDERIA or any other adverse
impacts that were not detected or recognized in the impact assessment
processDaniel M. Franks (2011), ‘Management of the social impacts of mining’, in P. Darling(Ed), SME Mining Engineering Handbook (3rd edn), pp.1817-1825; Daniel M. Franks andFrank Vanclay (2013), ‘Social impact management plans: Innovation in corporate andpublic policy’, Environmental Impact Assessment Review, 43, p.57. Cited in (Götzmann etal 2020, p. 95).
58

244

Revisiting the HUDERIA across the Project Lifecyle
Carrying out a HUDERIA at the beginning of an AI innovation project is only a first,
albeit critical, step in a much longer, end-to-end process of responsive evaluation
and re-assessment. In your impact assessment process, you must pay continuous
attention both to the dynamic and changing character of the AI production andimplementation lifecycle and to the shifting conditions of the real-world
environments in which your AI system will be embedded.
There are two factors that necessitate this demand for responsiveness insustainable AI innovation:
1. Production and implementation factors: Choices made at any point
along the design, development, and deployment workflow may impact priordecisions and assessments—leading to a need for re-assessment,
reconsideration, and amendment. For instance, design and development
choices could be made that were not anticipated in the initial impact
assessment (such choices might include adjusting the target variable,choosing a more complex algorithm, or grouping variables in ways that
may impact specific groups). These changes may influence how an AI
system performs or how it impacts affected individuals and groups.
Processes of AI model design, development, and deployment are also
iterative and frequently bi-directional, and this often results in the need forrevision and update. For these reasons, responsible and sustainable AI
design, development, and use must remain agile, attentive to change, andat-the-ready to move back and forth across the decision-making pipeline
as downstream actions affect upstream choices and evaluations.
2. Environmental factors: Changes in project-relevant social, regulatory,
policy or legal environments (occurring during the time in which the system
is in production or use) may have a bearing on how well the model works
and on how the deployment of the system impacts the rights and freedoms
of affected individuals and groups. Likewise, domain-level reforms, policychanges, or changes in data recording methods may take place in the
population of concern in ways that affect whether the data used to train
the model accurately portrays phenomena, populations, or related factors
in an accurate manner. In the same vein, cultural or behavioral shifts may
occur within affected populations that alter the underlying data distribution
and hamper the performance of a model, which has been trained on datacollected prior to such shifts. All of these alterations of environmental
conditions can have a significant effect on how an AI system performs andon the way it impacts affected rights-holders and communities.
Bearing these factors in mind, it becomes clear that revisitation of your HUDERIA
at relevant points across your project lifecycle will play a pivotal role in its
continued efficacy and reliability. As part of the IMP component of your initial
HUDERIA activities, a plan is established for monitoring impact mitigation efforts
and for re-assessing and re-evaluating your HUDERIA during subsequentdevelopment and deployment phases of the project lifecycle. Such follow-up

245

processes of re-assessment and re-evaluation should include an appropriate
stakeholder engagement aspect, which is proportionate to the risks posed by thesystem. Such processes should also remain as responsive as possible to the waythat system is interacting with its operating environments and with impacted
rights-holders. In rapidly evolving or changing contexts, there may arise a needfor more frequent re-assessment and re-evaluation interventions, and yourproject team must remain flexible and ready to amend previous iterations of the
HUDERIA and IMP.
Additionally, in the monitoring phase—during system deployment—considerationsof actual adverse impacts are added to considerations of potential impacts. Thismeans that options for mitigating actual impacts (i.e. restoration andcompensation) will now be included as options in the mitigation hierarchy. Keepin mind, however, that the avoidance, reduction, and remediation of any adverse
human rights impacts should be given primacy in mitigation efforts and thatcompensation cannot replace these mitigation actions.

246

HUDERIA Template

Human Rights, Democracy, and the Rule of Law Impact Assessment (HUDERIA)
Template
Questions

Responses

Respect for and protection of human dignity
IDENTIFYING POTENTIAL ADVERSE IMPACTS
How, if at all, could this system prompt confusion
or uncertainty in rights-holders about whether they
are interacting with an AI system rather than ahuman being?
How, if at all, could this system expose rightsholders to humiliation (i.e put them in a state ofhelplessness or insignificance; dehumanize them or
deprive them of a sense of individual identity)?
How, if at all, could the system expose rightsholders to instrumentalization or objectification(treating them solely as exchangeable, as
statistical aggregates, as means to ends, or as
objects to be freely manipulated or steered)?
How, if at all, could this system expose rightsholders to displacement, redundancy, or a sense of247

worthlessness in regard to their participation inwork life, creative life, or the life of the community?
In what other ways, if any, could the use of thissystem adversely impact the dignity of affectedrights-holders?
ASSESSING THE SEVERITY OF POTENTIAL IMPACTS
SCALE: Before answering the questions below, review the descriptions of harm in Impact Gravity Table contained in theaccompanying user’s guide.
For each potential impact identified in the previous
section, would you characterize it as a catastrophic,
critical, serious, or moderate/minor harm?
For each potential impact identified in the previous
section, are there individuals or groups who
possess characteristics that could make them more
vulnerable to the impact? If so, could these
individuals or groups experience the harm more
intensely or seriously than others?
For each potential impact identified in the previous
section, which individuals or groups could
encounter the gravest prejudice in their exercise of
the right or freedom under consideration?
SCOPE: Before answering the questions below, review the answers to Activity 1 (specifically in the Project section) of the PS
Report and questions 10 and 11 from the PCRA (on numbers of rights-holders affected by the AI system and timescale of thesystem’s potential impacts).
For each potential impact identified in the previous
section, does the answer given to question 10 of248

the PCRA seem to provide an accurate estimate of
the scope of the impact? If not, how many rightsholders could encounter a deprivation of the rightor freedom under consideration?
For each potential impact identified in the previous
are
there
groupswho
possess
section,characteristics that could make them vulnerable tohigher levels of exposure to the impact? If so, howmuch exposure could these groups face?
For each potential impact identified in the previous
section, consider the overall timescale of the AI
system’s impacts on the right or freedom underconsideration. Are there cumulative or aggregate
impacts of the system on rights-holders and theirprogeny that could expand the effects of thesystem beyond the scope of impact identified in theprevious answers?
REMEDIABILITY: Before answering the questions below, review the descriptions of harm in Remediability Table contained in theaccompanying user’s guide.
For each potential impact identified in the previous
section, how remediable is the potential adverse
effect on impacted right-holders? (Refer to theRemediability Table for categories of remediability
to consider)
For each potential impact identified in the previous
are
there
groupswho
possess
section,characteristics that could make them vulnerable tolower levels of remediability for the impact (i.e.
moresubjecttodifficult-to-overcomeor249

irreversible harm)? If so, to what degree is theimpact irreversible or irremediable?

Protection of Human Freedom and Autonomy
IDENTIFYING POTENTIAL ADVERSE IMPACTS
How, if at all, could the system adversely affect orhinder the abilities of rights-holders to make free,independent, and well-informed decisions about
their lives or about the system’s outputs?
How, if at all, could the system adversely affect orhinder the capacities of rights-holders to flourish, tofully develop themselves, and to pursue their own
freely determined life plans?
How, if at all, could the deployment of the system
result in the arbitrary deprivation of rights-holders'
physical freedom or personal security, or the denialof their freedoms of expression, thought,
conscience, or assembly?
In what other ways, if any, could the use of thissystem adversely impact the freedom or autonomy
of affected rights-holders?
ASSESSING THE SEVERITY OF POTENTIAL IMPACTS
SCALE: Before answering the questions below, review the descriptions of harm in Impact Gravity Table contained in theaccompanying user’s guide.

250

For each potential impact identified in the previous
section, would you characterize it as a catastrophic,
critical, serious, or moderate/minor harm?
For each potential impact identified in the previous
section, are there individuals or groups who
possess characteristics that could make them more
vulnerable to the impact? If so, what are thesecharacteristics and could those who possess themsuffer the harm more acutely or seriously thanothers?
For each potential impact identified in the previous
section, which individuals or groups could
encounter the gravest impact on the right or
freedom under consideration?
SCOPE: Before answering the questions below, review the answers to Activity 1 (Project Section) of the PS Report and answers
to questions 10 and 11 from the PCRA (on numbers of rights-holders affected by the AI system and timescale of the system’s
potential impacts).
For each potential impact identified in the previous
section, does the answer given to question 10 of
the PCRA seem like accurate and realistic
representation of the potential scope of the impact?
If not, how many rights-holders could encounter adeprivation of the right or freedom underconsideration?
For each potential impact identified in the previous
are
there
groupswho
possess
section,characteristics that could make them vulnerable to251higher levels of exposure to the impact? If so, howmuch exposure could these groups face?
For each potential impact identified in the previous
section, consider the overall timescale of the AI
system’s impacts on the right or freedom underconsideration (answer to question 11 of the PCRA).
Are there cumulative or aggregate impacts of thesystem on rights-holders or their progeny that
could extend the effects of the system beyond thescope of impact identified in the previous answers?
REMEDIABILITY: Before answering the questions below, review the Remediability Table contained in the accompanying user’s
guide.
For each potential impact identified in the previous
section, how remediable is the potential adverse
effect on affected right-holders? (Refer to theRemediability Table for categories of remediability
to consider)
For each potential impact identified in the previous
section,are
there
groupswho
possess
characteristics that could make them vulnerable tolower levels of remediability for the impact (i.e.
ormoresubjecttodifficult-to-overcome
irreversible harm)? If so, what degree of
irreversible harm do these groups face?

Prevention of harm and protection of the right to life and physical, psychological, and moral integrity
IDENTIFYING POTENTIAL ADVERSE IMPACTS

252

How, if at all, could interaction with the AI system
deprive rights-holders of their right to life or their
physical, psychological, or moral integrity?
How, if at all, could the AI system’s interactions
with the environment (either in the processes of its
production or in its deployment) harm thebiosphere or adversely impact the health of theplanet?
In what other ways, if any, could the use of thissystem adversely impact the right to life or
physical, psychological, and moral integrity ofaffected rights-holders?
ASSESSING THE SEVERITY OF POTENTIAL IMPACTS
SCALE: Before answering the questions below, review the descriptions of harm in Impact Gravity Table contained in theaccompanying user’s guide.
For each potential impact identified in the previous
section, would you characterize it as a catastrophic,
critical, serious, or moderate/minor harm?
For each potential impact identified in the previous
section, are there individuals or groups who
possess characteristics that could make them more
vulnerable to the impact? If so, what are thesecharacteristics and could those who possess themsuffer the harm more acutely or seriously thanothers?
For each potential impact identified in the previous
section, which individuals or groups could
253encounter the gravest impact on the right or
freedom under consideration?
SCOPE: Before answering the questions below, review the answers to Activity 1 (Project Section) of the PS Report and answers
to questions 10 and 11 from the PCRA (on numbers of rights-holders affected by the AI system and timescale of the system’s
potential impacts).
For each potential impact identified in the previous
section, does the answer given to question 10 of
the PCRA seem like accurate and realistic
representation of the potential scope of the impact?
If not, how many rights-holders could encounter adeprivation of the right or freedom underconsideration?
For each potential impact identified in the previous
are
there
groupswho
possess
section,characteristics that could make them vulnerable tohigher levels of exposure to the impact? If so, howmuch exposure could these groups face?
For each potential impact identified in the previous
section, consider the overall timescale of the AI
system’s impacts on the right or freedom underconsideration (answer to question 11 of the PCRA).
Are there cumulative or aggregate impacts of thesystem on rights-holders or their progeny that
could extend the effects of the system beyond thescope of impact identified in the previous answers?
REMEDIABILITY: Before answering the questions below, review the Remediability Table contained in the accompanying user’s
guide.

254

For each potential impact identified in the previous
section, how remediable is the potential adverse
effect on affected right-holders? (Refer to theRemediability Table for categories of remediability
to consider)
For each potential impact identified in the previous
section,are
there
groupswho
possess
characteristics that could make them vulnerable tolower levels of remediability for the impact (i.e.
ormoresubjecttodifficult-to-overcome
irreversible harm)? If so, what degree of
irreversible harm do these groups face?

Non-discrimination, fairness, and equality
IDENTIFYING POTENTIAL ADVERSE IMPACTS
How, if at all, could the AI system (either in theprocesses of its production or in its deployment)
result in discrimination, have discriminatory effects
on
impactedrights-holders,orperform
differentially for different groups in discriminatoryor harmful ways—including for intersectionalgroupswherevulnerable
orprotected
characteristics converge?
How, if at all, could use of the AI system expand
existing inequalities in the communities it affects oraugment historical patterns of inequity anddiscrimination in these communities?

255

In what other ways, if any, could the use of thissystem cause or contribute to inequity or
inequality?
ASSESSING THE SEVERITY OF POTENTIAL IMPACTS
SCALE: Before answering the questions below, review the descriptions of harm in Impact Gravity Table contained in theaccompanying user’s guide.
For each potential impact identified in the previous
section, would you characterize it as a catastrophic,
critical, serious, or moderate/minor harm?
For each potential impact identified in the previous
section, are there individuals or groups who
possess characteristics that could make them more
vulnerable to the impact? If so, what are thesecharacteristics and could those who possess themsuffer the harm more acutely or seriously thanothers?
For each potential impact identified in the previous
section, which individuals or groups could
encounter the gravest impact on the right or
freedom under consideration?
SCOPE: Before answering the questions below, review the answers to Activity 1 (Project Section) of the PS Report and answers
to questions 10 and 11 from the PCRA (on numbers of rights-holders affected by the AI system and timescale of the system’s
potential impacts).
For each potential impact identified in the previous
section, does the answer given to question 10 of
the PCRA seem like accurate and realistic256representation of the potential scope of the impact?
If not, how many rights-holders could encounter adeprivation of the right or freedom underconsideration?
For each potential impact identified in the previous
are
there
groupswho
possess
section,characteristics that could make them vulnerable tohigher levels of exposure to the impact? If so, howmuch exposure could these groups face?
For each potential impact identified in the previous
section, consider the overall timescale of the AI
system’s impacts on the right or freedom underconsideration (answer to question 11 of the PCRA).
Are there cumulative or aggregate impacts of thesystem on rights-holders or their progeny that
could extend the effects of the system beyond thescope of impact identified in the previous answers?
REMEDIABILITY: Before answering the questions below, review the Remediability Table contained in the accompanying user’s
guide.
For each potential impact identified in the previous
section, how remediable is the potential adverse
effect on affected right-holders? (Refer to theRemediability Table for categories of remediability
to consider)
For each potential impact identified in the previous
section,are
there
groupswho
possess
characteristics that could make them vulnerable tolower levels of remediability for the impact (i.e.
moresubjecttodifficult-to-overcomeor257

irreversible harm)? If so, what degree
irreversible harm do these groups face?

ofData protection and the right to respect of private and family life
IDENTIFYING POTENTIAL ADVERSE IMPACTS
How, if at all, could the design, development, anddeployment of the AI system harm the rights todata protection enshrined in data protection andprivacy law and in the Council of Europe's
Convention 108+?
How, if at all, could the AI system intrude on orinterfere with the private and family life of rightsholders in ways that prevent or impede them frommaintaining a personal sphere that is independent
from the transformative effects of AI technologies
and in which they are at liberty to freely think, form
opinions and beliefs, and develop their personal
identities and intimate relationships without the
influence of AI technologies?
ASSESSING THE SEVERITY OF POTENTIAL IMPACTS
SCALE: Before answering the questions below, review the descriptions of harm in Impact Gravity Table contained in theaccompanying user’s guide.
For each potential impact identified in the previous
section, would you characterize it as a catastrophic,
critical, serious, or moderate/minor harm?
For each potential impact identified in the previous
section, are there individuals or groups who258possess characteristics that could make them more
vulnerable to the impact? If so, what are thesecharacteristics and could those who possess themsuffer the harm more acutely or seriously thanothers?
For each potential impact identified in the previous
section, which individuals or groups could
encounter the gravest impact on the right or
freedom under consideration?
SCOPE: Before answering the questions below, review the answers to Activity 1 (Project Section) of the PS Report and answers
to questions 10 and 11 from the PCRA (on numbers of rights-holders affected by the AI system and timescale of the system’s
potential impacts).
For each potential impact identified in the previous
section, does the answer given to question 10 of
the PCRA seem like accurate and realistic
representation of the potential scope of the impact?
If not, how many rights-holders could encounter adeprivation of the right or freedom underconsideration?
For each potential impact identified in the previous
are
there
groupswho
possess
section,characteristics that could make them vulnerable tohigher levels of exposure to the impact? If so, howmuch exposure could these groups face?
For each potential impact identified in the previous
section, consider the overall timescale of the AI
system’s impacts on the right or freedom underconsideration (answer to question 11 of the PCRA).
Are there cumulative or aggregate impacts of the259

system on rights-holders or their progeny that
could extend the effects of the system beyond thescope of impact identified in the previous answers?
REMEDIABILITY: Before answering the questions below, review the Remediability Table contained in the accompanying user’s
guide.
For each potential impact identified in the previous
section, how remediable is the potential adverse
effect on affected right-holders? (Refer to theRemediability Table for categories of remediability
to consider)
For each potential impact identified in the previous
are
there
groupswho
possess
section,characteristics that could make them vulnerable tolower levels of remediability for the impact (i.e.
ormoresubjecttodifficult-to-overcome
irreversible harm)? If so, what degree of
irreversible harm do these groups face?

Social and economic rights
IDENTIFYING POTENTIAL ADVERSE IMPACTS
How, if at all, could the deployment of the AI
system harm the social and economic rights ofaffected persons, including the right to just working
conditions, the right to safe and healthy working
conditions, the right to organize, the right to social
security, and the rights to the protection of healthand to social and medical assistance as set out inthe European Social Charter?

260

In what other ways, if any, could the use of thissystem adversely impact other relevant social andeconomic rights?
ASSESSING THE SEVERITY OF POTENTIAL IMPACTS
SCALE: Before answering the questions below, review the descriptions of harm in Impact Gravity Table contained in theaccompanying user’s guide.
For each potential impact identified in the previous
section, would you characterize it as a catastrophic,
critical, serious, or moderate/minor harm?
For each potential impact identified in the previous
section, are there individuals or groups who
possess characteristics that could make them more
vulnerable to the impact? If so, what are thesecharacteristics and could those who possess themsuffer the harm more acutely or seriously thanothers?
For each potential impact identified in the previous
section, which individuals or groups could
encounter the gravest impact on the right or
freedom under consideration?
SCOPE: Before answering the questions below, review the answers to Activity 1 (Project Section) of the PS Report and answers
to questions 10 and 11 from the PCRA (on numbers of rights-holders affected by the AI system and timescale of the system’s
potential impacts).
For each potential impact identified in the previous
section, does the answer given to question 10 of
the PCRA seem like accurate and realistic261

representation of the potential scope of the impact?
If not, how many rights-holders could encounter adeprivation of the right or freedom underconsideration?
For each potential impact identified in the previous
are
there
groupswho
possess
section,characteristics that could make them vulnerable tohigher levels of exposure to the impact? If so, howmuch exposure could these groups face?
For each potential impact identified in the previous
section, consider the overall timescale of the AI
system’s impacts on the right or freedom underconsideration (answer to question 11 of the PCRA).
Are there cumulative or aggregate impacts of thesystem on rights-holders or their progeny that
could extend the effects of the system beyond thescope of impact identified in the previous answers?
REMEDIABILITY: Before answering the questions below, review the Remediability Table contained in the accompanying user’s
guide.
For each potential impact identified in the previous
section, how remediable is the potential adverse
effect on affected right-holders? (Refer to theRemediability Table for categories of remediability
to consider)
For each potential impact identified in the previous
are
there
groupswho
possess
section,characteristics that could make them vulnerable tolower levels of remediability for the impact (i.e.
moresubjecttodifficult-to-overcomeor262

irreversible harm)? If so, what degree
irreversible harm do these groups face?

of

Democracy
IDENTIFYING POTENTIAL ADVERSE IMPACTS
How, if at all, could the use or misuse of the AI
system lead to interference with free and fairelection processes or with the ability of impacted
individuals to participate freely, fairly, and fully inthe political life of the community through any of
the following: mass deception, mass manipulation,
mass intimidation or behavioural control, or mass
personalized political targeting or profiling, at the
local, national, or global levels?
How, if at all, could the use or misuse of the AI
system lead to the mass dispersal, at local,
national, or global levels, of misinformation or
disinformation?
How, if at all, could the use or misuse of the AI
system lead to obstruction of informational
plurality, at local, national, or global levels?
How, if at all, could the use or misuse of the AI
system (in particular, information filtering models
such as recommender systems, search engines, ornews aggregators) lead to an obstruction of thefree and equitable flow of the legitimate and valid
forms of information that are necessary for themeaningful democratic participation of impacted263

rights holders and for their ability to engage freely,
fairly, and fully in collective problem-solving?
How, if at all, could the use or misuse of the AI
system (in particular, models that track, identify,
de-anonymize or surveil rights-holders and socialgroups or enable the creation of social graphs) leadto interference with or obstruction of impacted
rights-holders' abilities to exercise their freedomsof expression, assembly, or association?
In what other ways, if any, could the use of thissystem adversely impact democratic principles or
the preconditions of democratic ways of life?
ASSESSING THE SEVERITY OF POTENTIAL IMPACTS
SCALE: Before answering the questions below, review the descriptions of harm in Impact Gravity Table contained in theaccompanying user’s guide.
For each potential impact identified in the previous
section, would you characterize it as a catastrophic,
critical, serious, or moderate/minor harm?
For each potential impact identified in the previous
section, are there individuals or groups who
possess characteristics that could make them more
vulnerable to the impact? If so, what are thesecharacteristics and could those who possess themsuffer the harm more acutely or seriously thanothers?
For each potential impact identified in the previous
section, which individuals or groups could264encounter the gravest impact on the right or
freedom under consideration?
SCOPE: Before answering the questions below, review the answers to Activity 1 (Project Section) of the PS Report and answers
to questions 10 and 11 from the PCRA (on numbers of rights-holders affected by the AI system and timescale of the system’s
potential impacts).
For each potential impact identified in the previous
section, does the answer given to question 10 of
the PCRA seem like accurate and realistic
representation of the potential scope of the impact?
If not, how many rights-holders could encounter adeprivation of the right or freedom underconsideration?
For each potential impact identified in the previous
are
there
groupswho
possess
section,characteristics that could make them vulnerable tohigher levels of exposure to the impact? If so, howmuch exposure could these groups face?
For each potential impact identified in the previous
section, consider the overall timescale of the AI
system’s impacts on the right or freedom underconsideration (answer to question 11 of the PCRA).
Are there cumulative or aggregate impacts of thesystem on rights-holders or their progeny that
could extend the effects of the system beyond thescope of impact identified in the previous answers?
REMEDIABILITY: Before answering the questions below, review the Remediability Table contained in the accompanying user’s
guide.

265

For each potential impact identified in the previous
section, how remediable is the potential adverse
effect on affected right-holders? (Refer to theRemediability Table for categories of remediability
to consider)
For each potential impact identified in the previous
section,are
there
groupswho
possess
characteristics that could make them vulnerable tolower levels of remediability for the impact (i.e.
ormoresubjecttodifficult-to-overcome
irreversible harm)? If so, what degree of
irreversible harm do these groups face?

Rule of Law
IDENTIFYING POTENTIAL ADVERSE IMPACTS
How, if at all, could the deployment of the AI
system harm impacted individuals' rights toeffective remedy and to a fair trial (equality ofarms, right to a natural judge established by law,
the right to an independent and impartial tribunal,
and respect for the adversarial process)?
In what other ways, if any, could the use of thissystem adversely impact the rule of law or theprecondition of a just legal order?
ASSESSING THE SEVERITY OF POTENTIAL IMPACTS
SCALE: Before answering the questions below, review the descriptions of harm in Impact Gravity Table contained in theaccompanying user’s guide.

266

For each potential impact identified in the previous
section, would you characterize it as a catastrophic,
critical, serious, or moderate/minor harm?
For each potential impact identified in the previous
section, are there individuals or groups who
possess characteristics that could make them more
vulnerable to the impact? If so, what are thesecharacteristics and could those who possess themsuffer the harm more acutely or seriously thanothers?
For each potential impact identified in the previous
section, which individuals or groups could
encounter the gravest impact on the right or
freedom under consideration?
SCOPE: Before answering the questions below, review the answers to Activity 1 (Project Section) of the PS Report and answers
to questions 10 and 11 from the PCRA (on numbers of rights-holders affected by the AI system and timescale of the system’s
potential impacts).
For each potential impact identified in the previous
section, does the answer given to question 10 of
the PCRA seem like accurate and realistic
representation of the potential scope of the impact?
If not, how many rights-holders could encounter adeprivation of the right or freedom underconsideration?
For each potential impact identified in the previous
are
there
groupswho
possess
section,characteristics that could make them vulnerable to267higher levels of exposure to the impact? If so, howmuch exposure could these groups face?
For each potential impact identified in the previous
section, consider the overall timescale of the AI
system’s impacts on the right or freedom underconsideration (answer to question 11 of the PCRA).
Are there cumulative or aggregate impacts of thesystem on rights-holders or their progeny that
could extend the effects of the system beyond thescope of impact identified in the previous answers?
REMEDIABILITY: Before answering the questions below, review the Remediability Table contained in the accompanying user’s
guide.
For each potential impact identified in the previous
section, how remediable is the potential adverse
effect on affected right-holders? (Refer to theRemediability Table for categories of remediability
to consider)
For each potential impact identified in the previous
are
there
groupswho
possess
section,characteristics that could make them vulnerable tolower levels of remediability for the impact (i.e.
ormoresubjecttodifficult-to-overcome
irreversible harm)? If so, what degree of
irreversible harm do these groups face?

ADDITIONAL QUESTION FOR THE PROJECT TEAM
These questions are meant as a supplement to Part I of the HUDERIA. They are intended to be answered by the project team andthen to be revisited during the dialogue between duty-bearers and rights-holders in Step 3 of the HUDERIA

268

Has a thorough assessment of the human rights
compliant business practices of all businesses,
parties and entities involved in the value chain of
the AI product or service been undertaken? This
would include all businesses, parties, and entities
directly linked to your project lifecycle throughsupply chains, operations, contracting, sales,
consulting, and partnering. (If not, do you haveplans to do this?)
Do you have processes in place and appropriate
resources dedicated to maintaining human rights
diligence with regard to all businesses, parties, andentities directly linked to your project lifecycle?
Do you require your vendors and suppliers todemonstrate compliance with legal protections inplace to protect rights-holders from modern
slavery,human
trafficking,andlabour
exploitation?

MITIGATING POTENTIAL IMPACTS AND SETTING UP ACCESS TO REMEDYA SUMMARY OF COMBINED IMPACT FINDINGS
Provide a summary of combined impact findings from the project team and engaged rights-holders with a detailed mapping ofpotential adverse impacts and any significant interrelationships between them.

269

PRESENTATION OF THE MEASURES AND ACTIONS TO BE TAKEN
Present the measures and actions that will be taken and, where needed, a presentation of the prioritization of actions accompanied
by an explanation of prioritization rationale (The project team and engaged rights-holders should also draw on recommendations
generated in the results of Section 1 of the PCRA when setting out needed measures and actions).

PRESENTATION OF PLANS TO EXERCISE LEVERAGE OR ALTER RELATIONSHIPS
For any potential adverse impacts identified in direct links of the business or organisation to businesses, parties and entities
involved in the value chain of the AI product or service, present plans to exercise leverage or alter relationships to mitigate impacts.

CLARIFICATION OF ROLES AND RESPONSIBILITESClarify the roles and responsibilities of the various actors involved in impact mitigation, management, and monitoring.

270

STATEMENT OF COMMITMENT OF TIME AND RESOURCES FOR IMP(From the project team directly): Provide a statement that indicates the allotted commitment of time and resources that will bemade available to carry out the IMP.

PLAN FOR MONITORING IMPACT MITIGATION EFFORTS
Formulate a plan for monitoring impact mitigation efforts and for re-assessing and re-evaluating the HUDERIA during subsequentdevelopment and deployment phases of the project lifecycle.

ACCESSIBLE PRESENTATION OF ACCESS TO REMEDY AND OPERATIONAL-LEVEL GRIEVANCE MECHANISMS
Provide an accessible presentation of the access to remedy and operational-level grievance mechanisms that will be available toimpacted rights-holders if they suffer any harms identified in the HUDERIA or any other adverse impacts that were not detectedor recognized in the impact assessment process.

271

272

Human Rights, Democracy, and Rule of Law
Assurance Case (HUDERAC)

Section Outline

1. Introduction
2. What is an Assurance Case?
3. Structure of a HUDERAC
4. Components of a HUDERAC
a. Goal(s)b. Properties and (Project or System Property) Claimsc. Evidence
5. Building a HUDERAC
a. HUDERAC Templateb. Goals, Actions, and Claims273

Introduction
Assurance is a process of establishing trust.
For a stakeholder to trust that an AI system respects human rights andfundamental freedoms, democratic principles, and the rule of law—especially innovel, high risk, or safety critical domains—they will typically require assurance
that their trust is placed judiciously. 59
Providing assurance can take the form of compiling clear and accessible evidence
that a risk has been sufficiently mitigated 60, or that mechanisms have been put inplace to prevent negative impacts from occurring. 61
In the context of the HUDERAF, trust and assurance are built on and emerge fromthe foundations of the preliminary context-based risk analysis (PCRA), thestakeholder engagement process (SEP) and the human rights, democracy, andrule of law impact assessment (HUDERIA). These three stages, therefore, are partof the human rights, democracy, and rule of law assurance framework (HUDERAF)
that is detailed in this section.
In addition to outlining this framework and showing how it relates to the threeother stages, this section also presents the final practical mechanism of thisprocess: a human rights, democracy, and rule of law assurance case (HUDERAC).

What is an Assurance Case?

In its simplest form, an assurance case is a form of documentation that
demonstrates how a goal (or set of goals) has been established by reference torelevant (evidence-backed) properties of the system or project lifecycle. 62
However, the process of developing an assurance case can,•••assist internal reflection and deliberation by providing a systematic andstructured means for evaluating how the development of AI systems or
products impact human rights, democracy, and the rule of law;provide a deliberate means for the anticipation and pre-emption of potential
risks and adverse impacts through mechanisms of end-to-end assessment
and redress;
facilitate transparent communication between developers and affectedstakeholders;

59 High-Level Expert Group on AI. (2019). ‘Ethics Guidelines for Trustworthy AI’. European
Commission.
https://ec.europa.eu/digital-single-market/en/news/ethics-guidelinestrustworthy-ai
60 Cleland, G. M., Habli, I., Medhurst, J., & Health Foundation. (2012). Evidence: Using
safety cases in industry and healthcare.
61 Sujan, M., Furniss, D., Grundy, K., Grundy, H., Nelson, D., Elliott, M., White, S., Habli,
I., & Reynolds, N. (2019). Human factors challenges for the safe use of artificial
intelligence in patient care. BMJ Health & Care Informatics, 26(1), e100081.
https://doi.org/10.1136/bmjhci-2019-100081
62 GSN Community. (2018). GSN Community Standard (Version 2). The Assurance Case
Working Group. https://scsc.uk/r141B:1?t=1

••

support mechanisms and processes of documentation (or, reporting) toensure accountability (e.g. audits, compliance);and build trust and confidence by promoting the adoption of best practices(e.g. standards for warranted evidence) and by conveying the integrationof these into design, development, and deployment lifecycles to impacted
stakeholders.

In the specific context of the human rights, democracy, and rule of law assurance
framework, we can define an assurance case as:
A structured argument that provides assurance to another party (orparties) that a particular claim (or set of related claims) about aproperty of an AI system is warranted given the available evidence.
There are three primary components that are required to build an assurance case:(a) a concise statement about the top-level normative goal (or goals) of the
project, (b) a set of claims about the project or system properties required tooperationalise the goal, and (c) a body of supporting evidence that demonstrates
how the properties and goals have been realised.

Table 1 – the main elements of an assurance
case

Additional components can be added to provide additional context or information,
but we will just focus on these three for present purposes. Figure 1 shows a
simplified overview of how these primary components fit together.
Assurance cases are widely used in safety-critical domains, such as energy oraviation, where systems and manufacturing processes are closely inspected and275assessed to ensure they are safe to operate and use in their intended context. 63
Different approaches to the development of an assurance case will incorporate
additional elements beyond those presented in Figure 1. However, each of thedifferent approaches will typically have a goal-structured focus, and include
property claims about the system or project, which are backed up by evidence. 64
Collectively, these elements form a structured argument, which is designed toprovide assurance that a goal (or set of goals) has been sufficiently established(e.g., safety, explainability, data integrity). 65 We can, therefore, refer to thedevelopment of this structured argument as an argument-based form ofassurance.

Structure of a HUDERAC

Whereas the PCRA, SEP, and HUDERIA serve as mechanisms for identifying andevaluating possible harms that could arise from the design, development, anddeployment of an AI system, an assurance case serves to demonstrate what
actions have been taken to prevent such harms from occurring. An assurance
case, therefore, is the primary means by which the basis for trust in the system
is communicated to and scrutinised by diverse stakeholders, including rights
holders, procurers, regulators, and policymakers.
Consider the following example. A developer has completed the PCRA and worked
with stakeholders to further analyse and evaluate possible risks, as outlined in the
SEP. They have also carried out the HUDERIA and have identified a set of actions
that they need to perform when designing, developing, and deploying their AI
system. One of the actions the developer has taken is to engage independent
domain experts to help evaluate their training data and ensure that it has thefollowing properties: accurate, representative, and up-to-date.
Presuming the domain experts are in agreement and believe the training data
possess such properties, would this justify the developer making the following
claim?
The training data were accurate, representative, and up-to-date.
The developer may be confident that these properties of their training data havebeen established, but what matters here is that the developer can justify thisconfidence to other stakeholders. This is what it means to provide assurance.
So, how does one provide assurance that is both warranted and justified?
63 Hawkins, R. et al., (2021) ‘Guidance on the Assurance of Machine Learning inAutonomous Systems’. University of York: Assuring Autonomy International Programme(AAIP) https://www.york.ac.uk/media/assuring-autonomy/documents/AMLASv1.1.pdf
64 Bloomfield, R., & Bishop, P. (2010). Safety and Assurance Cases: Past, Present andPossible Future – an Adelard Perspective. In C. Dale & T. Anderson (Eds.), Making Systems
Safer (pp. 51–67). Springer London. https://doi.org/10.1007/978-1-84996-086-1_4
65 Walton, D. (2009). Argumentation Theory: A Very Short Introduction. In G. Simari &I. Rahwan (Eds.), Argumentation in Artificial Intelligence (pp. 1–22). Springer US.
https://doi.org/10.1007/978-0-387-98197-0_1

276

The structure of an assurance case helps establish a justified (or warranted) basis
for believing and claiming that a goal has been met. The directionality of Figure 1,
for instance, depicts a process of inferential support in which the evidence
supports a claim about some system or project property, and the set of property
claims subsequently help realise the goal. Figure 2 adapts the original figure topresent a partial example oriented towards a goal of safety.

Figure 1 - An example of a partial assurance case

The inferential structure of an assurance case helps establish trust that some goalhas been obtained. However, the structure is not sufficient on its own. In addition
to the structural aspects of an assurance case, we must also consider thesubstantive aspects that help establish warrant.

Components of a HUDERAC

Let’s look at the specific components of a human rights, democracy, and rule oflaw assurance case in more detail. Specifically, we will discuss the goal(s),properties, and evidence.

Goal(s)The HUDERAF sets important constraints on the types of goals that may be
selected. For instance, the PCRA, SEP, or HUDERIA all help to identify how specificfundamental rights and freedoms may be violated, how an AI system is likely toharm vulnerable users, or how certain democratic principles are at risk in the
deployment of a particular AI system. Each of these prior stages are orientatedtowards some goal or set of goals, even if the goal is not made explicit. One
function of an assurance case, therefore, is to make these goals explicit and relate277them to key human rights and freedoms, or principles of democratic governanceand the rule of law.
To aid this process, we can start from the following list of goals that are referred
to as the SAFE-D goals (see Box 1). We have provided a general description of
the meaning for each of these goals, including reference to ancillary andneighbouring concepts.

SAFETY is of paramount importance for ensuring the sustainable

development, deployment, and use of an AI system. From a technicalperspective, this requires the system to be secure, robust, andreliable. And from a social sustainability perspective, this requires thepractices behind the system’s production and use to be informed byongoing consideration of the risk of exposing affected rights-holders
to harms, continuous reflection on project context and impacts,ongoing stakeholder engagement and involvement, and changemonitoring of the system from its deployment through to its
retirement or deprovisioning.

ACCOUNTABILITY can include specific forms of process transparency(e.g., as enacted through process logs or external auditing) that may
be necessary for mechanisms of redress, or broader processes of
responsible governance that seek to establish clear roles of
responsibility where transparency may be inappropriate (e.g.,
confidential projects).

FAIRNESS is inseparably connected with sociolegal conceptions ofequity and justice, which may emphasize a variety of features suchas non-discrimination, equitable outcomes, or procedural fairness
through bias mitigation, but also social and economic equality,
diversity, and inclusiveness.

EXPLAINABILITY is a key condition for autonomous and informed

decision-making in situations where AI systems interact with orinfluence human judgement and decision-making. Explainability goesbeyond the ability to merely interpret the outcomes of an AI system;
it also depends on the ability to provide an accessible and relevant
information base about the processes behind the outcome.

DATA QUALITY, INTEGRITY, PROTECTION AND PRIVACY must all beestablished to be confident that the (data-driven) AI system has beendeveloped on secure grounds.
••‘DATA QUALITY’ captures the static properties of data, such aswhether they are (a) relevant to and representative of thedomain and use context, (b) balanced and complete in termsof how well the dataset represents the underlying data
generating process, and (c) up-to-date and accurate as
required by the project.
‘DATA INTEGRITY’ refers to more dynamic properties of data
stewardship, such as how a dataset evolves over the course ofa project lifecycle. In this manner, data integrity requires (a)contemporaneous and attributable records from the start of a

278

•

project (e.g., process logs; research statements), (b) ensuringconsistent and verifiable means of data analysis or processing
during development, and (c) taking steps to establish findable,
accessible, interoperable, and reusable records towards theend of a project’s lifecycle.
‘DATA PROTECTION AND PRIVACY’ reflect ongoing developments
and priorities as set out in relevant legislation and regulationof data practices as they pertain to fundamental rights andfreedoms, democracy, and the rule of law. For example, theright for data subjects to have inaccurate personal data
rectified or erased.

Box 1 – SAFE-D Goals

Each of these goals plays an important role in upholding human rights, democracy,
and the rule of law. For example, ‘DATA PROTECTION’ 66 and ‘EXPLAINABILITY’ are both
required to ensure that an individual’s autonomy is protected, whereas ‘FAIRNESS’and ‘DATA INTEGRITY’ are both key components of social equality.
The specific meaning of the goals, however, will be delineated in different ways
depending on the use context or domain of the AI system, and the processes andresults of the SEP. For example, depending on the AI system, SAFETY could relate
to the physical safety of patients in the context of an AI system used in healthcare,or environmental safety for an AI system used in agriculture or forestry. Moreover,a group of stakeholders may wish to prioritise responsible governance overtransparency, in relation to ACCOUNTABILITY where a project requires greater (andjustifiable) levels of secrecy.
This means that the top-level goals will need to be further specified, both bydetailing the intended use context for the AI system 67 and by making specificclaims about relevant properties of the AI system or project.

Properties and (Project or System Property) Claims
Each of the SAFE-D goals has a variety of lower-level properties associated withthem. These properties ought to be established in either the project or the system
if the goal is said to have been obtained. In a HUDERAC, a corresponding claim
about the (project or system) property is, therefore, required. The following
properties can be identified for each of the SAFE-D goals.

Safety•Sustainability: the goal of safety must be achieved with an eye towards
the sustainability of a safe system. This goes beyond environmental
sustainability (e.g., the ecological footprint of the project and system). It
also includes an understanding of the long-term use context and impact of

66 To indicate that a concept is being employed as a goal we will use small caps formatting(e.g., ‘fairness’.)67 The ‘project summary’ output from the PCRA is likely to help here, as it will typicallycontain a lot of this information.

279

•

•

•

•

the system, and the resources needed to ensure the system continues tooperate safely over time. For instance, sustainability may depend upon
sufficient change monitoring processes that establish whether there has
been a substantive change in the underlying data distributions or social
operating environment. Sustainability also involves engaging andinvolving rights-holders in the design and assessment of AI systems that
could impact their human rights and fundamental freedoms.
Security: Security encompasses the protection of several operational
dimensions of an AI system when confronted with possible adversarial
attack. A secure system is capable of maintaining the integrity of theinformation that constitutes it. This includes protecting its architecture fromthe unauthorised modification or damage of any of its component parts. A
secure system also remains continuously functional and accessible to its
authorised users and keeps confidential and private information secure
even under hostile or adversarial conditions.
Robustness: The objective of robustness can be thought of as the goalthat an AI system functions reliably and accurately under harsh conditions.
These conditions may include adversarial intervention, implementer error,or skewed goal-execution by an automated learner (in reinforcement
learning applications). The measure of robustness is therefore the strengthof a system’s integrity the soundness of its operation in response to difficult
conditions, adversarial attacks, perturbations, data poisoning, andundesirable reinforcement learning behaviour.
Reliability: The objective of reliability is that an AI system behaves exactly
as its designers intended and anticipated. A reliable system adheres to thespecifications it was programmed to carry out. Reliability is therefore ameasure of consistency and can establish confidence in the safety of asystem based upon the dependability with which it operationally conforms
to its intended functionality.
Accuracy and Performance Metrics: In machine learning, the accuracyof a model is the proportion of examples for which it generates a correct
output. This performance measure is also sometimes characterised
conversely as an error rate or the fraction of cases for which the model
produces an incorrect output. As a performance metric, accuracy should
be a central component to establishing and nuancing the approach to safe
AI. Specifying a reasonable performance level for the system may also often
require refining or exchanging of the measure of accuracy. For instance, if
certain errors are more significant or costly than others, a metric for total
cost can be integrated into the model so that the cost of one class of errors
can be weighed against that of another.

Accountability•

Traceability: Traceability refers to the process by which all stages of thedata lifecycle from collection to deployment to system updating or
deprovisioning are documented in a way that is accessible and easily
understood. This may include not only the parties within the organisation

280

•

•

•

•

•

•

involved but also the actions taken at each stage that may impact theindividuals who use the system.
Answerability: Answerability depends upon a human chain of
responsibility. Answerability responds to the question of who is accountable
for an automation supported outcome.
Auditability: Whereas the property of answerability responds to thequestion of who is accountable for an automation supported outcome, thenotion of auditability answers the question of how the designers andimplementers of AI systems are to be held accountable. This aspect ofaccountability has to do with demonstrating and evidencing both the
responsibility of design and use practices and the justifiability of outcomes.
Clear Data Provenance and Data Lineage: Clear provenance and datalineage consists of records that are accessible and simultaneously detail
how data was collected and how it has been used and altered throughout
the stages of pre-processing, modelling, training, testing, and deploying.
Accessibility: Accessibility involves ensuring that information about theprocesses that took place to design, develop, and deploy an AI system areeasily accessible by individuals. This not only refers to suitable means of
explanation (clear, understandable, and accessible language) but also themediums for delivery.
Reproducibility: Related to and dependant on the above four properties,
reproducibility refers to the ability for others to reproduce the steps you
have taken throughout your project to achieve the desired outcomes andwhere necessary to replicate the same outcomes by following the sameprocedure.
Responsible
Governance:
Responsible
governanceensures
accountability and responsibility for the processes that occur throughout
the data lifecycle. This includes the identification and assignment of a dataprotection officer, as well as clearly identifying data controllers andprocessors. This may also include the creation of an independent oversightboard to ensure these individuals are held accountable and the processes
are well-documented.

Fairness••

Bias Mitigation: It is not possible to eliminate bias entirely. However,effective bias mitigation processes can minimise the unwanted andundesirable impact of systematic deviations, distortions, or disparate
outcomes that arise to a project governance problem, interfering factor, orfrom insufficient reflection on historical social or structural discrimination.
Diversity and Inclusiveness: A significant component of fairness aware
design is ensuring the inclusion of diverse voices and opinions in the designand development process through the participation of a more
representative range of stakeholders. This includes considering whether
values of civic participation, inclusion, and diversity been adequately
considered in articulating the purpose and setting the goals of the project.

281

•

•

Consulting with internal organisational stakeholders is also necessary tostrengthen the openness, inclusiveness, and diversity of the project.
Non-Discrimination: your system should not create or contribute tocircumstances whereby members of protected groups are treateddifferently or less favourably than other groups because of their respective
protected characteristic.
Equality: the outcome or impact of a system should either maintain or
promote a state of affairs in which every individual has equal rights andliberties, and equal access or opportunities to whatever good or service theAI system brings about.

Explainability
•

•

•

•

Interpretability: Interpretability consists of the ability to know how andwhy a model performed the way it did in a specific context and therefore tounderstand the rationale behind its decision or behaviour.
Responsible Model Selection: The normal expectations of intelligibility
and accessibility that accompany the function the system will fulfil in the
sector or domain in which it will operate. The availability of moreinterpretable algorithmic models or techniques in cases where the selectionof an opaque model poses risks to the physical, psychological, or moral
integrity of rights-holders or to their human rights and fundamental
freedoms. The availability of the resources and capacity that will be needed
to responsibly provide supplementary methods of explanation (e.g. simpler
surrogate models, sensitivity analysis, or relative feature important) incases where an opaque model is deemed appropriate and selected.
Accessible Rationale Explanation: the reasons that led to a decision—
especially one that is automated—delivered in an accessible and nontechnical way.
Responsible Implementation and User Training: training users tooperate the AI system may include
o conveying basic knowledge about the nature of machine learning,o explaining the limitations of the system,o educating users about the risks of AI-related biases, such asdecision-automation bias or automation-distrust bias 68, ando encouraging users to view the benefits and risks of deploying thesesystems in terms of their role in helping humans to come tojudgements, rather than replacing that judgement.

68 Decision-automation bias occurs when users of AI decision-support systems may
become hampered in their critical judgment and situational awareness as a result of an
overconfidence in the objectivity, or certainty of the AI system. At the other extreme,automation-distrust bias occurs when users of an automated decision-support system tend
to disregard its contributions to evidence-based reasoning as a result of their distrust or
scepticism about AI technologies in general (see ICO & ATI (2020) for further information).

282

Data Quality
•

•

•

•

•

Source Integrity and Measurement Accuracy: effective bias mitigation
begins at the very commencement of data extraction and collection
processes. Both the sources and instruments of measurement may
introduce discriminatory factors into a dataset. When incorporated as inputs
in the training data, biased prior human decisions and judgments—such as
prejudiced scoring, ranking, interview-data or evaluation—will become the‘ground truth’ of the model and replicate the bias in the outputs of thesystem in order to secure discriminatory non-harm, as well as ensuring thatthe data sample has optimal source integrity. This involves securing or
confirming that the data gathering processes involved suitable, reliable, andimpartial sources of measurement and sound methods of collection.
Timeliness and Recency: if datasets include outdated data then changes
in the underlying data distribution may adversely affect the generalisabilityof the trained model. Provided these distributional drifts reflect changing
social relationship or group dynamics, this loss of accuracy with regard to
the actual characteristics of the underlying population may introduce bias
into an AI system. In preventing discriminatory outcomes, timeliness andrecency of all elements of the data that constitute the datasets must be
scrutinised.
Relevance, Appropriateness, and Domain Knowledge: The
understanding and utilisation of the most appropriate sources and types ofdata are crucial for building a robust and unbiased AI system. Solid domain
knowledge of the underlying population distribution and of the predictive orclassificatory goal of the project is instrumental for choosing optimally
relevant measurement inputs that contribute to the reasonable
determination of the defined solution. Domain experts should collaborateclosely with the technical team to assist in the determination of the
optimally appropriate categories and sources of measurement.
Adequacy of Quantity and Quality: this property involves assessing
whether the data available is comprehensive enough to address the
problem set at hand, as determined by the use case, domain, function, andpurpose of the system. Adequate quantity and quality should address
sample size, representativeness, and availability of features relevant toproblem.
Balance and Representativeness: a balanced and representative dataset
is one in which the distribution of features that are included, and thenumber of samples within each class is similar to the underlying distributionthat exists in the overall population.

Data Integrity 69•

Attributable: Data should clearly demonstrate who observed and recordedit, when it was observed and recorded, and who it is about.

69 Adapted from SL Controls. (n.d.)

283

•

•
•
•

•

Consistent, Legible and Accurate: Data should be easy to understand,recorded permanently and original entities should be preserved. Datashould be free from errors and conform with the protocol. Consistency
includes ensuring data is chronological (e.g., has a date and time stampthat is in the expected sequence).
Complete: All recorded data requires an audit trail to show nothing has
been deleted or lost.
Contemporaneous: Data should be recorded as it was observed, and at
the time it was executed.
Responsible Data Management: Responsible data management ensuresthat the team has been trained on how to manage data responsibly andsecurely, identifying possible risks and threats to the system and assigningroles and responsibilities for how to deal with these risks if they were tooccur. Policies on data storage and public dissemination of results should
be discussed within the team and with stakeholders, as well as being clearly
documented.
Data Traceability and Auditability: Any changes or revisions to thedataset (e.g., additions, augmentations, normalisation) that occur after theoriginal collection should be clearly traceable and well-documented tosupport any auditing.

Data Protection and Privacy••

Consent (or legitimate basis) for processing: Each Party shall providethat data processing can be carried out on the basis of the free, specific,
informed and unambiguous consent of the data subject or of some other
legitimate basis laid down by law. The data subject must be informed ofrisks that could arise in the absence of appropriate safeguards. Suchconsent must represent the free expression of an intentional choice, given
either by a statement (which can be written, including by electronic means,or oral) or by a clear affirmative action and which clearly indicates in thisspecific context the acceptance of the proposed processing of personal data.
Mere silence, inactivity or pre-validated forms or boxes should not,therefore, constitute consent. No undue influence or pressure (which canbe of an economic or other nature) whether direct or indirect, may be
exercised on the data subject and consent should not be regarded as freely
given where the data subject has no genuine or free choice or is unable torefuse or withdraw consent without prejudice. The data subject has the rightto withdraw the consent he or she gave at any time (which is to be
distinguished from the separate right to object to pro- cessing).
Data Security: Each Party shall provide that the controller, and, whereapplicable the processor, takes appropriate security measures against risks
such as accidental or unauthorised access to, destruction, loss, use,
modification or disclosure of personal data. Each Party shall provide thatthe controller notifies, without delay, at least the competent supervisory
authority within the meaning of Article 15 of this Convention, of those data

284

•

•

•

•

•

•

•

breaches which may seriously interfere with the rights and fundamental
freedoms of data subjects.
Data Minimisation: Personal data being processed is adequate (sufficient
to properly fulfil the stated purpose), relevant (has a rational link to that
purpose), and limited to what is necessary do not hold more data thanneeded for that purpose).
Transparency: The transparency of AI systems can refer to several
features, both of their inner workings and behaviours, as well as thesystems and processes that support them. An AI system is transparent
when it is possible to determine how it was designed, developed, anddeployed. This can include, among other things, a record of the data thatwere used to train the system, or the parameters of the model thattransforms the input (e.g., an image) into an output (e.g, a description of
the objects in the image). However, it can also refer to wider processes,
such as whether there are legal barriers that prevent individuals fromaccessing information that may be necessary to understand fully how thesystem functions (e.g., intellectual property restrictions).
Proportionality: delivering a just outcome in ways that are proportionate
to the cost, complexity, and resources available. In a similar vein, the term
‘proportionality’ can also be used as an evaluative notion, such as in thecase of a data protection principle that states only personal data that arenecessary and adequate for the purposes of the task are collected.
Purpose Limitation: The purposes for data processing must be outlined
and documented from the beginning and made available to all individuals
through privacy information. Personal data must adhere to the original
purpose unless it is compatible with the original purpose, additional consent
is received, or there is an obligation or function set out in law.
Accountability: Appropriate measures and records must be in place todemonstrate compliance and responsibility for how data has been processedin alignment with the other principles.
Lawfulness, fairness, and transparency: These three principles
necessitate ‘lawful basis’ for the collection and use of personal data.
Personal data must be used in a fair manner that is not unduly detrimental,
unexpected, or misleading. Any processes in which data is used should not
be in breach of any other laws, and teams must be clear, open, and honestwith individuals about how their personal data is being used.
Respect for the rights of data subjects: respect for the rights of data
subjects requires putting in place adequate mechanisms or undertaking
necessary actions so as to ensure that the rights of data subjects as defined
under Convention 108+ and GDPR are upheld. Where necessary, thisincludes the responsible handling of sensitive data.

Providing assurance for one or more of the above properties requires thedemonstration and documentation of relevant actions that have been taken at
different stages of a project’s lifecycle. These actions are captured through claims
about the properties, which may apply to either the project or system. For
example, the action of security testing a system may give rise to a property thatis salient for either the reliability or robustness of the system. Devising a claim

285

about the property of the system that has resulted from such an action may be a
pre-requisite for arguing that a system is SAFE TO OPERATE.
The set of system or project property claims will collectively help specify how thegoal is to be understood in the context of the project lifecycle. Figure 3 provides
an example of this using the goal of explainability.

Figure 2 – an example of a partial assurance case with an ‘explainability’ goal and supporting claims

The PCRA, SEP, and HUDERIA all play an important role in helping to identify
necessary and (jointly) sufficient claims. For instance, the SEP is designed to be
maximally inclusive of differing perspectives, as it is through this open dialogue
and engagement that developers will be likely to identify additional properties thatneed to be established in their project or system.
The section below titled ‘Goals, Actions and Claims’ provides a set of example
claims that are grouped according to the relevant SAFE-D goal and with reference
to specific actions and respective properties.

Evidence
A property claim that is made about the system or project will typically require
supporting evidence that the relevant action was performed. For instance,
consider the following claim,
During model testing we compared the decision tree classifier withless interpretable models but found no improvements in accuracy tojustify the loss of interpretability.
This claim helps justify why a specific model was selected. However, an auditor
may want to see the results of the model comparison process with specificreference to the performance metrics that were used. Providing this evidence isan important justificatory step that helps ground the assurance case.
Generating this evidence will occur naturally over the course of a project’s
lifecycle, and the specific steps set out in the workflow summary have also been
carefully designed to give rise to relevant evidential artefacts. Each decision or286

action taken, whether as part of the SEP or during a specific development stage(e.g., exploratory data analysis) is an action that is likely to or could create
documentation that can serve as evidence for a property claim. It is not, therefore,
necessary to compile all the evidence for an assurance case as an additional stepat the end of a project’s lifecycle. Instead, the HUDERAF has been designed tonaturally accommodate the iterative process of building an assurance case as aproject evolves. In the same way that an iterative approach to stakeholder
engagement is recommended, therefore, the process of building a HUDERAC isalso best approached as a continuous and iterative process.

Building a HUDERAC

Building a HUDERAC is about connecting the evidence gathered during project
design, model development, and system deployment to specific property claims,
in order to form a structured argument that some goal (or set of goals) has been
met. To simplify this process, we can think of the HUDERAF as an end-to-end
process of reflection, action, and justification that gives rise to a HUDERAC.
Early-stage project design activities, such as the PCRA or SEP, are reflective andanticipatory—identifying risks and possible impacts early on before they lead toharm. They also help map out a project governance process that can identify which
actions need to be taken to assure properties that are relevant to the goals of thesystem, and which may have been co-defined with stakeholders. The justificatory
step of compiling evidence, therefore, is linked to the final production of a
HUDERAC.
Figure 4 provides a high-level schematic of this process of reflection, action, andjustification, showing how specific elements of an assurance case emerge fromeach of the stages.

Figure 3 – a high-level schematic of the reflect, act, and justify (HUDERAF) process

The process of justification depends on compiling the relevant evidence and linking
it to the specific claims. However, whether an assurance case is judged to bejustified is a relational process. Although a developer can “make their case” forwhy a specific goal has been achieved, it must also be presented to relevant287

stakeholders (e.g., regulators, procurers) to evaluate its justifiability. In principle,
this is an ongoing process as different groups of stakeholders may wish to contest
or examine different property claims or question specific forms of evidence.
For example, let’s pretend that a SEP report is used as evidence for a claim thata specific group of stakeholders were engaged and approved of a particular design
choice for the respective AI system. This could seem, prima facie, like a reasonable
evidential basis. However, let’s also pretend that the stakeholders were only sentletters with the proposed design plans for the AI system and given a week torespond with objections, and that any non-responses would be treated asindicating approval. As we discussed earlier, this approach to stakeholder
engagement is known as “informing”, and is the weakest method of engagement.
Whether it is accepted as reliable evidence is, therefore, likely to depend on wider,contextual factors.
The process of justifying a HUDERAC should, therefore, be approached as aprocess of building and maintaining a living document, and as part of the overall
dialogical and reflective process that defines the HUDERAF.
In the next sections we provide a template for the development of a HUDERAC.
This template serves a procedural function insofar as it offers a practicalmechanism for project teams. It does not provide guidance on the substantive
content that ought to be included within a particular HUDERAC, as this can only
be determined by carrying out the actual stages of the HUDERAF (i.e., PCRA, SEP
and HUDERIA).

288

A HUDERAC Template

The process of building a HUDERAC is a three-stage process:
Reflect ⇒ Act ⇒ Justify
Each stage can be broken into a series of steps, which help build a HUDERAC. The following sections outline these steps andprovide a list of goals, actions, and claims that provide illustrative examples.

Reflect
STEPS: Reflection is an anticipatory process, so the steps taken in this stage can bethought of as deliberative prompts:
1. What are the goals of your system?
2. How are these goals defined?
3. Which stakeholders have participated in the identification and defining of thesegoals?
4. What properties need to be implemented in the project or system to ensurethat these goals are achieved?
5. Which actions ought to be taken to establish these properties within the projector system?
OUTPUTS: The HUDERAC outputs of this section will be a preliminary set of goals andclaim.

289

Act
STEPS: Salient actions are carried out at all stages of the HUDERAF, as detailed in theprocess map at the start of this document. A detailed process log and secure repositoryof documentation should be maintained to keep track and record these actions,
including any associated artefacts (e.g., data protection impact assessment, report ofmodel performance evaluation).
As the HUDERAF is split into three stages—design, development, and deployment—
there is a high-level structure that can help with the organisation and documentationof relevant actions:
1. What actions have been undertaken during (project) design that have generated
salient evidence for your goals and claims?
2. What actions have been undertaken during (model) development that havegenerated salient evidence for your goals and claims?
3. What actions have been undertaken during (system) deployment that havegenerated salient evidence for your goals and claims?
OUPUTS: A well maintained repository (or process log) of documentation andevidential artefacts, which demonstrate that specific actions were undertaken at
specific times throughout the project lifecycle, by specific team members.

290Justify
STEPS: Once the goals, claims, and evidence have been established anddocumented, the final step is to justify that your evidence base is sufficient towarrant the claims that are being made about the properties of your project orsystem. This connection is a vital step and can expose weaknesses in the overallcase being developed.
To help evaluate the evidence, the following questions can be instructive:
1. Which stakeholders, identified in your stakeholder engagement plan, cansupport the evaluation of your evidence and overall case?
2. Is any evidence missing from your case?
3. Are the collection of property claims jointly sufficient to support your top-level
goal?
OUPUTS: A HUDERAC that can be presented and communicated to relevant
stakeholder groups for assessment, evaluation, or auditing.

291

Goals, Actions and Claims
It is not possible to offer a comprehensive set of claims that need to be established for each of the SAFE-D goals and properties,
as the task of specifying the goals and operationalising the relevant properties is highly contextual.
However, Table 2 provides a set of examples to show how this may occur in different contexts. The claims are organised
according to where in the design, development, and deployment lifecycle they occur. We do not give a claim for every stage,
but do provide at least one example for every property, and also offer further references in footnotes to wider literature foreach goal.
Finally, it is important to note that some of the properties may relate to and serve multiple goals, but are presented inconnection with one in this table.
Claims and Actions

Goals

Sustainability

DesignDevelopment

Deployment

Previous safety cases for similar
technologies
wereconsulted
during the planning of oursystem to anticipate and identify
possible risks.

Key
decision-makers
collaborated
withsystemdevelopers to set reasonable
expectations about what isachievable.

The system requires ongoing
monitoring from a human-onthe-loop to minimise the risk ofphysical harm that could arise
from its operation.

SAFETY 70

The model used within oursystem has been internally andexternally validated. External
validation has been carried out
across
a
wide
range
of
environments to ensure thesystem is robust.

Robustness

CDT, n.d.; Leslie, 2019; Diakopoulos, et al., 2017; Amodei, et al., 2016; Auernhammer et al., 2019; Demšar & Bosnić, 2018; Google,
2019; Göpfert et al., 2018; Irving & Askell, 2019; Kohli, et al.., 2019; Kolter & Madry, n.d.; Marcus, 2018; Muñoz-González, et al., 2017;
Nicolae, et al., 2018; Ortega & Maini, 2018; Ranjan, et al., 2017; Ratasich, et al., 2019; Salay & Czarnecki, 2018; Shi, et al., 2018; Song,et al., 2018; Warde-Farley & Goodfellow, 2016; Webb, et al., 2018; Zantedeschi, et al., 2017; Zhao, et al., et al., 2018; Zhang, et al.,2019; Varshney & Alemzadeh, 201770

292

Our model was evaluated anddevelopedtominimiseitsvulnerability to inversion attacks(NB: model inversion attacks
attempt to reconstruct training
data from model parameters).

Security

Accuracy and
Performance
Metrics

The dataset was augmented
using
synthetic
data,
incollaboration with a domain andsubject matter expert, to address
known gaps and ensure thetraining data were representativeand complete.

ACCOUNTABILITY 71

Traceability

Diverse stakeholder groups wereinvolvedwiththemodel’sdevelopment
toensureit
performed reliably within andbetween each group.

The runtime environment of ourmodel is well controlled andunderstood. Our model training
was oriented towards
performance in thisenvironment.

Reliability

The origin of our data sources
and collection processes havebeen sufficiently documented tosupport reproducibility.

An independent security analyst
hascarriedout
extensive
penetration
testingonoursystems to ensure that sensitive
data will not be revealed to nontrusted parties.

Modelreliability
hasbeen
evaluated and optimised using
sensitivity
analysesandperturbations to training data tominimise the risk of encounteringnovel data in the runtime
environment.
A complete description of all datacollected by our system isavailable through our privacy
policy, and the data types arelinked to their specific uses.

AI Now Institute, 2018; Leslie, 2019; Binns, 2018; Cavoukian, et al., 2010; CDT (n.d.); Diakopoulos, et al., 2015; Diakopoulos, et al.,2017; Kroll et al., 2016; Malgieri & Comandé, 2017; O’Sullivan et al., 2019; Reed, 2018; Stahp & Wright, 2018; Wachter, et al, (2017a);
Zook, et al., 2017; ACM US Public Policy Council, 2017; Ananny & Crawford, 2018; Antunes, et al., 2018; Burrell, 2016; Citron, 2008;
Citron & Pasquale, 2014; Crawford & Schultz, 2014; Edwards & Veale, 2017; Kemper & Kolkman, 2019; Turilli & Floridi, 2009; Weller,
2017; Lepri, et al., 2018; Mittelstadt, et al., 2016; Selbst, et al., 2019; Suresh & Guttag, 2019; Veale, et al., 2018; Davidson & Freire,2008; Buhmann, et al., 2020; Cech, 2020; Cobbe, et al., 2021; Diakopoulos, 2014; Diakopoulos, 2015; Diakopoulos, 2016; FAT/ML, 2016;
Fink, 2018; European Parliamentary Research Service, 2019; Hamilton, et al., 2014; Hutchinson, et al., 2021; Kacianka & Pretschner,
2021; Kaminski, 2018; Wieringa, 2020; Young, et al., 2019; Zicari, et al., 2021; Tagiou, et al., 2019; Shah, 2018; ICO, 2020; Rosenblat,
et al., 2014; Rosenbaum & Fichman, 2019; Reddy, et al., 2019

71

293

Auditability

All relevant details of our modeltraining, testing, and validation
have been recorded on anaccessible team repository (e.g.,predictor selection processes,
baselines for model comparison,
performance metrics).

Clear Data
Provenance andData Lineage

System logs are stored securely
and kept for a period of up to 1
year to support any internal orexternal audit or review.

Accessibility

All identified stakeholders wereconsulted prior to thedevelopment of our system tohelp critically evaluate our
project plans and ensure they
were intelligible.

Where our system is deployed inpublic spaces, clear messaging
is established to ensure allindividuals are aware that thesystem is in operation.

An extensive set of
documentation has been madeavailable to any individual who
wishes to deploy our model in
their own system, including
suggestions about
appropriate/inappropriate use
cases.

Reproducibility

Responsible
Governance

A data protection officer has
identified and confirmed theappropriate controllers andprocessers of the data.

A member of the team is tasked
with monitoring use of thesystem for compliance with
relevant
standards
andisauthorised to revoke use if foundto be misused and/or harmful.
Members of the development
team are free and empowered

Answerability294to object to potentially harmful
design decisions without fear of
reprisal.

Non-Discrimination

FAIRNESS 72
Equality

Diversity andInclusiveness

Membersofidentity
anddemographic groups that are
most at risk of harm by AI
systems are consulted about
design intentions to help identify
andremovediscriminatory
effects of the AI system.

During pre-processing, steps
were taken to balance the
distribution of the data sets’classes.

An equality impact assessment
was performed to identify andmitigate potential discriminatory
harms that could arise throughthe deployment of this system.

The system has been thoroughly
tested to ensure its benefits andharms are distributed acrossidentities and groups in an
equitable manner.

The problem being addressed by
the system was formulated in a
multistakeholder
processtoensure the inclusion of a broad
rangeofperspectives
andpotential concerns.

Users are constrained by designor policy from using the system
to profile or target persons
based on protectedcharacteristics.

Post-processing techniques wereused to minimise our classifier’s Persons affected by use of thecorrelation
withprotected systemhaveavenuesofattributes.
recourse, ability to contest
system outputs, and demand
human intervention.
The system has been evaluatedby independent domain experts
to ensure that it does not
impinge on expressive rights.
The system will not make use of
personal data or reveal personal
data to others without theunambiguous consent of thesubject.
The system has been tested byindependent domain experts toensure that it will function
similarly for differently situatedpersons.

Diakopoulos, et al., 2017; Leslie, 2019; Donovan, et al., 2018; Binns, 2017; Binns, et al., 2018; Holstein, et al., 2018; Lepri, et al.,2018; Custers, 2013; Custers & Schermer, 2014; European Commission Expert Group on FAIR Data, 2018; L’heureux, et al., 2017; Ruggieri,
et al., 2010; Hajian, et al., 2016; Kamiran & Calders, 2012; Lehr & Ohm, 2017; Passi & Barocas, 2019; Singhal & Jena, 2013; van der
Aalst, et al., 2017; Corbett-Davies, et al., 2017; Dwork, et al., 2012; Grgić-Hlača, et al., 2017; Kusner, et al., 2017; Verma & Rubin, 2018;
Zafar, et al., 2015; Žliobaitė , 2017
72295

Bias Mitigation

Accessible
Rationale
Explanation

Data were collected, extracted,
or acquired with the meaningfulconsent of the data subjects.

Users have been fully trained to
identify and mitigate “humanfactors” concerns associated withsystem use.

Independent domain experts
analysed the data to determine
acceptable levels of bias.
Data
sets
are
open
accessibleforanalysis
independent auditors.

and Our system was tested with aby diverse set of end users toensure that its outputs wereaccessible at the time they wereneeded and delivered relevant
and interpretable information.

EXPLAINABILITY 73
User testing was conducted prior
to full deployment of the system.
This involved a representative
sample of users who were asked
toensurethey
could
satisfactorilyinterprettheoutputs of the system andprovide suitable explanations

Responsible
Implementationand User Training

Previously identified
stakeholders were consulted
again towards the end of the
project lifecycle to evaluate
whether they were happy withthe accessibility andcomprehensiveness of the
explanations being offered by
the system.
The interface through whichusers will interact with oursystem, has been developed tomeet universal design standards
and promote accessibility for allusers, including those withvisual or cognitive impairments.

CDT, n.d.; Diakopoulos, et al., 2017; Leslie, 2019; Janssen & Kuk, 2016; Wachter, et al, (2017a); Wachter et al., 2017b; Demšar &Bosnić, 2018; Edwards & Veale, 2017; Adadi & Berrada, 2018; Bathaee, 2018; Bibal & Frénay, 2016; Bracamonte, 2019; Burrell, 2016;
Card, 2017; Caruana, et al., 1999; Chen, et al., 2018; Doshi-Velez & Kim, 2017; Doshi-Velez, et al., 2017; Dosilovic, et al., 2018;
Eisenstadt & Althoff, 2018; Feldmann, 2018; Gilpin, et al., 2018; Guidotti, et al., 2018; Kleinberg, et al., 2017; Kroll, 2018; Lakkaraju, etal., 2016; Lepri, et al., 2018; Lipton, 2016; Lipton & Steinhardt, 2018; Lundberg & Lee, 2017, Mittelstadt, et al., 2019; Moinar, 2018;
Murdoch, et al., 2019; Olhede & Wolfe, 2018; Park, et al., 2016; Pedreschi, et al., 2018; Pedreschi, et al., 2019; Poursabzi-Sangdeh, etal., 2018; Ribeiro, et al., 2016b; Rudin, 2018; Rudin & Ustun, 2018; Shmueli, 2010; Shaywitz, 2018; Simonite, 2017; Sokol & Flach,2018; ICO & ATI, 2020

73

296

about its functions to affected
persons.
Featureswereselectedtooptimise for both interpretability
and predictive power.

Responsible Model
Selection

Interpretability

A range of stakeholder
engagement session were held
to facilitate accessible
participation and identify specificneeds and challenges of therespective groups.

Source Integrityand Measurement
Accuracy

An automated script has been
established that automaticallyflags for human checking any
unexpected deviations, missing
data, or unexpected formatting.

Mechanisms
have
been
establishedthatmonitorcontinuous training data, andtest/verify
theircontinued
accuracy.

Timeliness andRecency

Only data that were collected
within the previous 3 months
were used to ensure the training
data were up-to-date.

New data are collected and used
to retrain/revalidate the model
every month.

Relevance,
Appropriateness,and Domain
Knowledge

Domain experts were consulted
about the findings from our
exploratory data analysis, andverified the relevance of the
input variables.

DATA QUALITY 74

External validation of the model
was carried out prior to full
deployment to verify whether the
training data were adequate to

Adequacy of
Quantity andQuality74

Abiteboul, et al., 2015; Dai, et al., 2018; Leslie, 2019297

stand-in for data encountered in
novel settings.

Cross-validation was carried out
to determine the generalisabilityof training data samples andmodels.

Balance andRepresentativeness

Attributable

Consistent, Legible
and Accurate

Metadata is stored alongside thedatasets to help aid and improvedata analysis.
Multiple copies of the dataset
were stored securely and used toverify consistency throughout
the project.

Contemporaneous

Data collection was carried out
using a bespoke application to
avoid timely transcription andensure the data were captured ina relevant and labelled format.

New data that is collected in theruntime environment of thesystemisgatheredinastructured format and withaccompanying metadata.

Responsible DataManagement

Our project team completed
tailoredtrainingondata
governance prior to the start ofthe project.

Data is stored in an interoperable
and reusable format to promote
replicability and transparency.

Data Integrity 75

Data Traceability
and Auditability

75

Automated triggers have been
setup
tofrequently
check
whether the model is still
representative of the original
data generation process.

Training and testing data splits
have been fully documented.

Ambacher, et al., 2007; Faudeen, 2017; Stoyanovich, et al., 2017; SL Controls (n.d.)298

Consent (orlegitimate basis)for processing

Data Security

Beforedata
iscopiedortransferred outside of our securedevelopment environment, a
multi-party approval system istriggered to minimise the risk of
security leaks.

Data Minimisation

Domainexpertsandstakeholders were consulted to
identify the minimum level ofdata that needed to be collectedtoensuresatisfactory
performance of the system.

Data Protection
and Privacy 76

A document is available on our
website that explains how ourmodel works and also provides
details of the relevant input
features and outputs.

Transparency

Modelcomparison
wasundertaken to determine that all
personal data that are collected
were necessary for the purposesof the task by iteratively
removing different types andrecording the decrease in model
performance.

Proportionality

Before using the system a user isprovided with a privacy policy
outlining the purposes for all data

Purpose Limitation

ICO, 2017; Council of Europe, 2018; Cavoukian, et al., 2010; ICO, 2021; Malgieri & Comandé, 2017; Veale, et al., 2018; Wachter et al.,2017b; Alper, et al., 2018; Antignac, et al., 2016; Kaminski, 2018

76

299

that is collected and required toacknowledge informed consent
to proceed.
Mechanisms
have
been
established that allow users torequest full data erasure if theyno longer want to use thesystem.

Accountability

Lawfulness,
Fairness, andTransparency

The system is designed to Personal data for each of thefunction without the need to records were collected to verify
collect any personal information. that none of the extracted
featurescontained
hiddenproxies. Once this assessment
had been carried out, thepersonal data were destroyed.
The model selection process wasconstrained by the requirementthat any model must be
interpretable to ensure their rightto be informed is respected.

Respect for therights of data
subjects

Table 2 – List of goals and example claims and actions that could be used in a HUDERAC

300

Appendix 1: Summary of Processes, Steps, and User Activities

301

Appendix 2: HUDERAF Project Lifecycle Process Map

302

Appendix 3: A Sociotechnical Approach to
the ML/AI Project Lifecycle77
There are many ways of carving up a project lifecycle for a ML/AI system or otherdata-driven technology (hereafter shortened to just ‘project lifecycle’). For
instance, Sweenor et al. (2020) break it into four stages: Build, Manage, Deploy& Integrate, Monitor. 78 Similarly, (Ashmore, Calinescu, and Paterson 2019)
identify four stages, which have a more specific focus on data science: data
management, model learning, model verification, and model deployment.
Furthermore, there are also well-established methods that seek to govern
common tasks within a project lifecyle, such as data mining (e.g. CRISP-DM or
SEMMA).
The multiplicity of approaches is likely a product of the evolution of diverse
methods in data mining/analytics, the significant impact of ML on research andinnovation, and the specific practices and considerations inherent to each of thevarious domains where ML techniques are applied. While there are many benefits
of existing frameworks (e.g. carving up a complex process into smaller
components that can be managed by a network of teams or organisations), they
do not tend to focus on the wider social or ethical aspects that interweave
throughout the various stages of a ML lifecycle. Figure 2, therefore, presents amodel of the ML lifecycle, which we have designed to support the process ofbuilding an ethical assurance case, while remaining faithful to the importance oftechnical requirements and challenges and also supporting more open, reflective,
and participatory forms of deliberation.

This Appendix reproduces a section of Christopher Burr and David Leslie (2021), “Ethical
Assurance: A practical approach to the responsible design, development, and deploymentof data-driven technologies” Forthcoming, The Alan Turing Institute.
78
These four stages are influenced by an ‘ML OPs’ perspective (Sweenor et al. 2020). The
term ‘MLOps’ refers to the application of DevOps practices to ML pipelines. The term isoften used in an inclusive manner to incorporate traditional statistical or data science
practices that support the ML lifecycle, but are not themselves constitutive of machinelearning (e.g. exploratory data analysis), as well as deployment practices that areimportant within business and operational contexts (e.g. monitoring KPIs).
77

303

The Project Lifecycle—the overarching stages of design, development, anddeployment (for a typical data-driven project) can be split into indicative tasks
and activities. In practice, both the stages and the tasks will overlap with theirneighbours, and may be revisited where a particular task requires an iterative
approach. The spiral indicates that this is a diachronic, macroscopic process thatevolves and develops over time, and as the deployment stage finishes, a newiteration is likely to begin.
To begin, the inner circle breaks the project lifecycle into a process of (project)design, (model) development, and (system) deployment. These terms areintended to be maximally inclusive. For example, the design stage encompasses
any project task or decision-making process that scaffolds or sets constraints onlater project stages (i.e. design system constraints). Importantly, this includes
ethical, social, and legal constraints, which we will discuss later.
Each of the stages shades into its neighbours, as there is no clearly delineatedboundary that differentiates certain project design activities (e.g. data extractionand exploratory analysis) from model design activities (e.g. preprocessing andfeature engineering, model selection). As such, the design stage overlaps with thedevelopment stage, but the latter extends to include the actual process of training,
testing, and validating a ML model. Similarly, the process of productionizing amodel within its runtime environment can be thought of as both a development
and deployment activity. And, so, the deployment stage overlaps with the‘development’ stage, and also overlaps with the ‘design’ stage, as the deployment304

of a system should be thought of as an ongoing process (e.g. where new data areused to continuously train the ML model, or, the decision to de-provision a model
may require the planning and design of a new model, if the older (legacy) system
becomes outdated). For these reasons, the project lifecycle is depicted as a spiral.
However, despite the unidirectional nature of the arrows, we also acknowledge
that ML/AI research and innovation is frequently an iterative process. Therefore,
the singular direction is only present at a macroscopic level of abstraction (i.e.,the overall direction of progress for a project), and allows for some inevitable backand forth between the stages at the microscopic level.
The three higher-level stages can be thought of as a useful heuristic forapproaching the project lifecycle. However, each higher-level stage subsumes awide variety of tasks and activities that are likely to be carried out by different
individuals, teams, and organisations, depending on their specific roles andresponsibilities (e.g. procurement of data). Therefore, it is important to breakeach of the three higher-level stages into their (typical) constituent parts, which
are likely to vary to some extent between specific projects or within particularorganisations. In doing so, we expose a wide range of diverse tasks, each of whichgive rise to a variety of ethical, social, and legal challenges. The following sections
provides an illustrative overview of these stages and tasks, as well as a nonexhaustive sample of the associated challenges.

(Project) Design Tasks and Processes
Project Planning
Rather than using ML/AI as a “hammer” to go looking for nails, it is best to havea clear idea in mind of what the project’s goals are at the outset. This can help toavoid a myopic focus on a narrow class of ML/AI-based “solutions,” and also helps
create space for a diversity of approaches—some of which may not require ML/AI
at all. Project planning, therefore, can comprise a wide variety of tasks, including,but not limited to:•an assessment of whether building an AI model is the right approach given
available resources and data, existing technologies and processes already inplace, the complexity of the use-contexts involved, and the nature of thepolicy or social problem that needs to be solved (Leslie et al 2021a);•

an analysis of user needs in relation to the prospective AI model and whether
a solution involving the latter provides appropriate affordances in keeping withuser needs and related functional desiderata;•

mapping of key stages in the project to support governance and businesstasks (e.g. scenario planning);•

an assessment of resources and capabilities within a team, which is necessary
for identifying any skills gaps,•

a contextual assessment of the target domain and of the expectations, norms,
and requirements that derive therefrom;

305

•

stakeholder analysis and team positionality reflection to determine theappropriate level and scope of community engagement activities (Leslie et al2021b);•

stakeholder impact assessment, supported by affected people andcommunities, to identify and evaluate possible harms and benefits associatedwith the project (e.g. socioeconomic inequalities that may be exacerbated asa result of carrying out the project), to gain social license and public trust,and also feed into the process of problem formulation in the next stage;•

wider impact assessments—both where required by statute and done
voluntarily for transparency and best practice (e.g. equality impact
assessments, data protection impact assessments, human rights impact
assessment, bias assessment)

Problem Formulation
Here, ‘problem’ refers both to a well-defined computational process (or a higherlevel abstraction of the process) that is carried out by the algorithm to map inputs
to outputs and to the wider practical, social, or policy issue that will be addressed
through the translation of that issue into the statistical or mathematical frame.
For instance, on the computational side, a convolutional neural network carries
out a series of successive transformations by taking (as input) an image, encoded
as an array, in order to produce (as output) a decision about whether some object
is present in the image. On the practical, social, and policy side, there will be aneed to define the computational “problem” being solved in terms of thealgorithmic system’s embeddedness in the social environment and to explain how
it contributes to (or affects) the wider sociotechnical issue being considered. In
the convolutional neural network example, the system being produced may be a
facial recognition technology that responds to a perceived need for the biometricidentification of criminal suspects by matching face images in a police database.
The social issue of wanting to identify suspects is, in this case, translated into thecomputational mechanism of the computer vision system. But, beyond this,diligent consideration of the practical, social, or policy issue being addressed by
the system will also trigger, inter alia, reflection on the complex intersection ofpotential algorithmic bias, the cascading effects of sociohistorical patterns ofracism and discrimination, wider societal and community impacts, and thepotential effects of the use of the model on the actors in the criminal justicesystems who will become implementers and subjects of the technology.
Sociotechnical considerations are also important for determining and evaluating
the choice of target variables used by the algorithm, which may ultimately be
implemented within a larger automated decision-making system (e.g. in averification system). The task of formulating the problem allows the project teamto get clear on what input data will be needed, for what purpose, and whether
there exists any representational issues in, for example, how the target variables
are defined. It also allows for a project team (and impacted stakeholders) to reflect
on the reasonableness of the measurable proxy that is used as a mathematicalexpression of the target variable, for instance, whether being taken into carewithin six months of a visit from child protective services is a reasonable proxy for306

a child’s being “at risk” in a predictive risk model for children’s social care. The
semantic openness and contestability of formulating problems and defining target
variables in ML/AI innovation lifecycles is why stakeholder engagement, whichhelps bring a diversity of perspectives to project design, is so vital, and why thisstage is so closely connected with the interpretive burdens of the project planningstage (e.g. discussion about legal and ethical concerns regarding permissible usesof personal or sensitive information).

Data Extraction or Procurement
Ideally, the project team should have a clear idea in mind (from the planning andproblem formulation stages) of what data are needed prior to extracting or
procuring them. This can help mitigate risks associated with over-collection of data
(e.g. increased privacy or security concerns) and help align the project with values
such as data minimisation (ICO and Institute 2020). Of course, this stage mayneed to be revisited after carrying out subsequent tasks (e.g. preprocessing,model testing) if it is clear that insufficient or imbalanced data were collected toachieve the project’s goals. Where data is procured, questions about provenancearise (e.g. legal issues, concerns about informed consent of human data subjects).
Generally, responsible data extraction and procurement require the incorporationof domain expertise into decision-making so that desiderata of data minimisation
as well as of securing relevant and sufficient data can be integrated into design
choices.

Data Analysis
Exploratory data analysis is an important stage for hypothesis generation or
uncovering possible limitations of the dataset that can arise from missing data, inturn identifying the need for any subsequent augmentation of the dataset to deal
with possible class imbalances. However, there are also risks that stem from
cognitive biases (e.g. confirmation bias) that can create cascading effects that
effect downstream tasks (e.g. model reporting).

(Model) Development Tasks and Processes
Preprocessing and Feature Engineering
Pre-processing and feature engineering is a vital but often lengthy process, which
overlaps with the design tasks in the previous section and shares with them thepotential for human choices to introduce biases and discriminatory patterns into
the ML/AI workflow. Tasks at this stage include data cleaning, data wrangling or
normalisation, and data reduction or augmentation. It is well understood that themethods employed for each of these tasks can have a significant impact on themodel’s performance (e.g. deletion of rows versus imputation methods forhandling missing data). As Ashmore et al. (2019) note, there are also various
desiderata that motivate the tasks, such as the need to ensure the dataset thatwill feed into the subsequent stages is relevant, complete, balanced, and accurate.
At this stage, human decisions about how to group or disaggregate input features(e.g. how to carve up categories of gender or ethnic groups) or about which input
features to exclude altogether (e.g. leaving out deprivation indicators in a307

predictive model for clinical diagnostics) can have significant downstream
influences on the fairness and equity of an ML/AI system.

Model Selection
This stage determines the model type and structure that will be produced in thenext stages. In some projects, model selection will result in multiple models forthe purpose of comparison based on some performance metric (e.g. accuracy). In
other projects, there may be a need to first of all implement a pre-existing set offormal models into code. The class of relevant models is likely to have been highly
constrained by many of the previous stages (e.g. available resources and skills,
problem formulation), for instance, where the problem demands a supervised
learning algorithm instead of an unsupervised learning algorithm; or whereexplainability considerations require a more interpretable model (e.g. a decisiontree).

Model Training
Prior to training the model, the dataset will need to be split into training and testing
sets to avoid model overfitting. The training set is used to fit the ML model,
whereas the testing set is a hold-out sample that is used to evaluate the fit of the
ML model to the underlying data distribution. There are various methods for
splitting a dataset into these components, which are widely available in popular
package libraries (e.g. the scikit-learn library for the Python programming
language). Again, human decision-making at this stage about the training-testing
split and about how this shapes desiderata for external validation—a subsequent
process where the model is validated in wholly new environments—can be very
consequential for the trustworthiness and reasonableness of the development
phase of an ML/AI system.

Model Validation and Testing
The testing set is typically kept separate from the training set, in order to provide
an unbiased evaluation of the final model fit on the training dataset. However, thetraining set can be further split to create a validation set, which can then be used
to evaluate the model while also tuning model hyperparameters. This process canbe performed repeatedly, in a technique known as (k-fold) cross-validation, where
the training data are resampled (k-times) to compare models and estimate theirperformance in general when used to make predictions on unseen data. This typeof validation is also known as ‘internal validation,’ to distinguish it from external
validation, and, in a similar way to choices made about the training-testing split,
the manner in which it is approached can have critical consequences for how theperformance of a system is measured against the real-world conditions that it will
face when operating “in the wild.”

Model Reporting
Although the previous stages are likely to create a series of artefacts while
undertaking the tasks themselves, model reporting should also be handled as aseparate stage to ensure that the project team reflect on the future needs ofvarious stakeholders and end users. While this stage is likely to include information
about the performance measures used for evaluating the model (e.g. decisionthresholds for classifiers, accuracy metrics), it can (and should) include wider
308

considerations, such as intended use of the model, details of the features used,training-testing distributions, and any ethical considerations that arise from thesedecisions (e.g. fairness constraints, use of politically sensitive demographic
features). 79(System) Deployment Tasks and Processes
Model Productionalization
Unless the end result of the project is the model itself, which is perhaps morecommon in scientific research, it is likely that the model will need to be
implemented within a larger system. This process, sometimes known as ‘model
operationalisation,’ requires understanding (a) how the model is intended tofunction in the proximate system (e.g. within an agricultural decision support
system used to predict crop yield and quality) and (b) how the model will impact—
and be impacted by—the functioning of the wider sociotechnical environment thatthe tool is embedded within (e.g. a decision support tool used in healthcare forpatient triaging that may exacerbate existing health inequalities within the wider
community). Ensuring the model works within the proximate system can be acomplex programming and software engineering task, especially if it is expectedthat the model will be updated continuously in its runtime environment. But, more
importantly, understanding how to ensure the model’s sustainability given its
embeddedness in complex and changing sociotechnical environments requires
active and contextually-informed monitoring, situational awareness, and vigilant
responsiveness.

User Training
Although the performance of the model is evaluated in earlier stages, the model’s
impact cannot be entirely evaluated without consideration of the human factorsthat affect its performance in real-world settings. The impact of human cognitive
biases, such as algorithmic aversion 80 must also be considered, as such biases canlead to over- and under-reliance on the model (or system), in turn negating any
potential benefits that may arise from its use. Understanding the social andenvironmental context is also vital, as sociocultural norms may contribute to how
training is received, and how the system itself is evaluated (see Burton et al.
2020).

System Use and Monitoring
Depending on the context of deployment, it is likely that the performance of themodel could degrade over time. This process of model drift is typically caused byincreasing variation between how representative the training dataset was at the
There is some notable overlap between this stage of the project lifecyle and the ethicalassurance methodology, as some approaches to model reporting often contain similar
information that is used in building an ethical assurance case (Mitchell et al. 2019;
Ashmore, Calinescu, and Paterson 2019), specifically in the process of establishingevidential claims and warrant (see §§4.3.3-4.3.4).
80
Algorithmic aversion refers to the reluctance of human agents to incorporate algorithmic
tools as part of their decision-making processes due to misaligned expectations of thealgorithm’s performance (see Burton et al. 2020).
79

309

time of development and how representative it is at later stages, perhaps due tochanging social norms (e.g. changing patterns of consumer spending, evolving
linguistic norms that affect word embeddings). As such, mechanisms formonitoring the model’s performance should be instantiated within the system’s
runtime protocols to track model drift, and key thresholds should be determinedat early stages of a project (e.g. during project planning or in initial impact
assessment) and revised as necessary based on monitoring of the system’s use.

Model Updating or De-provisioning
As noted previously, model updating can occur continuously if the architecture of
the system and context of its use allows for it. Otherwise, updating the model may
require either revisiting previous stages to make planned adjustments (e.g. model
selection and training), or if more significant alterations are required the extant
model may need to be entirely de-provisioned, necessitating a return to a newround of project planning.

310

Glossary
Access to effective remedy: A core component of human rights impact
assessments. This requires organisations to ensure appropriate steps are in place
to prevent, investigate, punish, and redress negative impacts. Stakeholder
engagement is required to ensure that remedies reflect the needs and interests ofimpacted stakeholders.
Accessibility: Ensuring that information about the processes that took place todesign, develop, and deploy an AI system are easily accessible by individuals. Thisnot only refers to suitable means of explanation (clear, understandable, andaccessible language) but also the mediums for delivery.
Accuracy and Performance Metrics: In machine learning, the accuracy of amodel is the proportion of examples for which it generates a correct output. This
performance measure is also sometimes characterised conversely as an error rateor the fraction of cases for which the model produces an incorrect output. As aperformance metric, accuracy should be a central component to establishing andnuancing the approach to safe AI. Specifying a reasonable performance level forthe system may also often require refining or exchanging of the measure of
accuracy. For instance, if certain errors are more significant or costly than others,a metric for total cost can be integrated into the model so that the cost of one
class of errors can be weighed against that of another
Adequate quantity and quality: This property involves assessing whether thedata available is comprehensive enough to address the problem set at hand.
Adequate quantity and quality should address sample size, representativeness,
and availability of features relevant to problem.
Arbitrary deprivation: Where a physical or expressive freedom, or the personal
security of a rightsholder is removed or withheld, such removal or withholding isarbitrary if it is not justified by a publicly legitimatised legal framework or social
practice that reflects the principles and intent of human rights doctrine. Where
such deprivation occurs through legislation or other policy instrument, hallmarksof legitimate deprivation include legal due process and the availability of aneffective remedy for affected rights-holders to contest the lawfulness of thedeprived freedom or security.
Attributable: Data should clearly demonstrate who observed and recorded it,
when it was observed and recorded, and who it is about.
Auditability: Whereas the notion of answerability responds to the question of
who is accountable for an automation supported outcome, the notion ofauditability answers the question of how the designers and implementers of AI
systems are to be held accountable. This aspect of accountability has to do withdemonstrating both the responsibility of design and use practices and thejustifiability of outcomes.
Automated Decision: The selection of an action or a recommendation madeusing computational processes. Automated decisions describe those that either
augment or replace decisional work typically performed by humans alone. Most

311

commonly, automated decisions are predictions about persons or conditions in theworld derived from machine learning analysis of data about past events and its
similarity to a given set of conditions.
Automated Decision System: "An automated decision system (ADS) augmentsor replaces human decision-making by using computational processes to produce
answers to questions either as discrete classifications (e.g., yes, no; male, female,non-binary; malignant, benign) or continuous scores (e.g., degree ofcreditworthiness, risk of crime occurrence, projected tumour growth). Most ADS
produce predictions about persons or conditions using machine learning and othercomputational logic by calculating the probability that a given condition is met.
Typically, an automated decision system is ""trained"" on historical data looking
for patterns of relationships between data points (e.g., the relationship between
barometer readings, ambient temperature, and snowfall). An automated decision
is made by comparing known patterns with existing inputs to estimate how closely
they match (e.g., weather prediction based on the similarity between today's
climate readings and those from the past). Examples of ADS include algorithms
that calculate credit scores and biometric recognition systems that attempt toidentify individual people based on physical traits, such as facial features.
Clear Provenance and data lineage: Clear provenance and data lineage
consists of records that are accessible and simultaneously detail how data wascollected and how it has been used and altered throughout the stages of preprocessing, modelling, training, testing, and deploying.
Complete: All recorded data requires an audit trail to show nothing has been
deleted or lost.
Concept drifts and shifts: Once trained, most machine learning systems operate
on static models of the world that have been built from historical data which havebecome fixed in the systems’ parameters. This freezing of the model before it isreleased ‘into the wild’ makes its accuracy and reliability especially vulnerable tochanges in the underlying distribution of data. When the historical data that havecrystallised into the trained model’s architecture cease to reflect the population
concerned, the model’s mapping function will no longer be able to accurately andreliably transform its inputs into its target output values. These systems canquickly become prone to error in unexpected and harmful ways. There has beenmuch valuable research done on methods of detecting and mitigating concept anddistribution drift, and you should consult with your technical team to ensure thatits members have familiarised themselves with this research and have sufficientknowledge of the available ways to confront the issue. In all cases, you should
remain vigilant to the potentially rapid concept drifts that may occur in thecomplex, dynamic, and evolving environments in which your AI project willintervene. Remaining aware of these transformations in the data is crucial for safe
AI, and your team should actively formulate an action plan to anticipate and to
mitigate their impacts on the performance of your system.
Consistent: Consistency includes ensuring data is chronological, i.e., has a date
and time stamp that is in the expected sequence.

312

Contemporaneous: Data should be recorded as it was observed, and at the time
it was executed.
Continuous learning: A type of machine learning where the underlying algorithm
or model are adjusted continuously while deployed.
Critical Functions: "Activities that are considered “critical” are economic andsocial activities of which “the interruption or disruption would have serious
consequences on: 1) the health, safety, and security of citizens; 2) the effectivefunctioning of services essential to the economy and society, and of thegovernment; or 3) economic and social prosperity more broadly” (OECD,
2019[2]); (OECD, 2019[4]). It is important to note that not all systems in a criticalsector are critical. For example, the administrative time tracking systems of a
hospital or a bank are not considered to be critical systems.
Data controller: The individual or organisation which, alone or jointly withothers, determines how and why personal data will be processed.
Data Protection Impact Assessment: "Where a type of processing in particularusing new technologies, and taking into account the nature, scope, context andpurposes of the processing, is likely to result in a high risk to the rights andfreedoms of natural persons, the controller shall, prior to the processing, carry out
an assessment of the impact of the envisaged processing operations on theprotection of personal data. Data Protection Impact Assessments should containat least: a systematic description of the envisaged processing operations and the
purposes of the processing, including, where applicable, the legitimate interest
pursued by the controller; an assessment of the necessity and proportionality ofthe processing operations in relation to the purposes; an assessment of the risks
to the rights and freedoms of data subjects referred to in paragraph 1; and themeasures envisaged to address the risks, including safeguards, security measuresand mechanisms to ensure the protection of personal data and to demonstratecompliance with this GDPR regulation taking into account the rights and legitimate
interests of data subjects and other persons concerned."
Data subject: An identified or identifiable natural person, who can be identified,directly or indirectly, by information such as a name or identity number, or by a
combination of characteristics specific to that individual.
Discrimination: According to the UN Human Rights Committee overseeing theInternational Covenant on Civil and Political Rights, discrimination “should beunderstood to imply any distinction, exclusion, restriction or preference which isbased on any ground such as race, colour, sex, language, religion, political orother opinion, national or social origin, property, birth or other status, and which
has the purpose or effect of nullifying or impairing the recognition, enjoyment or
exercise by all persons, on an equal footing, of all rights and freedoms.” Other
definitions highlight the effects of systemic rather than explicit discrimination inwhich social and legal structures and/or historical conditions produce
discriminatory effects even where a specific intention to discriminate is notpresent.

313

Disproportionate Impact: Where a system produces effects distributed acrossgroups characterised by identity or demographic categories and such distributions
are experienced in greater or lesser degree by members of one or more group.
Disproportionality is most often considered when effects are especially harmful tothe interests or life chances of marginalised groups.
Duty-Bearers: Actors who have human rights duties or responsibilities towards
rights-holders. These include the company operating a project or conducting its
activities, business suppliers and contractors, joint-venture and other business
partners, state actors such as local government authorities and regional andnational government departments and agencies.
Equality Impact Assessment: A formal evaluative process of identifying
rightsholders whose right to equal status in society, access to opportunities, or
treatment may be adversely affected by a technical system, policy, or practice inrelation to other rightsholders, the extent to which they are affected, and theavailability of redress or remediation.
European Social Charter: A Council of Europe treaty that guarantees
fundamental social and economic rights as a counterpart to the European
Convention on Human Rights, which refers to civil and political rights. It
guarantees a broad range of everyday human rights related to employment,
housing, health, education, social protection and welfare.
Feature engineering: Features are the input variables that are used duringmodel training and are transformed by an algorithm. For example, a house priceestimation tool used by mortgage valuators may use the following features,'number of rooms', 'location', 'age of building', 'parking space' and 'size (sq. ft)'to predict the market value of the property. Features are often selected on thebasis of prior domain knowledge but can also be engineered on the basis of theirpredictive or explanatory value. For instance, a machine learning algorithm could
be fed raw data and trained to extract the most predictive features. In these cases,
the features may not be easily recognisable using traditional concepts or labels.
Flourish: Human flourishing is a term used to refer to self-actualization or anindividual’s achievement of a desirable life goal and is most typically associatedwith well-being. For some theories of well-being, flourishing is a goal-oriented
process that is concerned with an individual’s psychology or emotional state. For
other theories, the term refers to a broader social good that depends on thecultivation of social relationships.
Fundamental Rights and Freedoms: The first international standard for
fundamental rights and freedoms was established by the United Nation’s adoptionof the Universal Declaration of Human Rights (UDHR), a document establishingbasic rights aimed to safeguard the inviolable dignity of every person, which would
become the basis for the many treaties, conventions, and charters on humanrights that have been adopted worldwide up to the present. These treaties,
charters, and conventions include: The European Convention on Human Rights(ECHR), which placed obligations on governments to protect ordinary people
against human rights violations. The European Social Charter (ESC), which
extends basic rights to include social and economic rights covering health, working
314conditions, housing, migrant labour, gender equality, and social security, The
International Covenant on Economic, Social and Cultural Rights (ICESCR) (taking
together the International Covenant on Civil and Political Rights and theInternational Covenant on Economic, Social and Cultural Rights)included freedom
from torture, right to a fair trial, non-discrimination, and privacy rights, andextending basic rights to include rights to just working conditions, health, living
standards, education, and social security. The most recent document establishingfundamental rights and freedoms is the Charter of Fundamental Rights of theEuropean Union (CFR), which codified a basic set of civil, political, social,economic, and cultural rights for citizens of the European Union in EU law,
including rights pertaining to human dignity, fundamental freedoms, equality,
solidarity, and economic rights, and rights to participation in the life of thecommunity.
Harm to dignity: An innate worth or status that is not earned and cannot be
forfeited. The dignity of rightsholders is harmed where, as subjects of action or
inaction, they experience the involuntary diminishment of their fundamental
worth, self-respect, or personal autonomy, or else are treated as objects,
humiliated, or otherwise degraded.
Hidden proxy: A variable which represents some property or feature of an object
or person that is closely correlated with another (hidden) property or feature. For
instance, the variable 'profession' is likely to have a high degree of mutual
information with the variable 'salary'. In this example, the 'salary' variable is thehidden proxy if it is not directly recorded, but the 'profession' variable is present
in the dataset. Hidden proxies can be problematic when they represent protectedcharacteristics that were not intended to be in the dataset.
High-dimensional: In the context of data, dimensionality refers to the numberof features (or variables) within a dataset. High-dimensional, therefore, is used toindicate a large number of features (e.g., genetics datasets). This is not to be
confused with the size of the data as determined by the number of records (e.g.,number of people).
Highly Regulated: A domain of practice can be considered highly regulated
where multiple laws and policies are targeted specifically to a significant numberof potential activities and relations of the domain. By example, most countries
have highly regulated financial sectors with government agencies or departments
dedicated to the sector, strict guidelines for the conduct of business, a range ofconsumer protections, and substantial penalties for rule violation.
Human Rights Impact Assessment: A formal evaluative process of identifying
rightsholders whose human rights or fundamental freedoms may be adversely
affected by a technical system, policy, or practice, the extent to which they areaffected, and the availability of redress or remediation.
Human-in-the-loop design: The outputs of an AI model can be used as part ofa wider process in which a human considers the output, as well as otherinformation available to them, and then acts (makes a decision) based on this.

315

Informational plurality: A condition or environment where information (broadly
defined) is available from multiple voices and perspectives and/or where there are
multiple, differentiated sources of information.
Intelligibility/Understandability:
Capableofbeing
comprehended, not requiring specific technical expertise.

understoodorInterpretability: Consists of the ability to know how and why a model performed
the way it did in a specific context and therefore to understand the rationale behind
its decision or behaviour.
Labelling bias: Labelling (or annotation) bias is variation in the meaning of alabel used to represent a feature in training data. This commonly occurs when aspecific label does not mean the same thing for all data subjects or rights holders(e.g., an emotion label that is applied to images of faces). Variation in a label's
meaning is problematic when the specific label is strongly predictive of an
outcome, as this can create situations where certain groups are miscategorised or
receive inaccurate outcomes.
Legal basis: Derived from identifiable law or doctrine that can assert thelawfulness of an action or set of actions.
Major risk: A degree of risk in which the effects of a project or condition could be
devastating but may be manageable if enacted with extreme care and close
supervision.
Marginalisation: To treat a person, group, or concept as insignificant or
peripheral. In race and social justice discourse, members of society are
marginalised by holding an identity or being placed in a demographic categorythat is not attached to the dominant side of the prevailing structure of power andby which they experience oppression.
Mass deception: To deceive someone is to present (false) information as if itwere true and to elicit some related behaviour on the basis of this information.
Mass manipulation: To manipulate someone is to get them to act in a particularway, perhaps by withholding salient information or influencing how they evaluate
some state of the world.
Measurement errors: Any difference in the recorded value for some variable andits real-world value (e.g., measurement of the variable height). Measurement
error can occur as a result of poor data collection processes, badly calibrated
measuring devices, or from a range of biases. In the case of measurement bias,
error can arise when the measurement scale does not capture data pertaining to
the subjects in an equal manner (e.g., a self-reported pain scale that obscuresunderlying differences between two sub-groups).
Moderate risk: A degree of risk in which potential harms are directly or indirectly
associated with risks of adverse impacts on the human rights and fundamental
freedoms of affected persons, democracy, and the rule of law but that provide
opportunities for risk reduction and mitigation that make the risks posed broadly
acceptable.

316

Non-deterministic: A process is non-deterministic if its outcome cannot be
predicted (or determined), due to a lack of knowledge about how its causal
structure transforms the initial conditions. A non-deterministic algorithm is one
where an input variable can take on the same value but provide different results.
Non-Linear: A nonlinear system is one in which the change in the output is notdirectly proportional to the change of the input. As a result, the dynamics of thesystem cannot be represented graphically using a straight line.
Other Relevant Parties: Stakeholders other than rights-holders and dutybearers, which may include individuals or organisations representing the interestsof rights holders and official representations at international, national and local
levels (e.g., the UN, national human rights institutions, NGOs or civil society
organisations).
Performance Metrics: Metrics used to define how good the algorithm is atreaching the correct decision that range beyond accuracy measures.
Physical or Mental Integrity: The core attributes of an individual's overall bodily
integrity. Bodily integrity is an inviolable right that an individual has regarding theself-determination of their own body and mind.
Precision: The number of correctly identified positive predictions divided by the
number of all incorrect and correct positive predictions. Precision helps us tounderstand how accurate the model is – out of those predicted positive, how manyof them are actually positive? This is an especially useful metric when theimportance of false positives is high.
Pre-designated High-Risk or Safety Critical Sector: Annex III of theEuropean Union Proposed Rules on Artificial Intelligence indicates ‘high-risk’sectors as those concerned with education, emergency services, employment,
financial services, public benefits, law enforcement, immigration, border control,
and the administration of justice and democratic processes.
Probabilistic: An event is probabilistic if the extent to which it occurs is
considered to be subject to chance variation. In many cases, this is a reflection of
our own uncertainty about the event's occurrence, rather than assuming theexistence of a non-deterministic process.
Prohibited AI system: As outlined in the European Union Proposed Rules on
Artificial Intelligence (Art. 5), prohibited AI systems are those that include
manipulation, exploitation, social scoring, or real-time remote biometric ID.
Prohibitive Risk: A degree of risk significant enough to trigger the precautionary
principle and precipitate pre-emptive measures to prevent adverse impacts on thehuman rights and fundamental freedoms of affected persons, democracy, and therule of law. Pre-emptive measures are appropriate where the severity, scale, andirremediableness of the potential harm outweigh levels of risk reduction andmitigation. May also refer to a degree of risk in which the effects of a project orcondition would be so devastating upon their occurrence that it would be irrational,
irresponsible, or immoral to proceed.

317

Proportionality: A legal principle that refers to the idea of delivering a justoutcome in ways that are proportionate to the cost, complexity, and resources
available. In a similar vein, it can also be used as an evaluative notion, such as in
the case of a data protection principle that states only personal data that arenecessary and adequate for the purposes of the task are collected.
Protected Characteristics: Identities, categories, or other descriptive featuresof natural persons defined by human rights, civil rights, or data protection law, as
well as other doctrines concerned with legal rights and equal protection. Suchcharacteristics are ‘protected’ from arbitrary or capricious use in the allocation orwithholding of rights and social goods. Examples in European law and regulation
include racial or ethnic origin, political opinion, religion or beliefs, trade union
membership, genetic or health status, and sexual orientation. Other commonly
protected characteristics include skin tone, gender-identity, and cognitive or
physical ability.
Relevance, Appropriateness, and Domain Knowledge: The understanding
and utilisation of the most appropriate sources and types of data are crucial forbuilding a robust and unbiased AI system. Solid domain knowledge of theunderlying population distribution and of the predictive or classificatory goal of the
project is instrumental for choosing optimally relevant measurement inputs that
contribute to the reasonable determination of the defined solution. Domain
experts should collaborate closely with the technical team to assist in thedetermination of the optimally appropriate categories and sources ofmeasurement.
Reliability: The objective of reliability is that an AI system behaves exactly as its
designers intended and anticipated. A reliable system adheres to the specifications
it was programmed to carry out. Reliability is therefore a measure of consistency
and can establish confidence in the safety of a system based upon thedependability with which it operationally conforms to its intended functionality.
Rights-Holders: All individuals are human rights-holders. These include workers
and families, contractor (goods and services providers) employees and families,
impacted community members, including individuals of all genders, children,indigenous peoples, migrant workers, ethnic minorities and so forth (both within
the geographic vicinity of operations but also impacted downstream, transboundary or neighbouring communities), human rights defenders, and consumers.
Within the HUDERIA, the primary focus is on rights-holders who are, or may be,
adversely affected by a project.
Robustness: The objective of robustness can be thought of as the goal that an
AI system functions reliably and accurately under harsh conditions. These
conditions may include adversarial intervention, implementer error, or skewed
goal-execution by an automated learner (in reinforcement learning applications).
The measure of robustness is therefore the strength of a system’s integrity thesoundness of its operation in response to difficult conditions, adversarial attacks,
perturbations, data poisoning, and undesirable reinforcement learning behaviour.
Security: The goal of security encompasses the protection of several operational
dimensions of an AI system when confronted with possible adversarial attack. A
318

secure system is capable of maintaining the integrity of the information that
constitutes it. This includes protecting its architecture from the unauthorised
modification or damage of any of its component parts. A secure system also
remains continuously functional and accessible to its authorised users and keeps
confidential and private information secure even under hostile or adversarialconditions.
Sensitivity/Recall: The number of correct predictions divided by the number ofall samples that should have been identified as positive. Recall helps us tounderstand how many actual positives are captured through the model labelling it
positive. This is an especially useful metric when the importance of false negatives
is high.
Snowball Approach: An approach to sampling where a researcher ororganisation begins with a small number of contacts or stakeholders who havebeen identified as relevant. During initial engagement with these stakeholders’recommendations are sought for other potentially relevant stakeholders toengage. As this process continues the sample of stakeholders increases anddiversifies.
Socio-technical System: A system that couples human (or social) behaviour tothe functioning of a technical system, and in doing so gives rise to novel (andemergent) functions that are not reducible to either the human or technicalelements. By intervening in human behaviours, attitudes, or their relations to theworld, the technical system restructures human behaviour. The socio-technicalperspective is one that considers the human desires or goals a technology is meantto, or does, achieve. We can also describe as socio-technical those systems whosevery existence, implementation, or effects implicate human political, economic, orsocial relations. For example, surveillance systems adopted by law enforcement
agencies are socio-technical because their adoption and use have politicaldimensions; the selected targets of police surveillance are affected more acutely
than others by the use of surveillance technologies based on the historical choices
made by government and law enforcement officials. From this socio-technicalperspective, surveillance technologies participate in relations between people andthe centres of power in society.
Stakeholder: Individuals or groups that (1) have interests or rights that may beaffected by the past, present, and future decisions and activities of anorganisations; (2) may have the power or authority to influence the outcome ofsuch decisions and activities; (3) possess relevant characteristics that put them inpositions of advantage or vulnerability with regard to those decisions andactivities.
Target Variable: A feature within data defined as the output that a machinelearning model tries to predict. Each of the other features considered by the model
are used to predict the target variable. For example, for an ML model built topredict what tomorrow’s temperature may be, the target variable would betomorrow’s temperature, features in input data such as precipitation levels,
humidity, amongst others would be used by the model to predict this target
variable.

319

Timeliness and Recency (up-to-date): If datasets include outdated data, thenchanges in the underlying data distribution may adversely affect thegeneralisability of the trained model. Provided these distributional drifts reflectchanging social relationship or group dynamics, this loss of accuracy with regard
to the actual characteristics of the underlying population may introduce bias intoan AI system. In preventing discriminatory outcomes, timeliness and recency of
all elements of the data that constitute the datasets must be scrutinised.
Traceability: Refers to the process by which all stages of the data lifecycle fromcollection to deployment to system updating or deprovisioning are documented ina way that is accessible and easily understood. This may include not only theparties within the organisation involved but also the actions taken at each stage
that may impact the individuals who use the system.
Vulnerability: Refers to being “at a higher risk of being unable to anticipate, cope
with, resist, and recover from project-related risks and/or adverse impacts.” Suchrisk may be corporeal, as in being subject to heightened risk of physical or
psychological harm, or non-corporeal, as in being at heightened risk of diminishedeconomic or social status. Vulnerable individuals or groups may include women,children, the elderly, the poor, ethnic, religious, cultural or linguistic minorities, orindigenous groups.
Vulnerable Group: Refers a degree of risk from harm associated membership ina particular group or category. For example, people living in poverty are membersof a vulnerable group (i.e., “the poor”) because they share characteristics that put
them at greater risk of particular forms of harm than persons belonging to more
affluent groups.

320References
Abiteboul, S., & Stoyanovich, J., & Weikum, G. (2015). Data, Responsibly. ACM
Sigmod Blog. Retrieved from http://wp.sigmod.org/?p=1900
ACM US Public Policy Council. (2017). Statement on algorithmic transparency
and accountability. Retrieved fromhttps://www.acm.org/binaries/content/assets/publicpolicy/2017_usacm_statement _algorithms.pdf
Adadi, A., & Berrada, M. (2018). Peeking inside the black-box: A survey on
Explainable Artificial Intelligence (XAI). IEEE Access, 6, 52138-52160. Retrieved
from https://ieeexplore.ieee.org/abstract/document /8466590
AI Now Institute. (2018). Algorithmic Accountability Policy Toolkit. Retrieved
from https://ainowinstitute.org /aap-toolkit.pdf
Aizenberg, E., & van den Hoven, J. (2020). Designing for human rights in AI. Big
Data and Society. https://doi.org/10.1177/2053951720949566
Alper, P., Becker, R., Satagopam, V., Grouès, V., Lebioda, J., Jarosz, Y., … _&Schneider, R. (2018). Provenance-enabled stewardship of human data in the
GDPR era. https://doi.org/10.7490/f1000research.1115768.1
Ambacher, B., Ashley, K., Berry, J., Brooks, C., Dale, R. L., & Flecker, D. (2007).
Trustworthy repositories audit & certification: Criteria and checklist. Center for
Research Libraries, Chicago/Illinois. Retrieved fromhttps://www.crl.edu/sites/default/files/d6/attachments/pages/trac_0.pdf
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D.
(2016). Concrete problems in AI safety. arXiv:1606.06565. Retrieved fromhttps://arxiv.org/abs/1606.06565
Ananny, M., & Crawford, K. (2018). Seeing without knowing: Limitations of thetransparency ideal and its application to algorithmic accountability. New Media &Society, 20(3), 973-989. Retrieved fromhttps://journals.sagepub.com/doi/abs/10.1177/1461444816676645
Antignac, T., Sands, D., & Schneider, G. (2016). Data minimisation: A languagebased approach (long version). ArXiv:1611.05642. Retrieved fromhttp://arxiv.org/abs/1611.05642
Antunes, N., Balby, L., Figueiredo, F., Lourenco, N., Meira, W., & Santos, W.
(2018). Fairness and transparency of machine learning for trustworthy cloudservices. 2018 48th Annual IEEE/IFIP International Conference on Dependable
Systems and Networks Workshops (DSN-W), 188–193.
https://doi.org/10.1109/DSN-W.2018.00063
Auernhammer, K., Kolagari, R. T., & Zoppelt, M. (2019). Attacks on Machine
Learning: Lurking Danger for Accountability [PowerPoint Slides]. Retrieved fromhttps://safeai.webs.upv.es/wp-content/uploads /2019/02/3.SafeAI.pdf

321

Bathaee, Y. (2018). The artificial intelligence black box and the failure of intent
and causation. Harvard Journal of Law & Technology, 31(2), 889. Retrieved fromhttps://www.questia.com/library/journal/1G1-547758123/the-artificialintelligence-black-box-and-the-failure
Bibal, A., & Frénay, B. (2016). Interpretability of Machine Learning Models andRepresentations: an Introduction. Retrieved fromhttps://www.researchgate.net/profile/Adrien_Bibal/publication/326839249_Interpretability_of_Machine_Learning_Models_and_Representations_an_Introduction/links/5b6861caa6fdcc87df6d58e4/Interpretability-of-MachineLearning-Models-and-Representations-an-Introduction.pdf
Binns, R. (2017). Fairness in machine learning: Lessons from politicalphilosophy. arXiv:1712.03586. Retrieved from https://arxiv.org/abs/1712.03586
Binns, R. (2018). Algorithmic accountability and public reason. Philosophy &Technology, 31(4), 543-556. Retrieved fromhttps://link.springer.com/article/10.1007/s13347-017-0263-5
Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., & Shadbolt, N. (2018).
'It's reducing a human being to a percentage': Perceptions of justice inalgorithmic decisions. In Proceedings of the 2018 CHI Conference on Human
Factors in Computing Systems (p. 377). ACM. Retrieved from https://dl.acm.org/citation.cfm?id=3173951
Bracamonte, V. (2019). Challenges for transparent and trustworthy machinelearning [Power Point]. KDDI Research, Inc. Retrieved fromhttps://www.itu.int/en/ITU-T/Workshops-and-Seminars/20190121/Documents/Vanessa_Bracamonte_Presentation.pdf
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf,
H., Yang, J., Toner, H., Fong, R., Maharaj, T., Koh, P. W., Hooker, S., Leung, J.,
Trask, A., Bluemke, E., Lebensbold, J., O’Keefe, C., Koren, M., … Anderljung, M.
(2020). Toward Trustworthy AI Development: Mechanisms for Supporting
Verifiable Claims. ArXiv:2004.07213 [Cs]. http://arxiv.org/abs/2004.07213
Buhmann, A., Paßmann, J., & Fieseler, C. (2020). Managing Algorithmic
Accountability: Balancing Reputational Concerns, Engagement Strategies, and
the Potential of Rational Discourse. Journal of Business Ethics, 163.
https://doi.org/10.1007/s10551-019-04226-4
Burrell, J. (2016). How the machine ‘thinks’: Understanding opacity in machinelearning algorithms. Big Data & Society, 3(1).
https://doi.org/10.1177/2053951715622512
Calvo, R. A., Peters, D., & Cave, S. (2020). Advancing impact assessment for
intelligent systems. Nature Machine Intelligence, 1–3.
https://doi.org/10.1038/s42256-020-0151-z
Card, D. (2017). The “black box” metaphor in machine learning. Towards Data
Science. Retrieved from https://towardsdatascience.com/the-black-boxmetaphor-in-machine-learning-4e57a3a1d2b0

322

Caruana, R., Kangarloo, H., Dionisio, J. D., Sinha, U., & Johnson, D. (1999).
Case-based explanation of non-case-based learning methods. Proceedings. AMIA
Symposium, 212–215. Retrieved fromhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC2232607/
Cavoukian, A., Taylor, S., & Abrams, M. E. (2010). Privacy by Design: essential
for organizational accountability and strong business practices. Identity in the
Information Society, 3(2), 405–413. https://doi.org /10.1007/s12394-0100053-z
Cech, F. (2020). Beyond Transparency: Exploring Algorithmic Accountability.
Companion of the 2020 ACM International Conference on Supporting Group
Work, 11–14. https://doi.org/10.1145/3323994.3371015
Center for Democracy & Technology. (n.d.). Digital decisions. Retrieved fromhttps://cdt.org/issue /privacy-data/digital-decisions/Chen, C., Li, O., Tao, C., Barnett, A., Su, J., & Rudin, C. (2018). This looks like
that: deep learning for interpretable image recognition. arXiv:1806.10574.
Retrieved from https://arxiv.org/abs /1806.10574
Citron, D. K. (2008). Technological due process. Washington University Law
Review, 85(6). Retrieved from https://heinonline.org/hol-cgibin/get_pdf.cgi?handle=hein.journals/walq85&section=38
Citron, D. K., & Pasquale, F. (2014). The scored society: Due process forautomated predictions. Wash. L. Rev., 89, 1. Retrieved fromhttps://heinonline.org/HOL/LandingPage?handle=hein.journals/washlr89&div=4&id=&page=&t=1560014586
Cobbe, J., Lee, M. S. A., & Singh, J. (2021). Reviewable Automated DecisionMaking: A Framework for Accountable Algorithmic Systems. Proceedings of the2021 ACM Conference on Fairness, Accountability, and Transparency, 598–609.
https://doi.org/10.1145/3442188.3445921
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017).
Algorithmic decision making and the cost of fairness. ArXiv:1701.08230.
https://doi.org/10.1145/3097983.309809
Coston, A., Guha, N., Ouyang, D., Lu, L., Chouldechova, A., & Ho, D. E. (2021).
Leveraging Administrative Data for Bias Audits: Assessing Disparate Coveragewith Mobility Data for COVID-19 Policy. Proceedings of the 2021 ACM Conference
on Fairness, Accountability, and Transparency, 173–184.
https://doi.org/10.1145/3442188.3445881
Council of Europe. (2018). Convention 108 +: Convention for the protection ofindividuals with regard to the processing of personal data.
https://rm.coe.int/convention-108-convention-for-the-protection-of-individualswith-regar/16808b36f1
Council of Europe Commissioner for Human Rights. (2019). Unboxing Artificial
Intelligence: 10 steps to protect Human Rights. https://rm.coe.int/unboxingartificial-intelligence-10-steps-to-protect-human-rights-reco/1680946e64
323

Crawford, K., & Schultz, J. (2014). Big Data and due process: Toward aframework to redress predictive privacy harms. BCL Rev., 55, 93. Retrieved fromhttps://heinonline.org/HOL/LandingPage?handle=hein.journals/bclr55&div=5&id=&page=&t=1560014537
Custers, B. (2013). Data dilemmas in the information society: Introduction andoverview. In Discrimination and Privacy in the Information Society (pp. 3-26).
Springer, Berlin, Heidelberg. Retrieved fromhttps://link.springer.com/chapter/10.1007/978-3-642-30487-3_1
Custers, B. H., & Schermer, B. W. (2014). Responsibly innovating data mining
and profiling tools: A new approach to discrimination sensitive and privacy
sensitive attributes. In Responsible Innovation 1 (pp. 335-350). Springer,
Dordrecht. Retrieved from https://link.springer.com/chapter/10.1007/978-94017-8956-1_19
Dai, W., Yoshigoe, K., & Parsley, W. (2018). Improving data quality throughdeep learning and statistical models. ArXiv:1810.07132, 558, 515–522.
https://doi.org/10.1007/978-3-319-54978-1_66
Davidson, S. B., & Freire, J. (2008). Provenance and scientific workflows:
challenges and opportunities. In Proceedings of the 2008 ACM SIGMOD
international conference on Management of data (pp. 1345-1350). ACM.
Retrieved from https://dl.acm.org/citation.cfm?id=1376772
Demšar, J., & Bosnić, Z. (2018). Detecting concept drift in data streams using
model explanation. Expert Systems with Applications, 92, 546–559.
https://doi.org/10.1016/j.eswa.2017.10.003
Diakopoulos, N. (2014). Algorithmic Accountability Reporting: On theInvestigation of Black Boxes. Tow Center for Digital Journalism.
http://www.nickdiakopoulos.com/wp-content/uploads/2011/07/AlgorithmicAccountability-Reporting_final.pdf
Diakopoulos, N. (2015). Algorithmic Accountability: Journalistic investigation ofcomputational power structures. Digital Journalism, 3(3), 398–415.
https://doi.org/10.1080/21670811.2014.976411
Diakopoulos, N. (2016). Accountability in algorithmic decision making.
Communications of the ACM, 59(2), 56–62. https://doi.org/10.1145/2844110
Diakopoulos, N., Friedler, S., Arenas, M., Barocas, S., Hay, M., Howe, B., ... &Wilson, C. (2017). Principles for accountable algorithms and a social impact
statement for algorithms. FAT/ML. Retrieved fromhttps://www.fatml.org/resources/principles-for-accountable-algorithms
Donovan, J., Caplan, R., Hanson, L., & Matthews, J. (2018). Algorithmic
accountability: A primer. Data & Society Tech Algorithm Briefing: How
Algorithms Perpetuate Racial Bias and Inequality. Retrieved fromhttps://datasociety.net/output/algorithmic-accountability-a-primer/

324

Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretablemachine learning. arXiv:1702.08608. Retrieved fromhttps://arxiv.org/abs/1702.08608
Doshi-Velez, F., Kortz, M., Budish, R., Bavitz, C., Gershman, S., O'Brien, D., ...
& Wood, A. (2017). Accountability of AI under the law: The role of explanation.
arXiv:1711.01134. Retrieved from https://arxiv.org/abs/1711.01134
Dosilovic, F. K., Brcic, M., & Hlupic, N. (2018). Explainable artificial intelligence:
A survey. 2018 41st International Convention on Information andCommunication Technology, Electronics and Microelectronics (MIPRO), 0210–
0215. https://doi.org/10.23919/MIPRO.2018.8400040
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012). Fairness
through awareness. In Proceedings of the 3rd innovations in theoreticalcomputer science conference (pp. 214-226). ACM. Retrieved fromhttps://dl.acm.org/citation.cfm?id=2090255
Edwards, L., & Veale, M. (2017). Slave to the algorithm: Why a right to an
explanation is probably not the remedy you are looking for. Duke L. & Tech.
Rev., 16, 18. Retrieved from https://heinonline.org/HOL/LandingPage?handle=hein.journals/dltr16&div=3&id=&page=&t=1560014649
Eisenstadt, V., & Althoff, K. (2018). A Preliminary Survey of Explanation
Facilities of AI-Based Design Support Approaches and Tools. LWDA. Presented at
the LWDA. https://www.researchgate.net/profile/Viktor_Eisenstadt/publication/327339350_A_Preliminary_Survey_of_Exp
lanation_Facilities_of_AIBased_Design_Support_Approaches_and_Tools/links/5b891ecd299bf1d5a7338b
1a /A-Preliminary-Survey-of-Explanation-Facilities-of-AI-Based-Design-SupportApproaches-and-Tools.pdf
Esteves, A. M., Factor, G., Vanclay, F., Götzmann, N., & Moreira, S. (2017).
Adapting social impact assessment to address a project's human rights impacts
and risks. Environmental Impact Assessment Review, 67, 73-87
European Agency for Fundamental Rights. (2020). “Getting the future right:
Artificial Intelligence and fundamental rights.”
https://fra.europa.eu/sites/default/files/fra_uploads/fra-2020-artificialintelligence_en.pdf
European Commission Expert Group on FAIR Data. (2018). Turning FAIR into
reality. European Union. Retrieved fromhttps://ec.europa.eu/info/sites/info/files/turning_fair_into_reality_1.pdf
European Parliamentary Research Service. (2019). A governance framework foralgorithmic accountability and transparency.
https://www.europarl.europa.eu/RegData/etudes/STUD/2019/624262/EPRS_ST
U(2019)624262(ANN1)_EN.pdf

325

Faundeen, J. (2017). Developing criteria to establish trusted digital repositories.
Data Science Journal, 16. Retrieved fromhttps://datascience.codata.org/article/10.5334/dsj-2017-022/
FAT/ML. (2016). Principles for Accountable Algorithms and a Social Impact
Statement for Algorithms: FAT ML. https://www.fatml.org/resources/principlesfor-accountable-algorithms
Feldmann, F. (2018). Measuring machine learning model interpretability.
Retrieved from https://hci.iwr.uniheidelberg.de/system/files/private/downloads/860270201/felix_feldmann_eml20
18_report.pdf
Fink, K. (2018). Opening the government’s black boxes: Freedom of information
and algorithmic accountability. Information, Communication & Society, 21(10),1453–1471. https://doi.org/10.1080/1369118X.2017.1330418
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. Explaining
explanations: An approach to evaluating interpretability of machine.
arXiv:1806.00069. Retrieved from https://arxiv.org/abs /1806.00069
Grgić-Hlača, N., Zafar, M.B., Gummadi, K.P., & Weller, A. (2017). On Fairness,
Diversity, and Randomness in Algorithmic Decision Making. arXiv:1706.10208.
Retrieved from https://arxiv.org/abs/1706.10208
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D.
(2018). A survey of methods for explaining black box models. ACM computing
surveys (CSUR), 51(5), 93. Retrieved fromhttps://dl.acm.org/citation.cfm?id=3236009
Google. (2019). Perspectives on issues in AI governance. Retrieved fromhttps://ai.google/static /documents/perspectives-on-issues-in-ai-governance.pdf
Göpfert, J. P., Hammer, B., & Wersing, H. (2018). Mitigating concept drift via
rejection. In International Conference on Artificial Neural Networks (pp. 456467). Springer, Cham. https://doi.org/10.1007/978-3-030-01418-6_45
Götzmann, N., Bansal, T., Wrzoncki, E., Veiberg, C. B., Tedaldi, J., & Høvsgaard,
R. (2020). Human rights impact assessment guidance and toolbox. The Danish
Institute for Human Rights.
https://www.humanrights.dk/sites/humanrights.dk/files/media/dokumenter/busi
ness/hria_toolbox/hria_guidance_and_toolbox_final_feb2016.pdf
Hajian, S., Bonchi, F., & Castillo, C. (2016). Algorithmic bias: Fromdiscrimination discovery to fairness-aware data mining. In Proceedings of the
22nd ACM SIGKDD international conference on knowledge discovery and data
mining (pp. 2125-2126). ACM. Retrieved fromhttps://dl.acm.org/citation.cfm?id=2945386
Halbertal, M. (2015). "Three Concepts of Human Dignity.” Dewey Lectures. 7.
https://chicagounbound.uchicago.edu/dewey_lectures/7326

Hamilton, K., Karahalios, K., Sandvig, C., & Eslami, M. (2014). A path tounderstanding the effects of algorithm awareness. Proceedings of the Extended
Abstracts of the 32nd Annual ACM Conference on Human Factors in Computing
Systems - CHI EA ’14, 631–642. https://doi.org/10.1145/2559206.2578883
High-level Expert Group on Artificial Intelligence. (2020). The Assessment List
for Trustworthy Artificial Intelligence (ALTAI) for self-assessment. European
Commission. https://op.europa.eu/en/publication-detail/-/publication/73552fcdf7c2-11ea-991b-01aa75ed71a1/language-es
Holstein, K., Vaughan, J. W., Daumé III, H., Dudík, M., & Wallach, H. (2018).
Improving fairness in machine learning systems: What do industry practitioners
need?. ArXiv:1812.05239. https://doi.org/10.1145 /3290605.3300830
Hutchinson, B., Smart, A., Hanna, A., Denton, E., Greer, C., Kjartansson, O.,
Barnes, P., & Mitchell, M. (2021). Towards Accountability for Machine Learning
Datasets: Practices from Software Engineering and Infrastructure. Proceedingsof the 2021 ACM Conference on Fairness, Accountability, and Transparency,
560–575. https://doi.org/10.1145/3442188.3445918
ICO. (2017). Big Data, artificial intelligence, machine learning and dataprotection. Retrieved from https://ico.org.uk/media/fororganisations/documents/2013559/big-data-ai-ml-and-data-protection.pdf
ICO. (2020). Guidance on the AI auditing framework. Information
Commissioner’s Office. https://ico.org.uk/media/about-theico/consultations/2617219/guidance-on-the-ai-auditing-framework-draft-forconsultation.pdf
ICO & ATI (2020). Explaining decisions made with AI. https://ico.org.uk/fororganisations/guide-to-data-protection/key-data-protection-themes/explainingdecisions-made-with-ai/
ICO. (2021). Guide to the UK General Data Protection Regulation (UK GDPR).
https://ico.org.uk/media/for-organisations/guide-to-data-protection/guide-tothe-general-data-protection-regulation-gdpr-1-1.pdf
Irving, G., & Askell, A. (2019). AI safety needs social scientists. Distill, 4(2).
https://doi.org/10.23915 /distill.00014
Janssen, M., & Kuk, G. (2016). The challenges and limits of Big Data algorithms
in technocratic governance. Government Information Quarterly, 33(3), 371–377.
https://doi.org/10.1016/j.giq.2016.08.011
Kacianka, S., & Pretschner, A. (2021). Designing Accountable Systems.
Proceedings of the 2021 ACM Conference on Fairness, Accountability, andTransparency, 424–437. https://doi.org/10.1145/3442188.3445905
Kaminski, M. E. (2018). Binary Governance: Lessons from the GDPR’s approach
to algorithmic accountability. S. Cal. L. Rev., 92, 1529.
Kamiran, F., & Calders, T. (2012). Data preprocessing techniques forclassification without discrimination. Knowledge and Information Systems,327

33(1), 1-33. Retrieved from https://link.springer.com/article/10.1007/s10115011-0463-8
Katell, M., Young, M., Dailey, D., Herman, B., Guetler, V., Tam, A., Bintz, C.,
Raz, D., & Krafft, P. M. (2020). Toward situated interventions for algorithmic
equity: Lessons from the field. Proceedings of the 2020 Conference on Fairness,
Accountability, and Transparency, 45–55.
https://doi.org/10.1145/3351095.3372874
Kaufmann, P., Kuch, H., Neuhäuser, C., & Webster, E. (2011). Humiliation,
degradation, dehumanization: Human dignity violated.
Springer. https://www.corteidh.or.cr/tablas/r30885.pdf
Kemper, J., & Kolkman, D. (2019). Transparent to whom? No algorithmic
accountability without a critical audience. Information, Communication &Society, 22(14), 2081–2096. https://doi.org/10.1080/1369118X.2018.1477967
Kernell, E. L., Veiberg, C. B., & Jacquot, C. (2020). “Guidance on Human Rights
Impact Assessment of Digital Activities: Introduction.” The Danish Institute for
Human Rights.
https://www.humanrights.dk/sites/humanrights.dk/files/media/document/A%20
HRIA%20of%20Digital%20Activities%20-%20Introduction_ENG_accessible.pdf
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017).
Human decisions and machine predictions. The Quarterly Journal of Economics.
https://doi.org/10.1093/qje/qjx032
Kohli, P., Dvijotham, K., Uesato, J., & Gowal, S. (2019). Towards a robust andverified AI: Specification testing, robust training, and formal verification.
DeepMind Blog. Retrieved from https://deepmind.com/blog /robust-and-verifiedai/
Kolter, Z., & Madry, A. (n.d.). Materials for tutorial adversarial robustness:
Theory and practice. Retrieved from https://adversarial-ml-tutorial.org/
Kroll, J. A. (2018). The fallacy of inscrutability. Philosophical Transactions of theRoyal Society A: Mathematical, Physical and Engineering Sciences, 376(2133),
20180084. https://doi.org/10.1098/rsta.2018.0084
Kroll, J. A., Huey, J., Barocas, S., Felten, E. W., Reidenberg, J. R., Robinson, D.
G., & Yu, H. (2016). Accountable algorithms. U. Pa. L. Rev., 165, 633. Retrieved
from https://heinonline.org/HOL/LandingPage?handle=hein.journals/pnlr165&div=20&id=&page=&t=1559932490
Kusner, M. J., Loftus, J., Russell, C., & Silva, R. (2017). Counterfactual fairness.
In Advances in Neural Information Processing Systems (pp. 4066-4076).
Retrieved from http://papers.nips.cc/paper /6995-counterfactual-fairness
L’heureux, A., Grolinger, K., Elymany, H. F., & Capretz, M. A. (2017). Machinelearning with Big Data: Challenges and approaches. IEEE Access, 5, 7776-7797.
Retrieved from https://ieeexplore.ieee .org/abstract/document/7906512/

328

Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016). Interpretable decision sets: A
joint framework for description and prediction. In Proceedings of the 22nd ACM
SIGKDD international conference on knowledge discovery and data mining (pp.
1675-1684). ACM. Retrieved from https://dl.acm.org /citation.cfm?id=2939874
Lehr, D., & Ohm, P. (2017). Playing with the data: What legal scholars should
learn about machine learning. UCDL Rev., 51, 653. Retrieved fromhttps://lawreview.law.ucdavis.edu/issues /51/2/Symposium/51-2_Lehr_Ohm.pdf
Lepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018). Fair,transparent, and accountable algorithmic decision-making processes. Philosophy& Technology, 31(4), 611-627. https://doi.org/10.1007/s13347-017-0279-x
Leslie, D. (2019). Understanding artificial intelligence ethics and safety. The Alan
Turing Institute. https://www.turing.ac.uk/sites/default/files/201906/understanding_artificial_intelligence_ethics_and_safety.pdf
Lipton, Z. C. (2016). The mythos of model interpretability.arXiv:1606.03490.
Retrieved from https://arxiv.org/abs/1606.03490
Lipton, Z. C., & Steinhardt, J. (2018). Troubling trends in machine learningscholarship. arXiv:1807.03341. Retrieved fromhttps://arxiv.org/abs/1807.03341
Loi, M., Mätzener, A., Müller, A., & Spielkamp, M. (2021). Automated DecisionMaking Systems in the Public Sector: An Impact Assessment Tool for Public
Authorities. Algorithm Watch. https://algorithmwatch.org/en/wpcontent/uploads/2021/06/ADMS-in-the-Public-Sector-Impact-Assessment-ToolAlgorithmWatch-June-2021.pdf
Lundberg, S., & Lee, S.-I. (2017). A unified approach to interpreting model
predictions. ArXiv:1705.07874. Retrieved from http://arxiv.org/abs/1705.07874
Mahajan, V., Venugopal, V. K., Murugavel, M., & Mahajan, H. (2020). The
Algorithmic Audit: Working with Vendors to Validate Radiology-AI Algorithms—
How We Do It. Academic Radiology, 27(1), 132–135. Scopus.
https://doi.org/10.1016/j.acra.2019.09.009
Malgieri, G., & Comandé, G. (2017). Why a right to legibility of automateddecision-making exists in the general data protection regulation. International
Data Privacy Law. Retrieved from https://academic.oup.com/idpl/articleabstract/7/4/243/4626991?redirectedFrom=fulltext
Mantelero, A., & Esposito, M.S. (2021). An evidence-based methodology forhuman rights impact assessment (HRIA) in the development of AI data-intensive
systems. Computer Law & Security Review, 41.
https://doi.org/10.1016/j.clsr.2021.105561.
Marcus, G. (2018). Deep learning: A critical appraisal. arXiv:1801.00631.
Retrieved from https://arxiv.org/abs /1801.00631
McGregor, L., Murray, D., & Ng, V. (2019). International human rights law as aframework for algorithmic accountability. International and Comparative Law

329

Quarterly, 68(2), 309–343. Scopus.
https://doi.org/10.1017/S0020589319000046
Mittelstadt, B. D., Allo, P., Taddeo, M., Wachter, S., & Floridi, L. (2016). The
ethics of algorithms: Mapping the debate. Big Data & Society, 3(2),205395171667967. https://doi.org/10.1177/2053951716679679
Mittelstadt, B., Russell, C., & Wachter, S. (2019). Explaining explanations in AI.
In Proceedings of the conference on fairness, accountability, and transparency(pp. 279-288). ACM. Retrieved from https://dl.acm.org/citation.cfm?id=3287574
Molnar, C. (2018). Interpretable machine learning. A guide for making black box
models explainable. Leanpub. Retrieved fromhttps://christophm.github.io/interpretable-ml-book/
Moss, E., Watkins, E., Metcalf, J., & Elish, M. C. (2020). Governing with
Algorithmic Impact Assessments: Six Observations. SSRN Electronic Journal.
https://doi.org/10.2139/ssrn.3584818
Muñoz-González, L., Biggio, B., Demontis, A., Paudice, A., Wongrassamee, V.,
Lupu, E. C., & Roli, F. (2017, November). Towards poisoning of deep learningalgorithms with back-gradient optimization. In Proceedings of the 10th ACM
Workshop on Artificial Intelligence and Security (pp. 27-38). ACM. Retrieved
from https://dl.acm.org/citation.cfm?id=3140451
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019).
Interpretable machine learning: definitions, methods, and applications.
arXiv:1901.04592. Retrieved from https://arxiv.org/abs /1901.04592
Olhede, S. C., & Wolfe, P. J. (2018). The growing ubiquity of algorithms insociety: implications, impacts and innovations. Philosophical Transactions of theRoyal Society A: Mathematical, Physical and Engineering Sciences, 376(2128).
https://doi.org/10.1098/rsta.2017.0364
Nicolae, M. I., Sinn, M., Tran, M. N., Rawat, A., Wistuba, M., Zantedeschi, V., ...
& Edwards, B. (2018). Adversarial Robustness Toolbox v0.4.0.
arXiv:1807.01069. Retrieved from https://arxiv.org/abs /1807.01069
OECD. (2021). OECD Framework for the Classification of AI Systems – Public
Consultation on Preliminary Findings. https://oecd.ai/classification
Ortega, P. A., & Maini, V. (2018). Building safe artificial intelligence:
specification, robustness, and assurance. DeepMind Safety Research Blog,
Medium. Retrieved from https://medium.com/@deepmindsafetyresearch/building-safe-artificial-intelligence-52f5f75058f1
O’Sullivan, S., Neveians, N., Allen, C., Blyth, A., Leonard, S., Pagallo, U., … &Ashrafian, H. (2019). Legal, regulatory, and ethical frameworks for developmentof standards in artificial intelligence (AI) and autonomous robotic surgery. The
International Journal of Medical Robotics and Computer Assisted Surgery, 15(1),e1968. https://doi.org/10.1002/rcs.1968

330

Park, D. H., Hendricks, L. A., Akata, Z., Schiele, B., Darrell, T., & Rohrbach, M.
(2016). Attentive explanations: Justifying decisions and pointing to the evidence.
arXiv:1612.04757. Retrieved from https://arxiv.org/abs/1612.04757
Passi, S., & Barocas, S. (2019). Problem formulation and fairness. In
Proceedings of the Conference on Fairness, Accountability, and Transparency(pp. 39-48). ACM. Retrieved from https://dl.acm.org/citation.cfm ?id=3287567
Pedreschi, D., Giannotti, F., Guidotti, R., Monreale, A., Pappalardo, L., Ruggieri,
S., & Turini, F. (2018). Open the black box data-driven explanation of black box
decision systems. arXiv:1806.09936. Retrieved fromhttps://arxiv.org/abs/1806.09936
Pedreschi, D., Giannotti, F., Guidotti, R., Monreale, A., Ruggieri, S., & Turini, F.
(2019). Meaningful explanations of black box AI decision systems. AAAI Press.
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., &Wallach, H. (2018). Manipulating and measuring model interpretability.
ArXiv:1802.07810. Retrieved from http://arxiv.org/abs/1802.07810
Raji, I. D., Smart, A., & White, R. N. (2020). Closing the AI Accountability Gap:
Defining an End-to-End Framework for Internal Algorithmic Auditing. 12.
Ranjan, R., Sankaranarayanan, S., Castillo, C. D., & Chellappa, R. (2017).
Improving network robustness against adversarial attacks with compact
convolution. arXiv:1712.00699. Retrieved from https://arxiv.org/abs/1712.00699
Ratasich, D., Khalid, F., Geissler, F., Grosu, R., Shafique, M., & Bartocci, E.
(2019). A roadmap toward the resilient internet of things for cyber-physicalsystems. IEEE Access, 7, 13260-13283. Retrieved fromhttps://ieeexplore.ieee.org/abstract/document/8606923
Reddy, E., Cakici, B., & Ballestero, A. (2019). Beyond mystery: Putting
algorithmic accountability in context. Big Data & Society, 6(1),2053951719826856. https://doi.org/10.1177/2053951719826856
Reed, C. (2018). How should we regulate artificial intelligence?. PhilosophicalTransactions of the Royal Society A: Mathematical, Physical and Engineering
Sciences, 376(2128), 20170360. Retrieved fromhttps://royalsocietypublishing.org/doi/abs/10.1098/rsta.2017.0360
Reisman, D., Schultz, J., Crawford, K., & Whittaker, M. (2018). Algorithmic
Impact Assessments: A Practical Framework for Public Accountability (p. 22). AI
Now.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016b). Why should I trust you?:
Explaining the predictions of any classifier. In Proceedings of the 22nd ACM
SIGKDD international conference on knowledge discovery and data mining (pp.
1135-1144). ACM. Retrieved from https://dl.acm.org/citation.cfm?Id=2939778
Rosenbaum, H., & Fichman, P. (2019). Algorithmic accountability and digitaljustice: A critical assessment of technical and sociotechnical approaches.

331

Proceedings of the Association for Information Science and Technology, 56(1),237–244. https://doi.org/10.1002/pra2.19
Rosenblat, A., Kneese, T., & Boyd, D. (2014). Algorithmic Accountability. The
Social, Cultural & Ethical Dimensions of “Big Data,” March.
Rudin, C. (2018). Please stop explaining black box models for high stakes
decisions. arXiv:1811.10154. Retrieved from https://arxiv.org/abs/1811.10154
Rudin, C., & Ustun, B. (2018). Optimized scoring systems: Toward trust in
machine learning for healthcare and criminal justice. Interfaces, 48(5), 449-466.
https://doi.org/10.1287/inte.2018.0957
Ruggieri, S., Pedreschi, D., & Turini, F. (2010). DCUBE: Discrimination discovery
in databases. In Proceedings of the 2010 ACM SIGMOD International Conference
on Management of data (pp. 1127-1130). ACM. Retrieved fromhttps://dl.acm.org/citation.cfm?id=1807298
Salay, R., & Czarnecki, K. (2018). Using machine learning safely in automotive
software: An assessment and adaption of software process requirements in iso26262. arXiv:1808.01614. Retrieved from https://arxiv.org/abs/1808.01614
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J.
(2019). Fairness and abstraction in sociotechnical systems. In Proceedings of theConference on Fairness, Accountability, and Transparency (pp. 59-68). ACM.
Retrieved from https://dl.acm.org/citation.cfm?id=3287598
Shah, H. (2018). Algorithmic accountability. Philosophical Transactions of theRoyal Society A: Mathematical, Physical and Engineering Sciences, 376(2128),
20170362. https://doi.org/10.1098/rsta.2017.0362
Shaywitz, D. (2018). AI doesn’t ask why – But physicians and drug developers
want to know. Forbes. Retrieved fromhttps://www.forbes.com/sites/davidshaywitz/2018/11/09/ai-doesnt-ask-whybut-physicians-and-drug-developers-want-to-know/
Shi, Y., Erpek, T., Sagduyu, Y. E., & Li, J. H. (2018). Spectrum data poisoning
with adversarial deep learning. In MILCOM 2018-2018 IEEE MilitaryCommunications Conference (MILCOM) (pp. 407-412). IEEE. Retrieved fromhttps://ieeexplore.ieee.org/abstract/document/8599832/
Shmueli, G. (2010). To explain or to predict?. Statistical science, 25(3), 289310. Retrieved from https://projecteuclid.org/euclid.ss/1294167961
Simonite, T. (2017). AI experts want to end “black box” algorithms ingovernment. Wired Business, 10, 17. Retrieved fromhttps://www.wired.com/story/ai-experts-want-to-end-black-box-algorithms-ingovernment/
Singhal, S., & Jena, M. (2013). A study on WEKA tool for data preprocessing,classification and clustering. International Journal of Innovative technology andexploring engineering (IJItee), 2(6), 250-253. Retrieved from332https://pdfs.semanticscholar.org/095c/fd6f1a9dc6eaac7cc3100
a16cca9750ff9d8.pdf
SL Controls. (n.d.). What is ALCOA+ and Why Is It Important to Validation andData Integrity. https://slcontrols.com/en/what-is-alcoa-and-why-is-it-importantto-validation-and-data-integrity/
Sokol, K., & Flach, P. (2018). Glass-box: Explaining AI decisions withcounterfactual statements through conversation with a voice-enabled virtualassistant. Proceedings of the Twenty-Seventh International Joint Conference on
Artificial Intelligence, 5868–5870. https://doi.org/10.24963/ijcai.2018/865
Solow-Niederman, A., Choi, Y., & Van den Broeck, G. (2019). The InstitutionalLife of Algorithmic Risk Assessment. Berkeley Tech. LJ, 34, 705.
Song, Q., Jin, H., Huang, X., & Hu, X. (2018). Multi-Label Adversarial
Perturbations. In 2018 IEEE International Conference on Data Mining (ICDM)(pp. 1242-1247). IEEE. Retrieved from https://ieeexplore.ieee.org/abstract/document/8594975
Stahl, B. C., & Wright, D. (2018). Ethics and Privacy in AI and Big Data:
Implementing Responsible Research and Innovation. IEEE Security & Privacy,
16(3), 26–33. https://doi.org/10.1109/MSP.2018.2701164
Stoyanovich, J., Howe, B., Abiteboul, S., Miklau, G., Sahuguet, A., & Weikum, G.
(2017). Fides: Towards a platform for responsible data science. In Proceedingsof the 29th International Conference on Scientific and Statistical Database
Management (p. 26). ACM. Retrieved from https://dl.acm.org/citation.cfm?id=3085530
Suresh, H., & Guttag, J. V. (2019). A Framework for Understanding Unintended
Consequences of Machine Learning. arXiv:1901.10002. Retrieved fromhttps://arxiv.org/abs/1901.10002
Tagiou, E., Kanellopoulos, Y., Aridas, C., & Makris, C. (2019). A tool supported
framework for the assessment of algorithmic accountability. Int. Conf. Inf.,Intell., Syst. Appl., IISA. 10th International Conference on Information,
Intelligence, Systems and Applications, IISA 2019. Scopus.
https://doi.org/10.1109/IISA.2019.8900715
Turilli, M., & Floridi, L. (2009). The ethics of information transparency. Ethics
and Information Technology, 11(2), 105–112. https://doi.org/10.1007/s10676009-9187-9
UNESCO. (2021a). Intergovernmental Meeting of Experts (Category II) related toa Draft Recommendation on the Ethics of Artificial Intelligence, online, 2021.
https://unesdoc.unesco.org/ark:/48223/pf0000377898
UNESCO (20121b). Draft Text of the Recommendation on the Ethics of Artificial
Intelligence.
https://unesdoc.unesco.org/ark:/48223/pf0000377897?posInSet=4&queryId=N-EXPLORE-91253d60-b068-44ff-9831-7a1998978040

333

van der Aalst, W. M., Bichler, M., & Heinzl, A. (2017). Responsible data science.
Springer Fachmedien Wiesbaden. https://doi.org/10.1007/s12599-017-0487-z
Varshney, K. R., & Alemzadeh, H. (2017). On the safety of machine learning:
Cyber-physical systems, decision sciences, and data products. Big Data, 5(3),246-255. Retrieved from https://www.liebertpub.com/doi/abs/10.1089/big.2016.0051
Veale, M., Binns, R., & Edwards, L. (2018). Algorithms that remember: model
inversion attacks and data protection law. Philosophical Transactions of theRoyal Society A: Mathematical, Physical and Engineering Sciences, 376(2133).
https://doi.org/10.1098/rsta.2018.0083
Veale, M., Van Kleek, M., & Binns, R. (2018). Fairness and accountability designneeds for algorithmic support in high-stakes public sector decision-making. In
Proceedings of the 2018 CHI Conference on Human Factors in Computing
Systems (p. 440). ACM. Retrieved from https://dl.acm.org/citation.cfm?id=3174014
Verma, S., & Rubin, J. (2018). Fairness definitions explained. In 2018 IEEE/ACM
International Workshop on Software Fairness (FairWare) (pp. 1-7). IEEE.
Retrieved from https://ieeexplore.ieee.org /abstract/document/8452913
Wachter, S., Mittelstadt, B., & Floridi, L. (2017a). Transparent, explainable, andaccountable AI for robotics. Science Robotics, 2(6).
https://doi.org/10.1126/scirobotics.aan6080
Wachter, S., Mittelstadt, B., & Floridi, L. (2017b). Why a Right to Explanation ofAutomated Decision-Making Does Not Exist in the General Data Protection
Regulation. International Data Privacy Law, 7(2), 76–99.
https://doi.org/10.1093/idpl/ipx005
Warde-Farley, D., & Goodfellow, I. (2016). Adversarial perturbations of deepneural networks. In T. Hazan, G. Papandreou, & D. Tarlow (Eds.), Perturbations,
Optimization, and Statistics, 311. Cambridge, MA: The MIT Press.
Weller, A. (2017). Challenges for transparency. arXiv preprint arXiv:1708.01870.
Retrieved from https://arxiv.org/abs/1708.01870
Webb, G. I., Lee, L. K., Goethals, B., & Petitjean, F. (2018). Analyzing concept
drift and shift from sample data. Data Mining and Knowledge Discovery, 32(5),1179-1199. Retrieved from https://link.springer.com /article/10.1007/s10618018-0554-1
Wieringa, M. (2020). What to account for when accounting for algorithms: A
systematic literature review on algorithmic accountability. Proceedings of the2020 Conference on Fairness, Accountability, and Transparency, 1–18.
https://doi.org/10.1145/3351095.3372833
Young, M., Katell, M., & Krafft, P. M. (2019). Municipal surveillance regulation
and algorithmic accountability. Big Data & Society, 6(2), 2053951719868492.
https://doi.org/10.1177/2053951719868492

334

Zafar, M. B., Valera, I., Rodriguez, M. G., & Gummadi, K. P. (2015). Fairness
constraints: Mechanisms for fair classification. arXiv:1507.05259. Retrieved fromhttps://arxiv.org/abs/1507.05259
Zantedeschi, V., Nicolae, M. I., & Rawat, A. (2017). Efficient defenses against
adversarial attacks. In Proceedings of the 10th ACM Workshop on Artificial
Intelligence and Security (pp. 39-49). ACM. Retrieved fromhttps://dl.acm.org/citation.cfm?id=3140449
Zhao, M., An, B., Yu, Y., Liu, S., & Pan, S. J. (2018). Data poisoning attacks on
multi-task relationship learning. In Thirty-Second AAAI Conference on Artificial
Intelligence. Retrieved from https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/viewPaper/16073
Zhang, W. E., Sheng, Q. Z., Alhazmi, A., & Li, C. (2019). Adversarial attacks on
deep learning models in natural language processing: A survey. 1(1).
arXiv:1901.06796. https://arxiv.org/abs/1901.06796
Zicari, R. V., Brusseau, J., Blomberg, S. N., Christensen, H. C., Coffee, M.,
Ganapini, M. B., Gerke, S., Gilbert, T. K., Hickman, E., Hildt, E., Holm, S.,
Kühne, U., Madai, V. I., Osika, W., Spezzatti, A., Schnebel, E., Tithi, J. J., Vetter,D., Westerlund, M., … Kararigas, G. (2021). On Assessing Trustworthy AI in
Healthcare. Machine Learning as a Supportive Tool to Recognize Cardiac Arrestin Emergency Calls. Frontiers in Human Dynamics, 3, 673104.
https://doi.org/10.3389/fhumd.2021.673104
Žliobaitė, I. (2017). Measuring discrimination in algorithmic decision making.
Data Mining and Knowledge Discovery, 31(4), 1060-1089. Retrieved fromhttps://link.springer.com/article/10.1007/s10618-017-0506-1
Zook, M., Barocas, S., boyd, danah, Crawford, K., Keller, E., Gangadharan, S.
P., ...Paasquale, F. (2017) Ten simple rules for responsible Big Data research.
PLOS Computational Biology, 13(3). https://doi.org/10.1371/journal.pcbi.1005399

335