The Foundation for 

Best Practices in 

Machine Learning

Championing ethical and responsible machine 

learning through open-source best practices

Organisation Best Practices 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

Organisation Best Practices

from

The Foundation for Best Practices in Machine Learning

Release: 

PDF Version 1.0.0. 

19 May 2021

Notice: This document and its content has been licensed under the Creative Commons 

Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk 

number: 82610363). Any subsequent and/or other use, copying and/or adaptation of this 

document or its content must abide by the appropriate licensing terms & conditions as reflected 

thereunder. 

Stichting For Best Practices in Machine Learning

Leiden, the Netherlands

KvK Nummer: 82610363

https://www.fbpml.org/ 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

3

 FBPML Organisation Best Practices v1.0.0.

Foreword

The promise and value of machine learning is great, but it has been hastily operationalised over the past decade 

with often little regard for its wider societal impact, sometimes resulting in harmful and unfair consequences.

The aim of these Best Practices is to be easily accessible to anyone working on or interested in machine 

learning. This means that they are designed for a large audience who come from a variety of backgrounds and 

organisations. 

At the same time these Best Practices also aim to be complete. Although this means that they can be long at 

times, please do not be intimidated - read as much or as little as you feel comfortable with and come back later 

for more. The Best Practices are designed to be adaptable to different organisation sizes, needs, risks, resources, 

and expected societal impact and so the implementation can be flexible.

Creative Commons Licence

Because we want to lower the barriers to ethical and responsible machine learning, our Best Practices have been 

licensed under the Creative Commons Attribution license. This means they are freely available for commercial 

and/or private use and/or adaption, subject to attributing (i.e. referencing) The Foundation of Best Practices of 

Machine Learning of course.

Who are we?

We are a team of seasoned data scientists, machine learning engineers, AI ethicists and governance experts, who 

are enthusiastic about lowering the barriers for pragmatic ethical and responsible machine learning.

Best regard, The Board of FBPML

May, 2021

We, at The Foundation for Best Practices in Machine Learning, want to help data scientists, 

governance experts, management and other machine learning professionals champion ethical and 

responsible machine learning. We do this through championing our technical and organisational best 

practices for machine learning, through the free, open-source guidelines you are currently reading. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

4

 FBPML Organisation Best Practices v1.0.0.

Introduction ......................................................................................... 6

 

Part A ORGANISATION ......................................................................................... 11

Section 2. Managerial Oversight .......................................................................................... 12

Section 3. Internal Organisation Management & Oversight .................................................... 13

Part B PRODUCT AND MODEL MANAGEMENT

Section 4. Data Governance .......................................................................................... 18

Section 5. Product and Model Oversight & Management .................................................... 19

Section 6. Product Validation .......................................................................................... 31

Part C RESOURCE MANAGEMENT .......................................................................................... 32

Section 7. Human Resources Management ............................................................................. 33

Section 8. Asset Management .......................................................................................... 34

Section 9. Asset Management .......................................................................................... 35

Part D INCIDENT MANAGEMENT .......................................................................................... 36

Section 10. Incident Management .......................................................................................... 37

Part E ETHICS, PUBLIC INTEREST AND LEGAL ................................................................. 38

Section 11. Third Party Contracts Management ................................................................. 39

Section 12. Ethics & Transparency Management ................................................................. 40

Section 13. Compliance, Auditing & Legal Management and Oversight ....................................... 41

Contents
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

5

 FBPML Organisation Best Practices v1.0.0.

If you are not familiar yet with the Foundation for Best Practices in Machine Learning, and you want to know more 

about who we are, what we do, and what the philosophy and vision behind the Best Practices are, please visit our 

website.

On the next pages, you will find an explanation and overview of the structure of our Best Practices. Before diving 

straight into that, we would like to let you know about the following:

• These Best Practices are available through our Wiki-style portal too, and you’ll also be able to find and 

contribute additional supporting material there.

• These Best Practices are open-source and rely on community contributions for continuous improvements. To 

find out how to contribute please have a look at our contribution guide; 

• For tips on where to get started with implementing the Best Practices please have a look at our User Guides. 

Come back often, as we will be continuously adding new advice.

How to read the best practices

FBPML has two Best Practices documents. You are currently looking at the Organisation Best Practices. The core 

content of both Best Practices are the subjects you see below. 

Introduction
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

6

Introduction - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

The Technical Best Practices are scoped for a single 

product (which includes the ML models) and are aimed at 

helping your team best develop and maintain this product 

in an ethical and responsible way. The subjects within the 

Best Practices are approached through Product Lifecycle 

phases:

Each subject’s Best Practices are grouped by phases, so 

that the Risks and Controls are in the same order as you 

would typically encounter them during the first iteration 

of your product. Of course, during the lifecycle of your 

product, you will revisit each phase very often. Therefore, 

you will revisit the associated Best Practices too. 

The Organisation Best Practices are scoped 

for the entire organisation. It advises how to 

effectively support product teams within an 

organisation. This support is clustered around 

the core subjects mentioned above. These are 

approached through Policies. Management 

and governance aspects that are overarching 

receive attention as well.

About the wording

“Controls” and “Aims”

The Best Practices are written in a certain format wherein each “rule” consists of a number, name, Control and 

Aim. 

• The Controls are actions to take and can be understood as the instructions. The “what to do”; 

• the Aim is why you should do it, sometimes phrased as a goal, but more often as a risk. The “this is what can 

happen if you do not do it” and/or “this is why it is important”. 

“Product”

The Product is our word for the technical system around which the Best Practices revolve. It is used to refer to 

not only the data, the machine learning model and code, but also every component and process from start to 

finish that is required to produce the desired effect in practice - from UI to the protocols and processes that 

embed models in the organization and everything in between. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

7

 FBPML Organisation Best Practices v1.0.0.

1.1. Adversarial Action means actions characterised by mala fide (malicious) intent and/or bad 

faith. 

1.2. Assessment means the action or process of making a series of determinations and 

judgments after taking deliberate steps to test, measure and collectively 

deliberate the objects of concern and their outcomes. 

1.3. Assets means information technology hardware that concerns Products Machine 

Learning. 

1.4. Business Stakeholders means the departments and/or teams within the Organisation who do 

not conduct data science and/or technical Machine Learning, but have a 

material interest in Products Machine Learning.

1.5. Corporate Governance 

Principles

mean the structure of rules, practices and processes used to direct and 

manage a company in terms of industry recognised and published legal 

guidelines. 

1.6. Data Governance means the systems of governance and/or management over data assets 

and/or processes within an Organisation. 

1.7. Data Quality means the calibre of qualitative or quantitative data. 

1.8. Data Science means an interdisciplinary field that uses scientific methods, processes, 

algorithms and computational systems to extract knowledge and insights 

from structured and/or unstructured data.

1.9. Domain means the societal and/or commercial environment within which the 

Product will be and/or is operationalised. 

1.10. Ethical Practices means the ethical principles, values and/or practices that are 

encapsulated and promoted in an ‘artificial intelligence’ ethics guideline 

and/or framework, such as (a) The Asilomar AI Principles (Asilomar AI 

Principles, 2017), (b) The Montreal Declaration for Responsible AI (Montreal 

Declaration, 2017), (c) The Ethically Aligned Design: A Vision for Prioritizing 

Human Well-being with Autonomous and Intelligent Systems (IEEE, 2017), 

and/or (d) any other analogous guideline and/or framework. 

1.11. Ethics Committee means the committee within the Organisation charged with managing and/

or directing organisation Ethical Practices. 

1.12. Executive Management means the managerial team at the highest level of management within the 

Organisation.

1.13. Explainability means the property of Models and Model outcomes to be interpreted and/

or explained by humans in a comprehensible manner.

As used in this Best Practice Guideline, the following terms shall have the following meanings where capitalised. 

All references to the singular shall include references to the plural, where applicable, and vice versa. Any terms 

not defined or capitalised in this Best Practice Guideline shall hold their plain text meaning as cited in English and 

data science.

Section 1. Definitions
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

8

 1. Definitions - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

1.14. Fairness & NonDiscriminationmeans the property of Models and Model outcomes to be free from bias 

against protected classes. 

1.15. Best Practice Guideline means this document.

1.16. Guide means an established and clearly documented series of actions or 

process(es) conducted in a certain order or manner to achieve particular 

outcomes. 

1.17. Human-Centric Design 

& Redress

means orienting Products and/or Models to focus on humans and their 

environments through promoting human and/or environment centric 

values and resources for redress. 

1.18. Incident means the occurrence of a technical event that affects the integrity of a 

Product and/or Model. 

1.19. Machine Learning means the use and development of computer systems and Models that 

are able to learn and adapt with minimal explicit human instructions by 

using algorithms and statistical modelling to analyse, draw inferences, and 

derive outputs from data.

1.20. Model means Machine Learning algorithms and data processing designed, 

developed, trained and implemented to achieve set outputs, inclusive of 

datasets used for said purposes unless otherwise stated. 

1.21. Organisation means the concerned juristic entity designing, developing and/or 

implementing Machine Learning. 

1.22. Performance 

Robustness

means the propensity of Products and/or Models to retain their desired 

performance over diverse and wide operational conditions.

1.23. Policy means a documented course of normative actions or set of principles 

adopted to achieve a particular outcome. 

1.24. Procedure means an established and defined series of actions or process(es) 

conducted in a certain order or manner to achieve a particular outcome. 

1.25. Product means the collective and broad process of design, development, 

implementation and operationalisation of Models, and associated 

processes, to execute and achieve Product Definitions, inclusive of, inter 

alia, the integration of such operations and/or Models into organisation 

products, software and/or systems. 

1.26. Product Team means the collective group of Organisation employees directly charged 

with designing, developing and/or implementing the Product. 

1.27. Product Lifecycle means the collective phases of Products from initiation to termination 

- such as design, exploration, experimentation, development, 

implementation, operationalisation, and decommissioning - and their 

mutual iterations.

1.28. Product Owner means the employee charged with (a) managing and maximising the 

value of the Product and its Product Team; and (b) engaging with various 

Business Stakeholders concerning the Product and its Product Definitions. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

9

1. Definitions - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

1.29. Public means society at large.

1.30. Public Interest means the welfare or well-being of the Public.

1.31. Representativeness means the degree to which datasets and Models reflect the true 

distribution and conditions of Subjects, Subject populations, and/or 

Domains.

1.32. Safety & Security means (a) the resilience of Products and/or Models against malicious and/

or negligent activities that result in Organisational loss of control over 

concerned Products and/or Models; and (b) real Product Domain based 

physical harms that result through Products and/or Models applications

1.33. Social Corporate 

Responsibilities

means the structure of rules, practices and processes used to direct and 

manage a company in terms of industry recognised and published legal 

guidelines to positively contribute to economic, environmental and social 

progress.

1.34. Software means information technology software that concerns Products Machine 

Learning. 

1.35. Special Interest Groups means a specific body politic, or a particular collective of citizens, who can 

reasonably be determined to have a material interest in the Product. 

1.36. Specification means the accuracy, completeness and exactness of Products, Models 

and/or datasets in reflecting Product Definitions, Product Domains and/

or Product Subjects, either in their design and development and/or 

operationalisation. 

1.37. Subjects means the entities and/or objects that are represented as data points in 

datasets and/or Models, and who may be the subject of Product and/or 

Model outcomes.

1.38. Systemic Stability means the stability of Organisation, Domain, society and environments as 

a collective ecosystem. 

1.39. Traceability means the ability to trace, recount, and reproduce Product outcomes, 

reports, intermediate products, and other artifacts, inclusive of Models, 

datasets and codebases.

1.40. Transparency means the provision of an informed target audiences understanding of 

Organisation and/or Products Machine Learning, and their workings, based 

on documented Organisation information. 

1.41. Workflows means the coordinated and standardised sequences of employee work 

activities, processes, and tasks.
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

10

 FBPML Organisation Best Practices v1.0.0.

Part A

Organisation
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

11

 FBPML Organisation Best Practices v1.0.0.

Section 2. Managerial Oversight 

& Management

Objective

To ensure managerial direction and support for Products in accordance with Organisation strategies, business 

requirements, Corporate Governance Principles, Social Corporate Responsibilities, legal regulations and Ethical 

Practices. 

2.1 Management Direction for Machine Learning

Control: Aim:

2.1.1. Management 

Committee

A managerial committee ought to be 

established to (a) oversee Organisation 

Machine Learning and Products; and 

(b) warrant their effective alignment in 

accordance with Organisation strategies, 

business requirements, Corporate 

Governance Principles, Social Corporate 

Responsibilities, legal regulations and 

Ethical Practices.

To ensure clear managerial 

responsibility, oversight and 

custody of Organisation Machine 

Learning and Products.

11.1.2. Management 

Committee 

Diversity

The Management Committee ought to 

hold a diversity of members from differing 

Organisation departments, including 

Executive Management, legal, finance, 

operations, public communications as 

well as Data Science. 

To (a) ensure the diversity of 

managerial opinions and oversight 

of Organisation Machine Learning 

and Products; and (b) foster 

Organisation buy-in for Machine 

Learning and Products. 

11.1.3. Managerial 

Oversight 

Procedures 

The Management Committee should 

establish appropriate Procedures to 

warrant managerial oversight and 

governance of Organisation Products, 

inclusive of the appointment of Data 

Science Managers. 

To ensure the operationalisation 

of the oversight and management 

of Organisation Machine Learning 

and Products by the Management 

Committee. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

12

 FBPML Organisation Best Practices v1.0.0.

Section 3. Internal Organisation 

Management & Oversight

Objective

To establish managerial Procedures to control and oversee the design, development and implementation of 

Products.

3.1 Internal Organisation 

Control: Aim:

3.1.1. Data Science 

Managers 

The Management Committee should 

appoint Data Science Managers to oversee 

Products and warrant their effective 

alignment in accordance with the 

directives of the Management Committee, 

Policies, and, more broadly, Organisation 

strategies, business requirements, 

Corporate Governance Principles, 

Social Corporate Responsibilities, legal 

regulations and Ethical Practices.

To ensure the clear management, 

oversight, ownership and custody 

of Products.

3.1.2. Data Science 

Managers Products 

Ownership and 

Custody

The Management Committee ought to 

define and allocate to Data Science 

Managers Products. 

To ensure clear managerial 

oversight, ownership and custody 

of Products.

3.1.3. Data Science 

Managers 

Segregation of 

Duties

Conflicting duties and areas of 

responsibility of Data Science Managers 

should be segregated to reduce 

opportunities for the unauthorised and/or 

unintentional modification and/or misuse 

of Products. 

To reduce the threat of Product 

abuse, misuse and/or mala fide 

actions by Data Science Managers. 

3.1.4. Product Owners Data Science Managers ought to appoint 

Product Owners to (a) oversee specific 

Products and Product Teams; and (b) 

warrant their effective management in 

accordance with the directives of Data 

Science Managers, the Management 

Committee, and Organisation Policies.

To ensure the clear management, 

oversight, ownership and custody 

of a Product and its Product Team.

3.1.5. Product Owners 

Ownership and 

Custody

Data Science Managers ought to define 

and allocate to designated Product 

Owners Products and Product Teams. 

To ensure clear managerial 

oversight, ownership and custody 

of a Product and its Product Team.
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

13

3. Internal Organisation Management & Oversight - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

3.1.6. Product Owners 

Segregation of 

Duties

Conflicting duties and areas of 

responsibility of Product Owners should 

be segregated to reduce opportunities 

for the unauthorised and/or unintentional 

modification and/or misuse of a Product. 

To reduce the threat of Product 

abuse, misuse and/or mala fide 

actions by Product Owners. 

3.1.7. Product Teams Data Science Managers, in consultation 

with Product Owners, should define and 

allocate Products to designated Product 

Teams.

To ensure clear Product ownership 

and custody.

3.1.8. Product Definitions Data Science Managers, Product Owners, 

Business Stakeholders and, when relevant, 

Product employees ought to collectively 

document and define clear Product 

definitions, aims, internal deliverables and 

outcomes. 

To ensure Products have clear 

scopes to warrant (a) their 

effective oversight, management 

and execution, as well as (b) to 

allow for the accurate evaluation of 

Product risks and controls.

3.1.9. Approval of 

Product Definitions

The Management Committee should 

review and approve Product Definitions. 

To ensure managerial oversight of 

Products scopes.

3.1.10. Product Definitions 

Review

Product Definitions ought to be reviewed 

periodically, or if significant changes 

occur, by Data Science Managers, Product 

Owners, Business Stakeholders and, when 

relevant, Product employees. 

To ensure that Product Definitions 

are kept up-to-date to ensure 

their continued effectiveness, 

suitability, and accuracy. 

3.1.11. Product Risk 

Classification 

Policy

A Policy and Guide, which standarises 

the approaches to assessing Product 

risks, ought to be derived by Data Science 

Managers and approved by the Managerial 

Committee.

To ensure that (a) clear guidelines 

exist on how to evaluate and 

determine Product based-risks for 

subsequent evaluation in Product 

Risk Portfolios; and (b) Products 

are assigned risk-appropriate 

mandatory minimum capacity and 

oversight. 

3.1.12. Product Risk 

Classification 

Portfolio

Data Science Managers, Product Owners, 

Business Stakeholders and, when 

relevant, Product employees ought to 

collectively document and interrogate 

(a) Product Definitions and (b) Product 

design, development and implementation 

to identify Product based-risks and assign 

Product risk values and classifications. 

To ensure Products have clear 

risk portfolios to warrant (a) their 

effective oversight, management 

and execution, as well as (b) to 

allow for the accurate evaluation of 

Product risks and controls.

3.1.13. Approval of 

Product Risk 

Classification 

Portfolio

The Management Committee should 

review and approve Product Risk 

Portfolios. 

To ensure managerial oversight of 

Products risks.

3.1.14. Product Product 

Risk Classification 

Portfolio Review

The Product Risk Classification Portfolio 

ought to be continuously reviewed and 

developed by Data Science Managers, 

Product Owners, Business Stakeholders 

and, when relevant, Product employees. 

To ensure that Product Risk 

Portfolios are kept up-to-date 

to ensure their continued 

effectiveness, suitability, and 

accuracy. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

14

3. Internal Organisation Management & Oversight - FBPML Organisation Best Practices v1.0.0.

Objective

To establish Procedures to control the design, development and implementation of Products.

3.2 Product Management

Control: Aim:

3.2.1. Product Lifecycle 

Guide

Data Science Managers and, when 

relevant, Product Owners should derive 

a clear Product Lifecycle Guide for the 

Organisation. 

To ensure a clear organisational 

Product Lifecycle Guide to warrant 

the effective management and 

oversight of Machine Learning. 

3.2.2. Product Lifecycle 

and Workflow 

Descriptions 

Having consideration for the Product 

Lifecycle Policy, Product Definitions, and 

the Product Risk Classification Portfolio, 

Product workflows ought to be derived, 

developed, and documented by Data 

Science Managers, Product Owners and, 

when relevant, Product employees for 

each Product. 

To ensure clear Lifecycle and 

Workflows for Products to warrant 

their effective management and 

oversight. 

3.2.3. Reviewed of 

Product Lifecycle 

Guide

The Product Lifecycle Guide should be 

reviewed and approved by Data Science 

Managers and, when relevant, the 

Management Committee.

To ensure managerial oversight of 

the Product Lifecycle Guide.

3.2.4. Reviewed of 

Product Lifecycle 

and Workflow 

Description

Product Lifecycle and Workflow 

Descriptions should be reviewed and 

approved by Data Science Managers 

and, when relevant, the Management 

Committee.

To ensure managerial oversight of 

Product Lifecycle and Workflow 

Descriptions. 

3.2.5. Product Lifecycle 

and Workflow 

Procedures

Each Product ought to derive, develop 

and implement a set of Procedures to 

operationalise Product Lifecycle and 

Workflow Descriptions.

To ensure the operationalisation 

of Product Lifecycle and Workflow 

Descriptions. 

3.2.6. Reviewed of 

Product Lifecycle 

and Workflow 

Procedures

The Product Lifecycle and Workflow 

Procedures should be reviewed 

periodically, or if significant changes 

occur, by the Product Team to ensure their 

continued effectiveness, suitability, and 

accuracy.

To ensure that Product Product 

Lifecycle and Workflow Procedures 

are kept up-to-date. 

3.2.7. Product Employee 

Roles and 

Responsibilities

Data Science Managers and Product 

Owners ought to define and allocate 

to Product employees defined 

responsibilities and roles in terms 

of Product Lifecycle and Workflow 

Descriptions. 

To establish clear employee 

responsibilities and custodies in 

terms of Product Lifecycle and 

Workflow Descriptions. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

15

3. Internal Organisation Management & Oversight - FBPML Organisation Best Practices v1.0.0.

3.2.8. Data Science 

Managers Reports 

Frequent reports detailing Product 

progress, changes and risks ought to be 

made to the Management Committee 

by Data Science Managers and, 

subsequently, reviewed timeously.

To ensure the clear communication 

and management of Product 

deliverables to the Management 

Committee. 

3.2.9. Product Owners 

Reports 

Frequent reports detailing Product 

progress, changes and risks ought to be 

made to the Data Science Managers and 

Business Stakeholders by Product Owners 

and, subsequently, reviewed timeously.

To ensure the clear communication 

and management of Product 

deliverables to Data Science 

Managers and Business 

Stakeholders. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

16

Part B

 PRODUCT AND MODEL 

MANAGEMENT
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

17

 FBPML Organisation Best Practices v1.0.0.

Section 4. Data Governance

Objective

To ensure the integrity, normalisation, fairness and non-discrimination of Projet and/or Model data.

4. Data Governance

Control: Aim:

4.1. Data Governance 

Policy 

A Policy and Guide, which promotes good 

Data Governance in Product and Model 

design, development, and implementation 

ought to be derived by Data Science 

Managers and approved by the Managerial 

Committee. If a generic Data Governance 

Policy already exists, the above should be 

integrated accordingly.

To (a) ensure the integrity, 

normalisation, fairness and nondiscrimination of Product and/or 

Model data; and (b) provide clear 

Organisation guidance to Products 

on how to warrant data integrity, 

normalisation, fairness and nondiscrimination.

4.2. Data Governance 

Procedures

A set of Procedures to operationalise 

the Data Governance Policy should be 

developed and implemented within 

Products in light of Product Definitions, 

the Product Risk Classification Portfolio, 

and Product Lifecycle and Workflow 

Descriptions. 

To ensure the Data Governance of 

Products and Models.
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

18

 FBPML Organisation Best Practices v1.0.0.

Section 5. Product and Model 

Oversight & Management

Objective

To (a) identify possible risks for protected classes of persons, animals and the natural environment; and (b) 

minimise the unequal distribution of Products and Models errors to prevent reinforcing and/or deriving social 

inequalities and/or ills.

5.1 Fairness & Non-Discrimination

Control: Aim:

5.1.1. Fairness Policy A Policy and Guide, which promotes 

Product Fairness in - (a) Product 

Definitions and Product design, 

development, and implementation; (b) 

data processing; and (c) Model design, 

development, training, and output 

ought to be derived by Data Science 

Managers and approved by the Managerial 

Committee. 

To ensure the Fairness of Machine 

Learning. 

To provide clear Organisation 

guidance to Products on how 

to - (a) identify biases in Product 

data and Models; (b) take 

remedial action against identified 

biases; (c) identify and reduce 

asymmetric error rates between 

subpopulations; and (d) implement 

design and processes to avoid and 

circumvent risks that cannot be 

solved by purely technical means.

5.1.2. Product Fairness 

Procedures

A set of Procedures to operationalise the 

Fairness Policy should be developed and 

implemented within Products in light of 

Product Definitions, the Product Risk 

Classification Portfolio, and the Product 

Lifecycle and Workflow Descriptions.

To ensure the Fairness of Products 

and Models.

5.1.3. Fairness 

Assessments

Products should regularly complete 

Fairness assessments according 

to Product Lifecycle and Workflow 

Descriptions to the extent that is 

reasonably practical. Assessment 

findings ought to be documented by 

the Product Team and reviewed by Data 

Science Managers and, when relevant, the 

Management Committee. 

To analyse, test, and report 

the risks identified with, and 

measures taken to ensure, Product 

Fairness and non-discrimination 

at regular intervals within the 

Product Lifecycle and Workflow 

Description.

5.1.4. Review of the 

Fairness Policy 

The Fairness Policy should be reviewed 

periodically, or if significant changes 

occur, by Data Science Managers to 

ensure its continued effectiveness, 

suitability, and accuracy. 

To ensure that the Fairness Policy 

is kept up-to-date. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

19

5. Product and Model Oversight & Management - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

5.1.5. Review of 

Product Fairness 

Procedures

Product Fairness Procedures should be 

reviewed periodically, or if significant 

changes occur, by the Product Team to 

ensure their continued effectiveness, 

suitability, and accuracy. 

To ensure that Product Fairness 

Procedures are kept up-to-date. 

Objective

To ensure Data Quality and prevent unintentional effects, changes and/or deviations in Products and Models 

outputs associated with poor Product data. 

5.2 Data Quality

Control: Aim:

5.2.1. Data Quality Policy A Policy and Guide, which describes 

the assessment and remediation of an 

Organisation’s Data Quality. Chapters 

relating to (a) Product Definitions and 

Product design, development, and 

implementation; (b) data processing; and 

(c) Model design, development, training, 

and output ought to be derived and 

included by Data Science Managers and 

approved by the Managerial Committee. 

To ensure the Data Quality of 

Machine Learning. 

To provide Product Teams with 

reliable guidance on how to - (a) 

identify Data Quality risks; (b) take 

remedial actions against identified 

Data Quality risks; and (c) take 

steps to account for Data Quality 

risks and associated issues.

5.2.2. Product Data 

Quality Procedures

A set of Procedures to operationalise the 

Data Quality Policy should be developed 

and implemented within Products in light 

of Product Definitions, the Product Risk 

Classification Portfolio, and the Product 

Lifecycle and Workflow Descriptions . 

To ensure the Data Quality of 

Products and Models.

5.2.3. Data Quality 

Assessments

Products should regularly complete Data 

Quality assessments according to Product 

Lifecycle and Workflow Descriptions to 

the extent that is reasonably practical. 

Assessment findings ought to be 

documented by the Product Team and 

reviewed by Data Science Managers 

and, when relevant, the Management 

Committee. 

To analyse, test, and report the 

risks identified with, and measures 

taken to ensure, Product Data 

Quality at regular intervals within 

the Product Lifecycle and Workflow 

Description.

5.2.4. Review of the Data 

Quality Policy 

The Data Quality Policy should be reviewed 

periodically, or if significant changes 

occur, by Data Science Managers to 

ensure its continued effectiveness, 

suitability, and accuracy. 

To ensure that the Data Quality 

Policy is kept up-to-date. 

5.2.5. Review of Product 

Data Quality 

Procedures

Product Data Quality Procedures should 

be reviewed periodically, or if significant 

changes occur, by the Product Team to 

ensure their continued effectiveness, 

suitability, and accuracy. 

To ensure that Product Data Quality 

Procedures are kept up-to-date. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

20

5. Product and Model Oversight & Management - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

Objective

To (a) ensure that Product data and Models are representative of, and accurately specified for, target 

environments as far as is reasonably practical; and (b) guard against unintentional Products and Models 

behaviours and outputs as far as is reasonably practical. 

5.3 Representativeness & Specification

Control: Aim:

5.3.1. Representativeness & 

Specification Policy 

A Policy and Guide, which promotes 

Representativeness and Specification 

in - (a) Product Definitions and Product 

design, development, and implementation; 

(b) data processing; and (c) Model design, 

development, training, and output ought to 

be derived by Data Science Managers and 

approved by the Managerial Committee. 

To ensure the 

Representativeness and 

Specification of Machine 

Learning. 

To provide Product Teams with 

reliable guidance on how to - 

(a) identify Representativeness 

and Specification risks; (b) 

redress misspecification; and (c) 

remedy mis-, under- or overrepresentation risks.

5.3.2. Product 

Representativeness 

& Specification 

Procedures

A set of Procedures to operationalise the 

Representativeness & Specification Policy 

should be developed and implemented 

within Products in light of Product 

Definitions, the Product Risk Classification 

Portfolio, and the Product Lifecycle and 

Workflow Descriptions.

To ensure the 

Representativeness and 

Specification of Products and 

Models.

5.3.3. Representativeness 

& Specification 

Assessments

Products should regularly complete 

Representativeness & Specification 

assessments according to Product 

Lifecycle and Workflow Descriptions to 

the extent that is reasonably practical. 

Assessment findings ought to be 

documented by the Product Team and 

reviewed by Data Science Managers 

and, when relevant, the Management 

Committee. 

To analyse, test, and report 

the risks identified with, and 

measures taken to ensure, 

Product Representativeness and 

Specification at regular intervals 

within the Product Lifecycle and 

Workflow Description.

5.3.4. Review of the 

Representativeness & 

Specification Policy 

The Representativeness & Specification 

Policy should be reviewed periodically, 

or if significant changes occur, by Data 

Science Managers to ensure its continued 

effectiveness, suitability, and accuracy. 

To ensure that the 

Representativeness & 

Specification Policy is kept upto-date. 

5.3.5. Review of Product 

Representativeness 

& Specification 

Procedures

Product Representativeness & 

Specification Procedures should be 

reviewed periodically, or if significant 

changes occur, by the Product Team to 

ensure their continued effectiveness, 

suitability, and accuracy. 

To ensure that 

Representativeness & 

Specification Procedures are 

kept up-to-date. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

21

5. Product and Model Oversight & Management - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

Objective

To warrant Model outcomes and prevent unintentional Model behaviour a priori under operational conditions as 

far as is reasonably practical.

5.4 Performance Robustness

Control: Aim:

5.4.1. Performance 

Robustness Policy 

A Policy and Guide, which promotes 

Product Performance Robustness in - (a) 

Product Definitions and Product design, 

development, and implementation; (b) 

data processing; and (c) Model design, 

development, training, and output ought to 

be derived by Data Science Managers and 

approved by the Managerial Committee. 

To ensure the Performance and 

Robustness of Machine Learning. 

To provide Product Teams with 

reliable guidance on how to - 

(a) test, control, and improve 

Performance Robustness under 

operational conditions before 

going live; and (b) assess and 

control Performance Robustness 

risks concerning unexpected 

conditions.

5.4.2. Product 

Performance 

Robustness 

Procedures

A set of Procedures to operationalise the 

Performance Robustness Policy should 

be developed and implemented within 

Products in light of Product Definitions, 

the Product Risk Classification Portfolio, 

and the Product Lifecycle and Workflow 

Descriptions .

To ensure the Performance 

Robustness of Products and 

Models.

5.4.3. Performance 

Robustness 

Assessments

Products should regularly complete 

Performance Robustness Assessments 

according to Product Lifecycle and 

Workflow Descriptions to the extent 

that is reasonably practical. Assessment 

findings ought to be documented by 

the Product Team and reviewed by Data 

Science Managers and, when relevant, the 

Management Committee. 

To analyse, test, and report 

the risks identified with, and 

measures taken to ensure, 

Product Performance Robustness 

at regular intervals within the 

Product Lifecycle and Workflow 

Description.

5.4.4. Review of the 

Performance 

Robustness Policy 

The Performance Robustness Policy should 

be reviewed periodically, or if significant 

changes occur, by Data Science Managers 

to ensure its continued effectiveness, 

suitability, and accuracy. 

To ensure that the Performance 

Robustness Policy is kept up-todate. 

5.4.5. Review of Product 

Performance 

Robustness 

Procedures

Product Performance Robustness 

Procedures should be reviewed 

periodically, or if significant changes 

occur, by the Product Team to ensure their 

continued effectiveness, suitability, and 

accuracy. 

To ensure that Performance 

Robustness Procedures are kept 

up-to-date. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

22

5. Product and Model Oversight & Management - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

Objective

To ensure that Products and Models remain within acceptable operational bounds. 

5.5 Monitoring & Maintenance

Control: Aim:

5.5.1. Monitoring & 

Maintenance Policy 

A Policy and Guide, which promotes 

Product monitoring and maintenance 

in - (a) Product Definitions and Product 

design, development, and implementation; 

(b) data processing; and (c) Model design, 

development, training, and output ought to 

be derived by Data Science Managers and 

approved by the Managerial Committee. 

To ensure the monitoring and 

maintenance of Machine Learning. 

To provide Product Teams with 

reliable guidance on how to -

 (a) define, monitor and maintain 

acceptable operating bounds, 

including, inter alia, guarding 

against model drift; (b) define 

and review alert conditions 

and severity; and (c) create 

scenario playbooks regarding 

responsibility, escalation, rollback and resolution.

5.5.2. Product Monitoring 

& Maintenance 

Procedures

A set of Procedures to operationalise the 

Monitoring & Maintenance Policy should 

be developed and implemented within 

Products in light of Product Definitions, 

the Product Risk Classification Portfolio, 

and the Product Lifecycle and Workflow 

Descriptions. 

To ensure the monitoring and 

maintenance of Products and 

Models.

5.5.3. Monitoring & 

Maintenance 

Assessments

Products should regularly complete 

Monitoring & Maintenance assessments 

according to Product Lifecycle and 

Workflow Descriptions to the extent 

that is reasonably practical. Assessment 

findings ought to be documented by 

the Product Team and reviewed by Data 

Science Managers and, when relevant, the 

Management Committee. 

To analyse, test, and report 

the risks identified with, and 

measures taken to ensure, 

Product monitoring and 

maintenance at regular intervals 

within the Product Lifecycle and 

Workflow Description.

5.5.4. Review of the 

Monitoring & 

Maintenance Policy 

The Monitoring & Maintenance Policy 

should be reviewed periodically, or 

if significant changes occur, by Data 

Science Managers to ensure its continued 

effectiveness, suitability, and accuracy. 

To ensure that the Monitoring & 

Maintenance Policy is kept up-todate. 

5.5.5. Review of Product 

Monitoring & 

Maintenance 

Procedures

Product Monitoring & Maintenance 

Procedures should be reviewed 

periodically, or if significant changes 

occur, by the Product Team to ensure their 

continued effectiveness, suitability, and 

accuracy. 

To ensure that Monitoring & 

Maintenance Procedures are kept 

up-to-date. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

23

5. Product and Model Oversight & Management - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

Objective

To ensure Products and Models functions and outputs are explainable and justifiable as far as is practically 

reasonable.

5.6 Explainability

Control: Aim:

5.6.1. Explainability 

Policy 

A Policy and Guide, which promotes 

Product Explainability in - (a) Product 

Definitions and Product design, 

development, and implementation; (b) 

data processing; and (c) Model design, 

development, training, and output ought to 

be derived by Data Science Managers and 

approved by the Managerial Committee. 

To ensure the Explainability of 

Machine Learning. 

To provide clear Organisation 

guidance to Products on how 

to - (a) ensure the justifiability 

of individual Model predictions 

and decisions; (b) maintain and 

promote Product and Model 

inner workings amongst Product 

Teams, Business Stakeholders, 

Organisation Stakeholders 

and end-consumers; and (c) 

provide Model transparency for 

authorities, Special Interest 

Groups and/or the Public.

5.6.2. Product 

Explainability 

Procedures

A set of Procedures to operationalise the 

Explainability Policy should be developed 

and implemented within Products in light 

of Product Definitions, the Product Risk 

Classification Portfolio, and the Product 

Lifecycle and Workflow Descriptions. 

To ensure the Explainability of 

Products and Models.

5.6.3. Explainability 

Assessments

Products should regularly complete 

Explainability assessments according 

to Product Lifecycle and Workflow 

Descriptions to the extent that is 

reasonably practical. Assessment 

findings ought to be documented by 

the Product Team and reviewed by Data 

Science Managers and, when relevant, the 

Management Committee. 

To analyse, test, and report 

the risks identified with, and 

measures taken to ensure, 

Product Explainability at 

regular intervals within the 

Product Lifecycle and Workflow 

Description.

5.6.4. Review of the 

Explainability 

Policy 

The Explainability Policy should be 

reviewed periodically, or if significant 

changes occur, by Data Science Managers 

to ensure its continued effectiveness, 

suitability, and accuracy. 

To ensure that the Explainability 

Policy is kept up-to-date.

5.6.5. Review of Product 

Explainability 

Procedures

Product Explainability Procedures should 

be reviewed periodically, or if significant 

changes occur, by the Product Team to 

ensure their continued effectiveness, 

suitability, and accuracy. 

To ensure that Product 

Explainability Procedures are kept 

up-to-date. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

24

5. Product and Model Oversight & Management - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

Objective

To (a) prevent adversarial actions against, and encourage graceful failures for, Products and/or Models; (b) avert 

malicious extraction of Models, data and/or intellectual property; (c) prevent Model based physical or irreparable 

harms; and (d) prevent erosion of trust in outputs or methods.

5.7 Safety & Security

Control: Aim:

5.7.1. Safety & Security 

Policy 

A Policy and Guide, which promotes 

Product Safety & Security in - (a) 

Product Definitions and Product design, 

development, and implementation; (b) 

data processing; and (c) Model design, 

development, training, and output ought to 

be derived by Data Science Managers and 

approved by the Managerial Committee. 

To ensure the Safety of Machine 

Learning. 

To provide clear Organisation 

guidance to Products on how 

to identify and guard against 

Product vulnerabilities from 

Adversarial Actions.

5.7.2. Product Safety 

Procedures

A set of Procedures to operationalise the 

Safety Policy should be developed and 

implemented within Products in light of 

Product Definitions, the Product Risk 

Classification Portfolio, and the Product 

Lifecycle and Workflow Descriptions . 

To ensure the Safety of Products 

and Models.

5.7.3. Safety 

Assessments

Products should regularly complete 

Safety assessments according to Product 

Lifecycle and Workflow Descriptions to 

the extent that is reasonably practical. 

Assessment findings ought to be 

documented by the Product Team and 

reviewed by Data Science Managers 

and, when relevant, the Management 

Committee. 

To analyse, test, and report 

the risks identified with, and 

measures taken to ensure, 

Product Safety at regular intervals 

within the Product Lifecycle and 

Workflow Description.

5.7.4. Review of the 

Safety Policy 

The Safety Policy should be reviewed 

periodically, or if significant changes 

occur, by Data Science Managers to ensure 

its continued effectiveness, suitability, and 

accuracy. 

To ensure that the Safety Policy is 

kept up-to-date. 

5.7.5. Review of Product 

Safety Procedures

Product Safety Procedures should be 

reviewed periodically, or if significant 

changes occur, by the Product Team to 

ensure their continued effectiveness, 

suitability, and accuracy. 

To ensure that Product Safety 

Procedures are kept up-to-date. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

25

5. Product and Model Oversight & Management - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

Objective

To ensure (a) building desirable solutions; (b) human control over Products and Models; and (c) that individuals 

affected by Products and Models outputs can obtain redress.

Control: Aim:

5.8.1. Human-centric 

Design & Redress 

Policy 

A Policy and Guide, which promotes 

Product Human-Centric Design & Redress 

in - (a) Product Definitions and Product 

design, development, and implementation; 

(b) data processing; and (c) Model design, 

development, training, and output ought to 

be derived by Data Science Managers and 

approved by the Managerial Committee. 

To ensure the Human-Centric 

Design & Redress of Machine 

Learning. 

To provide clear Organisation 

guidance to Products on how to - 

(a) adds value and desirability of 

your product for end users; 

(b) assess and implement humanin-control requirements; and 

(c) assess and implement humancentric remediation and redress. 

5.8.2. Product HumanCentric Design 

& Redress 

Procedures

A set of Procedures to operationalise the 

Human-Centric Design & Redress Policy 

should be developed and implemented 

within Products in light of Product 

Definitions, the Product Risk Classification 

Portfolio, and the Product Lifecycle and 

Workflow Descriptions. 

To ensure the Human-Centric 

Design & Redress of Products and 

Models.

5.8.3. Human-Centric 

Design & Redress 

Assessments

Products should regularly complete 

Human-Centric Design & Redress 

assessments according to Product 

Lifecycle and Workflow Descriptions to 

the extent that is reasonably practical. 

Assessment findings ought to be 

documented by the Product Team and 

reviewed by Data Science Managers 

and, when relevant, the Management 

Committee. 

To analyse, test, and report 

the risks identified with, and 

measures taken to ensure, 

Product Human-Centric Design 

& Redress at regular intervals 

within the Product Lifecycle and 

Workflow Description.

5.8.4. Review of the 

Human-Centric 

Design & Redress 

Policy 

The Human-Centric Design Policy & 

Redress should be reviewed periodically, 

or if significant changes occur, by Data 

Science Managers to ensure its continued 

effectiveness, suitability, and accuracy. 

To ensure that the Human-Centric 

Design & Redress Policy is kept 

up-to-date. 

5.8.5. Review of Product 

Human-Centric 

Design & Redress 

Procedures

Product Human-centric Design & 

Redress Procedures should be reviewed 

periodically, or if significant changes 

occur, by the Product Team to ensure their 

continued effectiveness, suitability, and 

accuracy. 

To ensure that Product HumanCentric Design & Redress 

Procedures are kept up-to-date. 

5.8 Human-Centric Design & Redress
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

26

5. Product and Model Oversight & Management - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

Objective

To prevent (in)direct adverse social and environmental effects as a consequence of interactions amongst 

Products, Models, the Organisation, and the Public. 

Control: Aim:

5.9.1. Systemic Stability 

Policy

A Policy and Guide, which promotes 

awareness and control of systemic effects 

and interactions in - (a) Product Definitions 

and Product design, development, and 

implementation; (b) data processing; and 

(c) Model design, development, training, 

and output ought to be derived by Data 

Science Managers and approved by the 

Managerial Committee. 

To ensure the Systemic Stability 

of Machine Learning. 

To provide clear Organisation 

guidance to Products on how to 

identify, analyse and prevent risks 

derived from (higher-order and/or 

highly complex) relations between 

Products, Models, Product design, 

Organisation processes and 

society at large. 

5.9.2. Product Systemic 

Stability 

Procedures

A set of Procedures to operationalise 

the Systemic Stability Policy should be 

developed and implemented within 

Products in light of Product Definitions, 

the Product Risk Classification Portfolio, 

and the Product Lifecycle and Workflow 

Descriptions. 

To ensure the Systemic Stability 

of Products.

5.9.3. Systemic Stability 

Assessments

Products should regularly complete 

Systemic Stability according to Product 

Lifecycle and Workflow Descriptions to 

the extent that is reasonably practical. 

Assessment findings ought to be 

documented by the Product Team and 

reviewed by Data Science Managers 

and, when relevant, the Management 

Committee. 

To analyse, test, and report 

the risks identified with, and 

measures taken to ensure, 

Product Systemic Stability at 

regular intervals within the 

Product Lifecycle and Workflow 

Description.

5.9.4. Review of the 

Systemic Stability 

Policy

Systemic Stability should be reviewed 

periodically, or if significant changes 

occur, by Data Science Managers to ensure 

its continued effectiveness, suitability, and 

accuracy. 

To ensure that the Systemic 

Stability Policy is kept up-to-date.

5.9.5. Review of 

Systemic Stability 

Procedures

Product Systemic Stability should be 

reviewed periodically, or if significant 

changes occur, by the Product Team to 

ensure their continued effectiveness, 

suitability, and accuracy. 

To ensure that Product Systemic 

Stability Procedures are kept upto-date. 

5.9 Systemic Stability
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

27

5. Product and Model Oversight & Management - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

Objective

To ensure the clear and complete Traceability of Products, Models and their assets (inclusive of, inter alia, data, 

code, artifacts, output, and documentation) for as long as is reasonably practical. 

Objective

To ensure that Product decision-making is done in a clear, informed, unbiased and collaborative manner with a 

diversity of Organisation opinions, when relevant. 

Control: Aim:

5.10.1. Product 

Traceability Policy

A Policy and Guide, which promotes 

Product Traceability during and after - (a) 

Product Definitions and Product design, 

development, and implementation; (b) 

data processing; and (c) Model design, 

development, training, and output ought to 

be derived by Data Science Managers and 

approved by the Managerial Committee. 

To ensure the Product Traceability 

of Machine Learning. 

To provide clear Organisation 

guidance to Products on how to 

manage Product Traceability. 

5.10.2. Product 

Traceability 

Procedures

A set of Procedures to operationalise the 

Traceability Policy should be developed 

and implemented within Products in light 

of Product Definitions, the Product Risk 

Classification Portfolio, and the Product 

Lifecycle and Workflow Descriptions. 

To ensure the Product Traceability 

of Products.

5.10.3. Review of 

the Product 

Traceability Policy

The Product Traceability should be 

reviewed periodically, or if significant 

changes occur, by Data Science Managers 

to ensure its continued effectiveness, 

suitability, and accuracy. 

To ensure that the Product 

Traceability Policy is kept up-todate.

5.10.4. Review of Product 

Traceability 

Procedures

Product Product Traceability should be 

reviewed periodically, or if significant 

changes occur, by the Product Team to 

ensure their continued effectiveness, 

suitability, and accuracy. 

To ensure that Product Product 

Traceability Procedures are kept 

up-to-date. 

Control: Aim:

5.11.1. Product 

Decision-Making 

Policy

A Policy which promotes Product decisionmaking clarity in - (a) Product Definitions 

and Product design, development, and 

implementation; (b) data processing; 

(c) Model design, development, training, 

and output; and (d) consideration for the 

Product Risk Classification Portfolio ought 

to be derived by Data Science Managers 

and approved by the Managerial Committee. 

To ensure Product decisions - (a) are 

based on all available information, 

inclusive of those derived from 

Policies and Procedures of this 

document; (b) follow from and are 

aligned with the Product Lifecycle 

and Workflow Description; and (c) 

are made with reasonable care to 

avoid cognitive bias.

5.10 Product Traceability

5.11 Product Decision-Making
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

28

5. Product and Model Oversight & Management - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

5.11.2. Product 

Decision-Making 

Procedures 

A set of Procedures to operationalise the 

Decision-Making Policy within Products 

should be developed and implemented, 

inclusive of consultations with Business 

Stakeholders. 

To ensure clear, informed, unbiased 

and collaborative decision-making 

in Products.

5.11.3. Product 

Decision-Making 

Diversity

A diversity of Organisation stakeholder and 

Business Stakeholder opinions and input 

should be obtained, considered and, when 

relevant, weighed when making material 

Product decisions 

To ensure a diversity of opinions 

when making material Product 

decisions. 

5.11.4. Product 

Decision-Making 

Documentation

Product decisions should be documented, 

indexed, stored and, when relevant, 

reviewed. 

To ensure that Product decisions 

are documented and indexed to 

warrant their effective management 

and oversight.

Objective

To ensure that Products have sufficient capacity and capabilities to meet Product Definitions. 

Control: Aim:

5.12.1. Financial 

Resource 

Allocation 

Sufficient financial resources ought 

to be allocated to Products based on 

their Product Definitions, Product Risk 

Classification Portfolio, and the Product 

Lifecycle and Workflow Descriptions. 

To ensure that Products have 

sufficient financial resources to 

allow for their success. 

5.12.2. Human Resource 

Allocation

Sufficient human resources ought 

to be allocated to Products based on 

their Product Definitions, Product Risk 

Classification Portfolio, and the Product 

Lifecycle and Workflow Descriptions. 

To ensure that Products have 

sufficient human resources to allow 

for their success. 

5.12.3. Assets Allocation Sufficient Assets ought to be allocated 

to Products based on their Product 

Definitions, Product Risk Classification 

Portfolio, and the Product Lifecycle and 

Workflow Descriptions. 

To ensure that Products have 

sufficient Assets to allow for their 

success. 

5.12.4. Software 

Allocation

Sufficient Software ought to be allocated 

to Products based on their Product 

Definitions, Product Risk Classification 

Portfolio, and the Product Lifecycle and 

Workflow Descriptions. 

To ensure that Products have 

sufficient Software to allow for their 

success. 

5.12.5. Knowledge & 

Development

Product Teams should receive sufficient 

training and development based on 

the Product Definitions, Product Risk 

Classification Portfolio, and the Product 

Lifecycle and Workflow Descriptions. 

To ensure that Product Teams have 

sufficient training and development 

to allow for their success. 

5.12 Product Capabilities
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

29

5. Product and Model Oversight & Management - FBPML Organisation Best Practices v1.0.0. FBPML Organisation Best Practices v1.0.0.

5.12.6. Review of 

Product 

Capabilities

Product resource allocation ought to be 

periodically reviewed, or if significant 

changes occur, by Data Science Managers, 

in consultation with Product Owners, to 

ensure their continued effectiveness, 

suitability, and accuracy. 

To ensure that Products resources 

are sufficiently maintained. 

Objective

To promote the documentation and recording of Product, Product Team and employees tasks, deliverables and 

progress. 

Control: Aim:

5.13.1. Product Record A clear and detailed Product record should 

be continually kept of Product design, 

development, implementation, deliverables, 

progress and employee tasks. 

To ensure that a clear Product 

record is kept to warrant effective 

oversight, management and 

accountability. 

5.13.2. Employee and 

Product Team 

Documentation 

Employee and Product Team processes 

ought to promote the documentation of 

Product tasks, discussions and deliverables 

when relevant and as much as is reasonably 

practical. 

To ensure that sufficient 

documentation is kept to warrant 

effective oversight, management 

and accountability. 

5.13.3. Log of User 

Access 

A formal log of user access rights to 

Products should be maintained and 

reviewed at regular intervals by Product 

Owners. 

To ensure the management, 

integrity and review of user access. 

5.13.4. Event Logs Event logs of Product user activities, 

exceptions, and faults should be produced, 

kept and regularly reviewed by Product 

Owners. 

To formally index and manage 

user activities to maintain Product 

oversight and integrity.

5.13 Product Record
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

30

 FBPML Organisation Best Practices v1.0.0.

Section 6. Product Validation

To autonomously and impartially validate Products, Models and their outputs. 

Objective

Control: Aim:

6.1. Validation 

Department

The Organisation ought to have a 

department that is independent of Data 

Science Managers and Product Teams 

who validate Products.

To validate Product compliance, 

ethics, and performance.

6.2. Validation Policy A Policy detailing the responsibilities 

of the Validation Department and the 

requirements for Products to be Validated 

ought to be derived by the Managerial 

Committee. 

To ensure the validity of 

Organisation Machine Learning. 

6.3. Validation 

Procedures

The Validation Department should develop 

and implement a set of Procedures to 

operationalise the Validation Policy within 

the Organisation. 

To validate Product compliance, 

Ethics, and performance within the 

Organisation.

6.4. Review of the 

Validation Policy 

The Validation Policy should be reviewed 

periodically, or if significant changes 

occur, by the Managerial Committee 

to ensure its continued effectiveness, 

suitability, and accuracy. 

To ensure that the Validation Policy 

is kept up-to-date. 

6.5. Review of Product 

Validation 

Procedures

Product Validation Procedures should 

be reviewed periodically, or if significant 

changes occur, by the Validation 

Department to ensure their continued 

effectiveness, suitability, and accuracy. 

To ensure that Product Validation 

Procedures are kept up-to-date. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

31

 FBPML Organisation Best Practices v1.0.0.

Part C

 RESOURCE MANAGEMENT
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

32

 FBPML Organisation Best Practices v1.0.0.

Section 7. Human Resources 

Management

To ensure that Product employees and, when relevant, contractors are suitably identified, qualified and trained 

for their responsibilities and roles. 

Objective

Control: Aim:

7.1. Index of Product 

Employees

Product employees and, when relevant, 

contractors should be identified and an 

inventory composed and maintained. 

To ensure that Product employees 

can easily be identified to warrant 

their effective oversight and 

management. 

7.2. Human Resources 

Procedures

Procedures ought to be implemented 

to ensure that the human resources 

demands of Products are met in 

accordance with Organisation policies and 

the law. 

To ensure that Product human 

resources demands are adequately 

met. 

7.3. Human Resources 

Training

Product employees and, when relevant, 

contractors ought to receive regular 

training, including, inter alia, on industry 

best practices, state of the art, and 

ethical practices. 

To ensure that employees and, 

when relevant, contractors receive 

appropriate training. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

33

 FBPML Organisation Best Practices v1.0.0.

Section 8. Asset Management

To ensure the appropriate utilisation, maintenance, and, when relevant, termination of Product Assets. 

Objective

Control: Aim:

8.1. Inventory of Assets Product Assets should be identified and 

an inventory of these Assets composed 

and maintained. 

To ensure that Product Assets are 

known to warrant their effective 

oversight and management. 

8.2. Assets Roles and 

Responsibilities

Product Assets should have clear 

employee roles and responsibilities 

associated with their use, maintenance 

and security.

To ensure that a clear chain of 

employee roles and responsibilities 

are associated with Product 

Assets.

8.3. Acceptable Usage 

of Assets

Product Assets should have clear 

guidelines for their acceptable usage. 

To ensure that Product Assets are 

used in terms of best practices. 

8.4. Maintenance & 

Security of Assets

Product Assets should be maintained 

and secured in terms of industry best 

practices. 

To ensure that Product Assets 

are maintained in terms of best 

practices. 

8.5. Redundancy of 

Assets

Product Assets should be made redundant 

in terms of business needs and industry 

best practices. Redundant Assets must 

be appropriately decommissioned and all 

Product data permanently deleted. 

To ensure that Product Assets are 

decommissioned in a manner that 

eliminates organisation risk. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

34

 FBPML Organisation Best Practices v1.0.0.

Section 9. Asset Management

To ensure the integrity of Product Software and, when relevant, its termination. 

Objective

Control: Aim:

9.1. Inventory of 

Software

Product Software should be identified 

and an inventory of these programmes 

composed and maintained. 

To ensure that Product Software 

is known to warrant its effective 

oversight and management. 

9.2. Installation of 

Software

Procedures should be implemented to 

control and regulate the installation of 

Product Software. 

To ensure that Product Software is 

installed in terms of best practices. 

9.3. Updating of 

Software

Product Software should be regularly 

updated as and when applicable. 

To ensure that Product Software is 

kept up-to-date. 

9.4. Redundancy of 

Software

Product Software should be made 

redundant in terms of business needs 

and industry best practices. Redundant 

Software must be appropriately 

decommissioned and deleted. 

To ensure that Product Software is 

decommissioned in a manner that 

eliminates organisation risk. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

35

 FBPML Organisation Best Practices v1.0.0.

Part D

INCIDENT MANAGEMENT
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

36

 FBPML Organisation Best Practices v1.0.0.

Section 10. Incident Management

To warrant an effective and standardised approach to Incidents. 

Objective

Control: Aim:

10.1. Incident Policy A Policy, which promotes the 

standardisation, quick, effective and 

orderly response to Incidents, ought to 

be derived by Data Science Managers and 

approved by the Managerial Committee. 

To ensure the standardisation, 

quick, effective and orderly 

response to Incidents. 

10.2. Product Incident 

Procedures

Products should derive, develop and 

implement a set of Procedures to 

operationalise the Incident Policy within 

them. 

To ensure the standardisation, 

quick, effective and orderly 

response to Incidents within 

Products. 

10.3. Reporting 

Incidents

Incidents should be reported through 

Organisation channels to Product 

Owners, Data Science Managers, and the 

Management Committee as timeously, 

effectively and efficiently as reasonably as 

is practical. 

To ensure the Incidents are 

reported quickly, effectively and 

orderly.

10.4. Assessment of 

and Decisions on 

Incidents

Incidents should be appropriately 

assessed on reporting and decisions 

regarding responses thereto must 

be made as timeously, efficiently and 

effectively as reasonably as is practical. 

To ensure the Incidents response 

decisions are made quickly, 

effectively and orderly.

10.5. Response to 

Incidents

Responses to Incidents should take place 

in accordance with Product Incident 

Procedures. 

To ensure that Incidents are 

responded to in terms of 

organisation best practices and 

guidelines. 

10.6. Learning from 

Incidents

Information related to Incidents should 

be documented and knowledge gained 

therefrom must be analysed and studied 

to improve future Incident responses. 

To ensure that Incidents are 

properly learnt from. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

37

 FBPML Organisation Best Practices v1.0.0.

Part E

 ETHICS, PUBLIC INTEREST 

AND LEGAL 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

38

 FBPML Organisation Best Practices v1.0.0.

Section 11. Third Party Contracts 

Management

To ensure the integrity and implementation of Policies and Procedures in third-party contracts. 

Objective

Control: Aim:

11.1. Policies and 

Procedures 

for Third-Party 

Contracts 

Procedures ought to be designed and 

implemented to ensure that Policies and 

Procedures are legally enforceable in 

relevant third-party contracts. 

To ensure that Policies and 

Procedures are implemented by 

third-party contractors. 

11.2. Compliance 

in Third-Party 

Contracts 

Procedures should be implemented to 

ensure that Policies and Procedures are 

complied with to agreed-upon standards 

in relevant third-party contracts. 

To ensure that Policies and 

Procedures are complied with by 

third-party contractors.

11.3. Monitoring, Review 

and Auditing 

of Third-Party 

Contracts 

The Organisation should regularly monitor, 

review and audit third party contracts 

to warrant third-party compliance with 

Policies and Procedures. 

To ensure effective oversight of 

third-party contractors. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

39

 FBPML Organisation Best Practices v1.0.0.

Section 12. Ethics & 

Transparency Management

To ensure that Products are transparent and ethical. 

Objective

Control: Aim:

12.1. Ethics Policy An ethics policy, which promotes Ethical Practices 

in Machine Learning, ought to be derived by Data 

Science Managers and approved by the Management 

Committee and Ethics Committee. The Ethics Policy 

must be communicated to the Public. 

To ensure that Machine 

Learning is designed, 

developed and implemented 

in accordance with the 

Ethical Practices.

12.2. Review of the 

Ethics Policy 

The Ethics Policy should be reviewed periodically, 

or if significant changes occur, by Data Science 

Managers to ensure its continued effectiveness, 

suitability, and accuracy. 

To ensure that the Ethics 

Policy is kept up-to-date. 

12.3. Transparency 

Policy

A Public Interest and Transparency Policy, which 

promotes Public engagement, regulator engagement, 

and Transparency in Machine Learning, ought to be 

derived by Data Science Managers and approved by 

the Management Committee and Ethics Committee. 

The Transparency Policy must be communicated to 

the Public.

To ensure that Machine 

Learning is made 

transparent to the Public and 

is designed, developed and 

implemented in accordance 

with the Public Interest.

12.4. Review of the 

Transparency 

Policy 

The Transparency Policy should be reviewed 

periodically, or if significant changes occur, by 

Data Science Managers to ensure its continued 

effectiveness, suitability, and accuracy. 

To ensure that the 

Transparency Policy is kept 

up-to-date.

12.5. Speaking-Out 

Policy

A policy, which promotes Product Teams and/

or Product Teams members to speak-out against 

unethical practices, ought to be derived by Data 

Science Managers and approved by the Management 

Committee and Ethics Committee. 

To ensure that Product 

Teams and/or Product Teams 

members have a safe space 

to voice their concerns about 

Machine Learning and/or 

Products practices and/or 

decisions.

12.6. Review of the 

Speaking-Out 

Policy 

The Speaking-Out Policy should be reviewed 

periodically, or if significant changes occur, by 

Data Science Managers to ensure its continued 

effectiveness, suitability, and accuracy. 

To ensure that the SpeakingOut Policy is kept up-to-date. 

12.7. Contact with 

Authorities

Appropriate contact with relevant sector authorities 

regarding Products, and their implementation, should 

be maintained. Products and Product Teams ought 

to work in close collaboration with relevant sector 

authorities in a collaborative and bona fide manner. 

To ensure awareness and 

oversight of Products by 

authorities.
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

40

 FBPML Organisation Best Practices v1.0.0.

Section 13. Compliance, Auditing & 

Legal Management and Oversight

To ensure that Policies and Procedures are designed, developed and implemented in accordance with the law, 

industry best practices, contractual obligations, and/or regulatory Guidelines. 

Objective

Control: Aim:

13.1. Identification of 

Laws, Regulations and 

Contracts

Relevant laws, regulations and contracts 

concerning Machine Learning ought to be 

identified and documented. 

To ensure that relevant 

Machine Learning laws, 

regulations and contracts 

have been identified and 

documented. 

13.2. Response to Laws, 

Regulations and 

Contracts

Procedures should be implemented to 

incorporate relevant laws, regulations and 

contracts into Policies and Procedures. 

To ensure that relevant 

Machine Learning laws, 

regulations and contracts are 

implemented and abided by. 

13.3. Privacy and 

Protection of 

Personally Identifiable 

Information 

Procedures

Procedures should be implemented to 

ensure the privacy and protection of 

personally identifiable information in 

Products as required in law, regulations 

and/or contracts. 

To ensure that relevant data 

protection and privacy laws, 

regulations and contracts are 

implemented and abided by. 

13.4. Intellectual Property 

Rights Procedures

Procedures should be implemented 

to protect intellectual property rights 

and the use of proprietary products in 

Products as required in law, regulations 

and/or contracts. 

To ensure that relevant 

intellectual property rights 

laws, regulations and contracts 

are implemented and abided 

by. 

13.5. Internal Audit 

of Policies and 

Procedures

Defined organisational employees should 

audit the implementation of Policies and 

Procedures within the Organisation. 

To ensure the quality and 

integrity of implemented 

Policies and Procedures. 
Notice: This document and its content has been licensed under the Creative Commons Attribution license by the Foundation (Stichting) for Best Practices in Machine Learning (kvk number: 

82610363). Any subsequent and/or other use, copying and/or adaptation of this document or its content must abide by the appropriate licensing terms & conditions as reflected thereunder. 

41

 FBPML Organisation Best Practices v1.0.0.