The use of artificial intelligence and machinelearning by market intermediaries
and asset managers
Final Report

The Board
OF THE
INTERNATIONAL ORGANIZATION OF SECURITIES COMMISSIONS

FR06/2021

SEPTEMBER 2021

Copies of publications are available from:The International Organization of Securities Commissions website www.iosco.org© International Organization of Securities Commissions 2021. All rights reserved. Brief
excerpts may be reproduced or translated provided the source is stated.

ii

Contents
ChapterPage

1

Executive summary

1

2

Background and scope

4

3

How firms are using AI and ML techniques

64

Identified risks and harms posed by the use of AI and ML

95

Firms’ response to the potential risks arising from the use of AI and ML

14

6

Guidance

17

A1

How regulators are addressing the challenges created by AI and ML

22

A2

Guidance published by supranational bodies

34

A3

Feedback Statement

38

iii

Chapter 1 - Executive Summary
Background
Artificial Intelligence (AI) and Machine Learning (ML) are increasingly used in financial
services, due to a combination of increased data availability and computing power. The use of
AI and ML by market intermediaries and asset managers may be altering firms’ businessmodels. For example, firms may use AI and ML to support their advisory and support services,risk management, client identification and monitoring, selection of trading algorithms andportfolio management, which may also alter their risk profiles.
The use of this technology by market intermediaries and asset managers may create significant
efficiencies and benefits for firms and investors, including increasing execution speed andreducing the cost of investment services. However, this use may also create or amplify certain
risks, which could potentially have an impact on the efficiency of financial markets and could
result in consumer harm. The use of, and the controls surrounding, AI and ML within financial
markets is, therefore, a current focus for regulators across the globe.
IOSCO identified its work on the use of AI and ML by market intermediaries and asset
managers as a key priority. The IOSCO Board approved a mandate in April 2019 forCommittee 3 on Regulation of Market Intermediaries (C3) and Committee 5 on Investment
Management (C5) to examine best practices arising from the supervision of AI and ML. 1 The
committees were asked to propose guidance that member jurisdictions may consider adopting
to address the conduct risks associated with the development, testing and deployment of AI
and ML.
Potential risks identified in the Consultation Report
IOSCO surveyed and held roundtable discussions with market intermediaries and conducted
outreach to asset managers to identify how AI and ML are being used and the associated risks.
The following areas were highlighted in the Consultation Report released in June 2020 2 where
potential risks and harms may arise in relation to the development, testing and deployment of
AI and ML:
•
•
•
•
•
•

Governance and oversight;
Algorithm development, testing and ongoing monitoring;
Data quality and bias;
Transparency and explainability;
Outsourcing; andEthical concerns.

1

Board Priorities - IOSCO work program for 2019, March 25, 2019, available at:https://www.iosco.org/library/pubdocs/pdf/IOSCOPD625.pdf

2

The use of artificial intelligence and machine learning by market intermediaries and asset
managers, IOSCO Board Consultation Report, June 2020, available at:https://www.iosco.org/library/pubdocs/pdf/IOSCOPD658.pdf

1

IOSCO Guidance
Based on the responses received to the Consultation Report, this final report provides guidance
to assist IOSCO members in supervising market intermediaries and asset managers that utilise
AI and ML.
The guidance consists of six measures that reflect expected standards of conduct by market
intermediaries and asset managers using AI and ML. Although the guidance is not binding,
IOSCO members are encouraged to consider these measures carefully in the context of theirlegal and regulatory frameworks. IOSCO members and firms should also consider the
proportionality of any response when implementing these measures.
The use of AI and ML will likely increase as the technology advances, and it is plausible thatthe regulatory framework will need to evolve in tandem to address the associated emerging
risks. Therefore, this report, including the definitions and guidance, may be reviewed in the
future to remain up to date.

Measure 1: Regulators should consider requiring firms to have designated senior management
responsible for the oversight of the development, testing, deployment, monitoring and controlsof AI and ML. This includes a documented internal governance framework, with clear lines ofaccountability. Senior Management should designate an appropriately senior individual (orgroups of individuals), with the relevant skill set and knowledge to sign off on initial
deployment and substantial updates of the technology.
Measure 2: Regulators should require firms to adequately test and monitor the algorithms tovalidate the results of an AI and ML technique on a continuous basis. The testing should be
conducted in an environment that is segregated from the live environment prior to deployment
to ensure that AI and ML:(a) behave as expected in stressed and unstressed market conditions; and(b) operate in a way that complies with regulatory obligations.
Measure 3: Regulators should require firms to have the adequate skills, expertise andexperience to develop, test, deploy, monitor and oversee the controls over the AI and ML thatthe firm utilises. Compliance and risk management functions should be able to understand andchallenge the algorithms that are produced and conduct due diligence on any third-party
provider, including on the level of knowledge, expertise and experience present.
Measure 4: Regulators should require firms to understand their reliance and manage theirrelationship with third-party providers, including monitoring their performance and conducting
oversight. To ensure adequate accountability, firms should have a clear service level agreement
and contract in place clarifying the scope of the outsourced functions and the responsibility ofthe service provider. This agreement should contain clear performance indicators and should
also clearly determine rights and remedies for poor performance.

2

Measure 5: Regulators should consider what level of disclosure of the use of AI and ML isrequired by firms, including:(a) Regulators should consider requiring firms to disclose meaningful information to customers
and clients around their use of AI and ML that impact client outcomes.
(b) Regulators should consider what type of information they may require from firms using AI
and ML to ensure they can have appropriate oversight of those firms.
Measure 6: Regulators should consider requiring firms to have appropriate controls in place
to ensure that the data that the performance of the AI and ML is dependent on is of sufficientquality to prevent biases and sufficiently broad for a well-founded application of AI and ML.

3Chapter 2 - Background and ScopePrevious IOSCO work in this area
IOSCO has undertaken several workstreams on the use of AI and ML in financial markets,
including:•Committee on Emerging Risks (CER): The CER undertook a mandate on the use ofnovel technologies deployed by regulators to increase the efficiency and effectivenessof supervisory and oversight programs and published a report in February 2017. 3 CER
examined the regulatory use of tools such as big data analytics and data visualisation
technologies; AI and ML, and deep learning technologies; and distributed ledger
technologies.

•

Committee on Regulation of Secondary Markets (C2): C2 published a report in
April 2013 on Technological Challenges to Effective Market Surveillance Issues andRegulatory Tools. 4 The report made recommendations to help market authorities
address the technological difficulties facing effective market surveillance.

•

IOSCO Fintech Network: The IOSCO Fintech Network was established in May 2018
to facilitate the sharing of knowledge and experiences among IOSCO members. The
IOSCO Fintech Network considered the ethical implications of the use of AI and ML
technologies.

IOSCO Mandate
Building on its previous work, IOSCO released a Consultation Report on the use of AI and ML
by market intermediaries and asset managers in June 2020, proposing guidance to address thepotential risks and harms that may be caused by the use of AI and ML by these market
intermediaries and asset managers. The proposed guidance looked to help ensure that market
intermediaries and asset managers have:••••appropriate governance, controls and oversight frameworks over thedevelopment, testing, use and performance monitoring of AI and ML;
staff with adequate knowledge, skills and experience to implement, oversee, andchallenge the outcomes of the AI and ML;
robust, consistent and clearly defined development and testing processes toenable firms to identify potential issues prior to full deployment of AI and ML;
andappropriate transparency and disclosures to their investors, regulators and otherrelevant stakeholders.

3

IOSCO Research Report on Financial Technologies (Fintech), February 2017, available at:https://www.iosco.org/library/pubdocs/pdf/IOSCOPD554.pdf

4Technological Challenges to Effective Market Surveillance Issues and Regulatory Tools, August
2012, available at: https://www.iosco.org/library/pubdocs/pdf/IOSCOPD389.pdf

4

This Final Report confirms the guidance proposed in the Consultation Report, amending it totake account of responses received as appropriate.
Defining the terms Al and ML for this report
Artificial Intelligence
The term Artificial Intelligence, first coined by data scientist John McCarthy 5 in 1956, isdefined as “the science and engineering of making intelligent machines”, or simply, the studyof methods for making computers mimic human decisions to solve problems. AI includes tasks
such as learning, reasoning, planning, perception, language understanding and robotics. AI in
the financial services industry is still in its relative infancy and is poised to become morecommon, and with that will come legal, ethical, economic and regulatory challenges.
Machine Learning
The term Machine Learning is a subset and application of AI, which focuses on thedevelopment of computer programs - designed to learn from experience without being
explicitly programmed to do so.
There are three categories of ML algorithms – supervised learning, unsupervised learning andreinforcement learning. These categories are used based on the type of data available and thelevel of human intervention required in providing feedback. Deep Learning involves training
neural networks (computing systems) with many layers of units, inspired by the structure of
the human brain and can also include any of these categories: 6•

Supervised learning: the algorithm is fed an initial set of data that has been labelled.
Based on this training set, the algorithm will learn classification rules and predict thelabels for the remaining observations in the data set.

•

Reinforcement learning: the algorithm is fed an initial set of data that has not beenlabelled and is asked to identify clusters of observations underpinned by similar
characteristics. As it chooses an action for the data points, it receives feedback thathelps it learn. 7•
Unsupervised learning: the algorithm detects patterns in the data by identifying
clusters of observations underpinned by similar characteristics – it uncovers the structure of
the data on its own.
5

What is AI? available at: http://jmc.stanford.edu/artificial-intelligence/index.html

6

Deep learning is a method that analyses data in multiple layers, starting with learning about simple
concepts then learning more complex concepts. Deep learning can be used for all three categories
of machine learning algorithms.
Adapted from ‘What is Machine Learning?’ at https://www.ibm.com/cloud/learn/machinelearning. A famous example of this is the “move 37” in the game of Go: when Google’s AlphaGo5
algorithm was pitted against professional Go player Lee Sedol in March 2016, making a move on
the 37th turn that was previously unimaginable. This algorithm used deep learning, a form of ML
technique that efficiently learns associations and statistical patterns from a very large dataset.

7

R Sutton, A Barto, Reinforcement Learning: an introduction, MIT Press, 1998.

5

Chapter 3 – How firms are using AI and ML techniques
AI and ML use by market intermediaries and asset managers
Market intermediaries and asset managers’ use of AI and ML is growing, as their understandingof the technology and its utility evolves. The rise in the use of electronic trading platforms andthe increase of available data have led firms to consider the use of AI and ML in several areas,
including for their trading and advisory activities, as well as for risk management andcompliance.
The IOSCO firm engagement revealed that within financial markets, AI and ML are being
adopted to augment existing processes and activities, with a view to reducing cost and increase
efficiency. AI and ML are freeing up resources to focus on more cognitive aspects, such as
strategy, portfolio selection and generating investment ideas. Market intermediaries are
deploying this technology in:
•
•
•
•
•

Advisory and support services;
Risk management;
Client identification and monitoring;
Selection of trading algorithm; andAsset management/ Portfolio management.

The use of AI and ML by asset managers appears to be in its nascent stages and is mainly used
to support human decision-making. AI and ML techniques are being used to:•••

Optimise portfolio management;
Complement human investment decision-making processes by suggesting investment
recommendations; andImprove internal research capabilities, as well as back office functions.

Some asset managers are also beginning to use AI and ML for order execution, broker selection
and order routing purposes (including through methods such as algo-wheels). 8
Advisory and support services
According to IOSCO’s industry engagement, most robo-advisors or automated investment
advisors use simple, rule-based (i.e., deductive) algorithms, although some are beginning toutilise predictive ML algorithms. Where ML is used to provide advisory services, most firms
have manual intervention processes. The automated advice system is, therefore, usually limited
to generating potential advice or asset allocation for the investment adviser to review. The
investment adviser can then use this AI-generated advice as appropriate and, where suitable, tomake a recommendation to the client.
8

Algo-wheels may perform different functions in different parts of the world. In this context, we
define algo-wheels to mean a software/model that aggregates data to select the strategy and broker
through which to route orders before generating a report that sets out the reason behind how andwhere a particular trade was made.

6

Risk management
Risk management involves using data to price and manage exposure, including credit, market,
operational and liquidity risk. Market intermediaries are harnessing ML-based risk
management systems for credit risk monitoring, which could help provide an early-warning
indicator of potential customer defaults and can help create a dynamic measurement of acustomer’s risk profile to better understand, for example, when to write off a debt.
ML is improving the efficiency of back office processing and reporting functions within market
intermediaries. It is also increasingly used to visualise market risk by analysing volatility
trends, and to gauge liquidity risk by analysing multi-dimensional risk and exposure data. ML
algorithms are increasingly used to monitor staff e-mails by leveraging advanced patternrecognition.
Some market makers, who provide liquidity to market participants by facilitating transactions
are adopting ML models and reinforcement learning to minimise their inventory risk andmaximise the utility of their balance sheet.
Similarly, some asset managers are seeking to harness the advantages of these techniques inrisk management. Some hedge funds and asset managers are automating risk management andcompliance processes by tracking the behaviour of individual portfolio managers, automating
execution quality reports and assessing market liquidity risk.
Client identification and monitoring
ML has allowed market intermediaries to automate their client onboarding, fraud detection,
money laundering and cyber-attack monitoring. Market intermediaries should generally
undertake Know Your Customer (KYC) checks before onboarding clients and selling themproducts and services. KYC entails collection and verification of comprehensive personal
information from potential clients, involving processing unstructured metadata.
Inductive reasoning algorithms help accurately identify fake photo IDs, while recognisingdifferent photos of the same person. ML can also be used for screening and monitoring clientsand transactions against sanctions or other lists, to detect evidence of possible money
laundering, terrorist financing and other financial crimes.
Selection of trading algorithms
Many market intermediaries currently offer a software solution to their clients that selects anappropriate trading strategy and/or a broker depending on the market situation and trading
objectives for best execution purposes, often named an algo wheel. Algo-wheels seek toclassify historical trading and performance, predict the performance of strategies and broker
algorithms, and recommend when to use a particular algorithm. Using algo-wheels to optimally
execute simpler orders allows the trader to focus on more complex trade flows.
Predictive data analytics are enabling the identification of potential market conditions
conducive to a flash crash type event.

7

Asset management/Portfolio management
Supervised learning, where a function is inferred from labelled training data, has been used forsmall-scale pattern recognition and simple prediction models to aid trading decisions within
asset managers and market intermediaries for several years.
Margin pressure and competition is driving innovation amongst asset managers. To compete,some active managers that traditionally emphasised their fundamental research capabilities are
beginning to expand already existing quantitative approaches by leveraging diversified data
sources – such as social media, geospatial data, and other metadata to enhance internal research.
In certain cases, these methods are being used for asset-allocation and pricing such as
identifying relationships in metadata which could be used to generate trade ideas or alphasignals and forecast asset prices based on historical prices as well as current trends. Moreover,asset managers may apply these techniques to price new investment products competitively.
They do so by using data on existing investment products with similar structures and/or
constituent assets. Other applications include investment compliance checks, transfer agency
activities and client servicing.

8Chapter 4 – Identified Potential Risks and Harms Posed by the Use of AI
and MLIOSCO’s industry engagement revealed that the evolution and increasing adoption of AI and
ML may raise (either intentionally or unintentionally) a number of conduct concerns formarket intermediaries and asset managers, regarding:
•
•
•
•
•
•

Governance and oversight;
Algorithm development, testing and ongoing monitoring;
Data quality and bias;
Transparency and explainability;
Outsourcing; andEthical concerns.

Governance and oversightFirms implementing AI and ML mostly rely on existing governance and oversightarrangements to sign off and oversee the development and use of the technology. In most
instances, the existing review and senior leadership-level approval processes were followed todetermine how risks were managed, and how compliance with existing regulatory requirements
was met. AI and ML algorithms were generally not regarded as fundamentally different from
more traditional algorithms and few firms identified a need to introduce new or modify existingprocedural controls to manage specific AI and ML risks.
Some firms indicated that the decision to involve senior leadership in governance and oversightremains a departmental or business line consideration, often in association with the risk and IT
or data science groups. There were also varying views on whether technical expertise isnecessary from senior management participating in and overseeing control functions such as
risk management. Despite this, most firms expressed the view that the ultimate responsibility
and accountability for the use of AI and ML would lie with the senior leadership of the firm.
Some firms noted that the level of involvement of risk and compliance tends to focus primarily
on development and testing of AI and ML rather than through the lifecycle of the model (i.e.,
implementation and ongoing monitoring). Generally, once implemented, some firms rely on
the business line to effectively oversee and monitor the use of the AI and ML. Respondents
also noted that risk, compliance and audit functions should be involved throughout all stagesof the development of AI and ML.
Many firms did not employ specific compliance personnel with the appropriate programmingbackground to appropriately challenge and oversee the development of ML algorithms. With
much of the technology still at an experimental stage, the techniques and toolkits at the disposalof compliance and oversight (risk and internal audit) functions currently seem limited. In somecases, this is compounded by poor record keeping, resulting in limited compliance visibility as
to which specific business functions are reliant on AI and ML at any given point in time.

9

Algorithm development, testing and ongoing monitoring
It is important that firms have robust and well understood development and testing frameworks
in place, regardless of whether they are using AI and ML or traditional algorithms.
Overall, IOSCO’s engagement showed that in most cases there is not an established frameworkfor specifically developing AI and ML. Instead, many firms use the same development andtesting frameworks that they use for traditional algorithms and standard system development
management processes.
Firms that use algorithms should consider maintaining an appropriate development and testing
framework, which is consistently applied across all relevant aspects of the business. This isparticularly important where firms are using AI and ML within their algorithmic trading
strategies.
Algorithms rely on quality data and most firms recognise the need for quality data inputs.
Excessive immaterial, or “noisy” data is unwanted data that does not contribute to a relationshipand may cause ML algorithms to miss the signal in the data and behave unexpectedly.
Robust development and testing controls are necessary to distinguish signals and statisticallysignificant data from the noise. Unlike traditional algorithms, as more data is processed by ML
algorithms, they may behave unpredictably as they are exposed to new data patterns. ML
algorithms should therefore also be continuously monitored throughout their deployment tohelp ensure they do not behave inexplicably owing to a subtle shift in the operating conditions
or excessive noise. While some firms noted they review and revise existing models asnecessary, some firms focus less on managing models in the post-production phase so that they
perform as they should over time. Unlike traditional algorithms, ML algorithms continuallylearn and develop over time. It is important that they are monitored to ensure that they continue
to perform as originally intended.
Data quality and bias
The performance of AI and ML is inherently dependent on the quality of the dataset particularly
when building the model. According to a few of the respondents to the Consultation Report,
the quality of the datasets used in the learning phase can have a material impact on the potential
outcomes and performance of AI and ML applications. In the view of the respondents, as suchit is a key risk. According to some respondents, another risk related to datasets is the sourcingof sufficiently large datasets to train the applications, particularly when related to investment
decisions.
Learned bias in the dataset can impact the decisions made by such algorithms and may resultin discriminative decisions and provide undesirable outcomes to market participants. For
example, asking questions phrased in a certain way or in a certain sequence may lead to a
response that introduces implicit or explicit bias from the respondents. Such a dataset, where abias may have been introduced by either the questioner or by the respondents, will influence
the conclusions reached by the algorithm. Any output based on such a bias will likely degrade10the performance of the algorithm more quickly over time and could result in consumer
detriment. 9
Bias may also be inadvertently introduced during data cleansing – a pre-processing step often
necessary to improve data quality. Cleaning data before applying ML can increase the signalto-noise ratio allowing for more meaningful interpretations to be derived. However, cleansing
data involves subjective decisions, which may inadvertently introduce other biases.
Transparency and explainability
The effective use and adoption of AI and ML require algorithms that are not only accurate but
are also understandable by firms (including front line, compliance and risk personnel), market
counterparties, clients and regulators. It is possible that risks are introduced if outcomes cannot
be fully explainable. While increased transparency in firms’ use of AI and ML could improve
public understanding and confidence in the use of the technology, excessive transparency could
create confusion or opportunities for individuals to exploit or manipulate the models. The levelof transparency will also differ depending on the audience; for example, a regulator may require
more detailed information than a client. These considerations need to be balanced indetermining the appropriate level of transparency in the use of AI and ML.
It is important that firms appropriately disclose information about their service offerings, tohelp clients understand the nature and risks of products and service offerings, so that they canmake informed decisions. Applying unexplainable ML algorithms to refine a trading strategy
could expose the firm to unacceptable levels of legal and regulatory risk. Firms offering
automated investment services, including robo advice and automated portfolio construction
should appropriately disclose the nature of the use of ML and the automation in their service
offering.
Some ML models operate as a “black box” with limited clarity on the reasoning behind theoutput. For example, in the case of deep unsupervised learning algorithms, the decisions madeby the AI and ML models can be non-interpretable or unexplainable. 10 The interpretability or
explainability of ML models on an ex-ante basis may therefore be challenging.
Firms had differing views on what is reasonably expected to be disclosed to investors. Somefirms suggested that more transparency should be provided with respect to the use of AI and
ML in investment decisions as opposed to administrative and back office functions. While mostfirms supported a level of generic transparency, others were against disclosures on the specificimplemented approaches. Some firms also suggested there may be differences in disclosuresprovided to institutional versus retail clients.

9

The French regulator Authorite des Marches Financiers (AMF) acknowledges that algorithms canreinforce biases and lead to financial exclusion of certain groups. For example, models based on
behaviour history perform less well for younger customers with a limited history, in which caseother explanatory variables need to be found. available at: https://acpr.banquefrance.fr/sites/default/files/medias/documents/2018_12_20_intelligence_artificielle_en.pdf

10

Basic ML models are explainable, where the more advanced ones (that use deep learning and haveneural networks) may encounter the “black box” problem.

11

Outsourcing
IOSCO’s engagement observed that firms used external providers of AI and ML to varying
extents. Some larger firms indicated they develop and implement AI and ML in-house or inpartial collaboration with external providers, however smaller firms tend to revert to solutions
offered by external providers. There is a risk that the concentration of expertise in the field of
AI and ML creates outsourcing risk if there is reliance on a small number of firms within thisspace. The IOSCO Principles on Outsourcing provide a useful guidance to address the potential
risks that might be associated with outsourcing.
Many firms also use externally offered data pools, and some firms noted they rely on external
providers for cloud computing. Use of external providers for AI and ML solutions, including
data pools and cloud computing may raise concerns about data privacy, cybersecurity andoperational risk in addition to concerns about concentrations of certain providers (e.g., data
providers, cloud providers, technology providers), particularly where firms may not have theappropriate level of expertise to conduct effective due diligence assessments ahead of utilising
their services. It should be noted that these risks are not unique to the use of AI and ML andcan be mitigated through proper and efficient outsourcing processes.
Ethical concerns
Since the financial crisis, significant attention has been paid to the role of ethics and trust in
financial services by market participants. The Chartered Financial Analyst (CFA) Institute has,for example, defined ethics as “a set of moral principles or rules of conduct that provide
guidance for our behaviour when it affects others” and has suggested that fundamental ethicalprinciples include honesty, fairness, diligence, care and respect for others. 11 Ethical conduct
follows those principles and balances self-interest with both the direct and indirectconsequences of that behaviour for other people.
In the context of AI and ML, ethical concerns may arise when models develop certain social
biases and potentially recommend undesirable outcomes, for example where data cleaning,data transformation and anonymisation of data were not adequately considered. There arequestions about how these types of ethical considerations, including those around diversity andinclusion, can continue to be met by firms and their employees as algorithmic models play anincreasingly important role in the functioning of markets.
The IOSCO Fintech Network sought to leverage existing academic literature and industry
engagement to understand how the use of AI and ML may affect ethical behaviour in securities
markets. In its engagement with industry participants and member organisations, the Fintech
Network has identified robo-advice as one AI and ML application with significant potential
ethical implications.
While most robo-advisors use simple rule-based algorithms, some are moving towards the useof predictive machine learning algorithms. Market participants should be careful whendeveloping such AI and ML using large pools of alternative (i.e., non-traditional) datasets, suchas satellite data or twitter feeds, to seek to ensure that the developed models would not11

https://www.cfainstitute.org/en/ethics/codes/std-of-practice-guidance/ethics-and-investementindustry

12

discriminate against a certain segment of the population and that the AI and ML driven
decisions are fair and unbiased.
In this context, IOSCO’s Fintech Network identified five primary themes that could underpin
the ethical use of AI and ML techniques:
•
•
•
•

•

Beneficence – “do good”: ensuring the model is being used and/or acting in goodfaith, in the best interest of investors and with market integrity.
Non-malfeasance – “do no harm”: having the ability to understand and interpret
AI and ML based decisions to identify where misconduct may be taking place.
Human autonomy, including auditability: ensuring humans have power over what
the model can and cannot decide.
Justice: accountability and transparency: ensuring there is accountability for theactions of the model at a senior level; and that accountability comes withappropriate understanding of the models. This theme is also about firms
understanding the level of transparency they need to demonstrate both internallyand with clients.
Explainability: ensuring the outcomes arising out of the models used can be
explained.

Many of these themes are explored, directly and indirectly, in Chapter 5 – Firm responses tothe potential risks arising from the use of AI and ML and in IOSCO’s guidance.

13

Chapter 5 –Firms’ responses to the potential risks arising from the use of
AI and ML
Obligations under existing regulations
The AI and ML techniques described above are currently being deployed by some firms andused in the context of existing regulatory frameworks. This includes high level generic rules inrelation to systems and controls, senior management accountability, risk management,
outsourcing and governance. However, the rise in the use of these techniques has led toincreased interest from regulators.
Many jurisdictions have overarching requirements for firms’ overall systems and controls;
however, few jurisdictions have regulatory requirements that specifically apply to AI and ML
based algorithms. These overarching requirements include rigorous testing of the algorithm ina safe environment away from the market before its controlled deployment, as well as
continuous performance monitoring throughout the lifecycle of the algorithm. In this context,
firms should consider whether solely relying on existing processes remains appropriate when
meeting regulatory requirements or whether those processes should be further revised.
Moreover, not all ML techniques will be compatible with existing legal or regulatory
requirements. Regulatory regimes that require algorithms to be fully understood andexplainable throughout their lifecycle invariably limits the use of algorithms that evolve
through the course of deployment in response to environmental changes.
Firms not using AI and ML
Some firms stated that they are opting not to use or are limiting their use of AI and ML
algorithms until they are able to comply with jurisdictions’ laws and regulations applicable tothe use of the technology. Numerous survey respondents commented that they did not have thehuman resources or the right expertise at all levels to always fully understand AI and ML
algorithms. Some firms therefore cannot implement the technology in a compliant manner.
Operational and infrastructural concerns such as compatibility between AI and ML and theirlegacy technology, and operational processes are also precluding firms from meaningfully
adopting AI and ML. Costs may be an additional consideration for small and medium-sized
firms.
Potential mitigations to the risks posed by the use of AI and ML
Through IOSCO’s engagement with firms, including through questions asked in theConsultation Report, several considerations were put forward as a way to mitigate the potential
risks posed by the use of AI and ML, including ethical ones as identified by IOSCO’s Fintech
Network. These firms generally considered the following elements as pertinent to risk
mitigation:•
Culture;•Accountability;•
Knowledge/expertise/skills;•
Operational resilience; and•Client disclosures.
14Culture
As the culture of each firm is different, a one size fits all approach would unlikely be effective.
However, when considering a framework conducive to good market conduct for the use of AI
and ML, the key elements of codes of ethics (e.g., honesty, fairness, diligence, care and respect
for others) can provide the foundations to build upon. Senior leaders of a firm can manage thedrivers and behaviours within the firm to create and maintain a culture, or tone from the top,which can help reduce any potential harm to investors caused by inappropriate behaviour.
Firms also reported that the ethical implications from the use of AI and ML techniques could
be viewed within this existing framework. Senior leaders can promote the organisation’s
culture and can help minimise the risk from their use of ML. Firms that are considering ethicalframeworks for the use of AI and ML systems in the broader context of the organisation’s
culture could reduce the harm to investors by designing AI and ML systems and a control
environment that reflect ethical values, organisational values and cultural behaviour.
This can be demonstrated by developing a robust governance framework and the availabilityof the right skills, knowledge, expertise and experience that is supported by a culture that
permits adequate challenge and accountability across the firm. Responsibilities do not solely
lie with senior leadership and second or third line functions, but also with the front office
functions which act as the first line of defence with regards to the development and operationsof AI and ML. Firms should also have consistent standards for developing and testing AI and
ML algorithms. In this context, considerations about the appropriate level of disclosure toclients, regulators and all relevant stakeholders of where AI and ML techniques are used canalso assist the firm in promoting a transparent culture.
Accountability
Many jurisdictions indicated that they have recently considered whether and how individual
accountability can promote fairness and trust within financial services. Accountability aims to
reduce harm to investors and strengthen market integrity by making individuals personallyaccountable for their conduct and competence. For example, under the UK’s Senior Managers& Certification Regime (SM&CR), senior managers are ultimately accountable for theactivities of the firm, and there is a certification requirement for staff responsible foralgorithmic trading.
It is important that senior managers have adequate oversight and control over the development,
testing, implementation and monitoring process for AI and ML techniques, as well as over any
third-party providers to which their firm may be outsourcing.
Developments in AI and ML technologies may lead regulators and firms to further consider
the responsibilities of other individuals, such as data analysts, data scientists and data engineers
who may not typically have performed roles subject to regulatory scrutiny.

15

Knowledge/Skills/Expertise
Ensuring the right skills, knowledge and expertise to discharge the responsibilities of an
employee’s role is essential. This includes standards of ethical behaviour.
Firms may want to consider whether to reassess their approach to evaluating individuals’
skillsets and also ethical attributes, such as honesty, integrity and reputation, in order to meetthe challenges posed by AI and ML. One firm specifically mentioned that it has educated all
employees and senior management on the development and application of AI and ML
techniques and how the use of this technology is consistent with the firm’s culture. This hasreportedly helped bridge the gap between technical and non-technical staff across the firm.
Where firms outsource certain components of their technology, there may also be concerns
about the level of understanding, at firm level, of the activities that take place at third-party
provider level which can in turn impact the quality of due diligence reviews.
Operational resilience
Operational disruption can impact market integrity, threaten the viability of individual firms,
or cause harm to consumers and other market participants. Should the use of AI and ML
become widespread, operational resilience could also impact financial stability as new andunexpected forms of interconnectedness between financial markets and institutions emerge. As
AI and ML are reliant on large data sets and computational resources, more firms are leveraging
cloud computing offered by third-party service providers, who offer potentially cheaper, moresecure and “ever-green” technology solutions. Many firms flagged operational resilience as
one of their key concerns when using AI and ML techniques and mentioned their due diligenceand oversight of third-party service providers as a key mitigating control.
Client Disclosure

According to some respondents to the Consultation Report, client disclosure may be a way tomitigate some of the potential risks posed by the use of AI and ML, and they stated that it isimportant to distinguish between different forms of AI, since disclosure is not equally relevant
to every type of application. Some stated that client disclosures are important, and a few
indicated that a degree of proportionality is used by these firms based upon the investment
strategy being undertaken, and the sophistication and type of client. They indicated that thisdisclosure can be used to demonstrate that firms are acting in the client’s best interest when
deploying or developing these applications. They also indicated this proportionality approachthey use to disclosure includes both the amount and quality of information that is disclosed andthe type of information that is disclosed.

16Chapter 6 – GuidanceThis chapter contains guidance to IOSCO members. Each measure is designed to address one
or more of the key potential risks and harms identified in Chapter 4. They are informed by goodpractices currently being carried out by some firms or expected by several regulators across theglobe.
Although the guidance is non-binding, IOSCO encourages its members to consider theguidance within the context of their legal and regulatory framework when regulating firms
utilising AI and ML. The guidance reflects an expectation of high standards of conduct acrossmarket intermediaries and asset management firms, and includes:
•
•
•
•
•
•

Governance and responsibilities;
The development, testing, and ongoing monitoring of AI and ML techniques;
Knowledge and skills required by firms’ staff;
Operational resilience;
Systems and controls; andTransparency and disclosure.

Proportionality
Proportionality should underpin the consideration and implementation of the guidance.
Regulators and firms should consider the risks posed by AI and ML while not undermining key
safeguards that the guidance seeks to put in place. In judging proportionality, consideration
should be given to the activity that is being undertaken, the complexity of the activity, risk
profiles, the degree of autonomy of the AI and ML applications, and the potential impact thatthe technology has on client outcomes and market integrity. Regulators should use their own
judgement when applying this overarching principle.
When considering proportionality, the size of the firm is only one factor. In practice, theactivities of a relatively small firm could have a material impact on market participants and theorderly functioning of markets. Regulators should consider what impact the technology could
have on client outcomes, as well as markets. Client facing tools that rely on AI and ML, forexample, might have a greater impact on client outcomes as opposed to technology that is
supporting back or middle office functions.
Given the significant potential risks and harms the measures intend to address, IOSCO
members are encouraged to consider the extent to which these measures should be implemented
in the context of regulated entities’ business models, and their own legal and regulatory
framework.
Measure 1: Regulators should consider requiring firms to have designated senior
management responsible for the oversight of the development, testing, deployment,
monitoring and controls of AI and ML. This includes a documented internal governanceframework, with clear lines of accountability. Firms should designate an appropriately
senior individual (or groups of individuals), with the relevant and current skill set andknowledge to sign off on initial deployment and substantial updates of the technology,
which might also be combined with an already existing person overseeing overall
technology and/ or data.
17

This measure looks to embed accountability in all aspects of a firm’s use of AI and ML andhelps ensure the technology (and its underlying data) is appropriately understood, tested,
deployed and monitored. The most senior people (senior management) in firms who performkey roles (such as “senior management functions” in the UK) are typically accountable for theoverall performance of the firm. This accountability extends to the actions and outcomes of AI
and ML models, including externally sourced models. This responsibility may require clear
lines of accountability, including procedures to approve the development, deployment andsubsequent updates of trading algorithms and to solve problems identified when monitoring
trading algorithms. Clear lines of accountability include firms having a documented internal
governance framework (which include the legal, compliance and risk management functions).
Regulators should consider requiring firms to:••••Understand how AI and ML are being utilised, including the intended outcomes;
Implement appropriate controls and governance frameworks to oversee and challenge
the outcomes derived from the AI and ML models, including underlying data and/or
externally sourced models;
Formulate a clear methodology document their methodology and have an audit trail ofthe use of AI and ML across the life cycle of the use of AI and across their business;
andAssess whether the technology is applied consistently with the firm’s risk appetite andclient’s risk tolerance, and in an ethical manner.

Senior management may already have the appropriate technical knowledge required toeffectively oversee the firm’s use of AI and ML techniques. However, where they do not havesuch knowledge, they could designate appropriate senior personnel within the firm (who may
be – but do not necessarily need to be – part of the firm’s senior management, e.g., executive
committee) to support them in discharging this oversight role while retaining ultimate
accountability. This role could be combined with an already existing individual (or groups ofindividuals). Taking into account the principle of proportionality, the firm could also use
specific skills existing within the global or regional firm governance structures.
Measure 2: Regulators should require firms to adequately test and monitor thealgorithms to validate the results of an AI and ML technique on a continuous basis. The
testing should be conducted in an environment that is segregated from the live
environment prior to deployment to ensure that AI and ML:(a) behave as expected in stressed and unstressed market conditions; and(b) operate in a way that complies with regulatory obligations.
Any testing should reflect the underlying complexity and systematic risks posed by the use of
AI and ML. Any material changes to the firms’ systems should trigger further testing.
The testing should be conducted in an environment that is segregated from the live environment
prior to deployment to ensure that AI and ML:a) Behave as expected in stressed and unstressed market conditions; and18b) Operate in a way that complies with regulatory obligations.
Once deployed, the performance and output of the AI and ML algorithms should be monitored
on a continuous basis. A few respondents to the Consultation Report expressed the view that it
is expected that a continuous basis would not require real-time testing and monitoring, andthese firms would like to take a proportionate approach given the tasks that they are performing,balancing the protections that ongoing testing provide against the costs of such testing. Firms
should also ensure, where appropriate, adequate kill switch functionality is built into theircontrol framework, and that appropriate functions and people have access to it. Kill switch
functionality should also be appropriately tested.
The use of AI and ML should be properly assessed and tested in light of their risks, including
market abuse, data privacy, risk management, and cybersecurity to help ensure they work as
intended. Risk and compliance functions should be involved in the development and testing ofAI and ML, as well as continuously monitoring the outputs post-deployment.
The behaviour of AI and ML may change in an unforeseen manner as more data is processedover time. Firms should therefore think beyond the existing testing and oversight arrangementsthat may be used for traditional algorithms and ensure the AI and ML techniques can bemonitored continuously as the algorithms adjust and transform.
It would not be enough for the AI and ML algorithm to be tested thoroughly before deployment;
they need to be continuously monitored throughout their deployment to ensure that analgorithm does not behave in inexplicable ways owing to a subtle shift in the operating
conditions or due to excessive noise. The “kill switch” function should also be complemented
with appropriate back-up solutions and tested prior to live deployment to ensure that thisfunction can indeed be depended on and initiated should it be required in future.
Measure 3: Regulators should require firms to have the adequate skills, expertise andexperience to develop, test, deploy, monitor and oversee the controls over the AI and ML
that the firm utilises. Compliance and risk management functions should be able tounderstand and challenge the algorithms that are produced and conduct due diligence on
any third-party provider, including on the level of knowledge, expertise and experience
present.
The lack of adequate internal skills, knowledge, expertise and experience to maintain andoversee AI and ML may lead to difficulties in updating the model or over-reliance on external
parties. Therefore, regulators should consider requiring firms to have the appropriate internal
skills, knowledge, experience and expertise including front line, compliance, risk management
and senior management staff, to understand and continuously supervise AI and ML within
existing legal and regulatory parameters.
Firms should consider establishing multi-disciplinary teams involving the business line usersof these solutions, data scientists, IT and database administration staff, as well as risk
management and compliance teams.
Finally, firms should consider the importance of model continuity and mitigate the risk that the
departure of key quantitative researchers or data scientists disrupt the proper functioning of the19AI and ML solutions. Firms should therefore have processes and documentation requirementsthat provide for model continuity in the event of staff departures.
Measure 4: Regulators should require firms to understand their reliance on and manage
their relationship with third-party providers of AI and ML, including monitoring
providers’ performance and conducting oversight. To ensure adequate accountability,
firms should have a clear service level agreement and contract in place clarifying thescope of the outsourced functions and the responsibility of the service provider. Where
appropriate, this agreement should contain clear performance indicators and should also
clearly determine rights and remedies for poor performance.
Operational resilience and heavy reliance on third-party service providers is one of firms’ key
concerns when using AI and ML techniques. To ensure adequate accountability, firms should
perform initial and ongoing due diligence and have a clear service level agreement and contract
in place clarifying the scope of the outsourced functions and the responsibility of the service
provider. This agreement should contain clear key performance indicators and should also
clearly determine suitable recourse for poor performance.
In addition, in recognition of the new challenges and risks involved in operating internationallyor outsourcing significant level of activities to third parties, IOSCO has updated its existing
principles on outsourcing to consider the risks associated with the use of third-party providers.
IOSCOs outsourcing principles should be considered when outsourcing the development,
testing and oversight of AI and ML.
Measure 5: Regulators should consider what level of disclosure of the use of AI and ML,
is required by firms, including:(a) Regulators should consider requiring firms to disclose meaningful information tocustomers and clients around their use of AI and ML that impact client outcomes.
(b) Regulators should consider what disclosures they may require from firms using AI
and ML to ensure they can have appropriate oversight of those firms.
Firms should disclose meaningful information to customers and clients around their use of AI
and ML algorithms that impact their outcomes. A majority of respondents to the Consultation
Report agreed with this measure, and some expressed concerns that the measure should be
proportionate to the risks and the services being provided. The objective is to disclose sufficient
information to enable clients to understand the nature of, and key characteristics of the products
and services that they are receiving, and how they are impacted by the use of the technology.
Firms should consider the level of detail necessary to satisfy the disclosure objective. Thistransparency should aim to provide, on a non-discriminatory basis, all relevant parties with thenecessary information to evaluate, to the extent possible, the benefits and risks associated withthe technology.
The language used in the disclosures should be comprehensible to investors. This will help
build trust and enable clients to understand the service and products that they are being offeredand sold and allow them to make informed decisions.

20

Measure 6: Regulators should consider requiring firms to have appropriate controls inplace to ensure that the data that the performance of the ML and AI is dependent on isof sufficient quality to minimise biases and sufficiently broad for a well-founded
application of AI and ML.
The performance of AI and ML is largely dependent on data quality and lack of bias inprocessing. The existence of bias in the results of AI and ML algorithms may jeopardise boththose firms that use them and their customers, due to the risks of discrimination or inadequate
advice.
Firms should therefore ensure an adequate level of data quality, by checking the quality of thesources used, as well as the relevance and completeness of the data with regard to theunderlying objectives of the algorithm. In particular, firms should ensure that the data set isrepresentative of the target population so that they do not lead to exclusion phenomena.
Biases may exist both in the data collected and in the manner in which they are processed.
They may be present directly in the variables used, for example, with variables considereddiscriminatory such as gender. They can be implicit due to the interaction of several variables
that are not in themselves discriminatory. Firms should ensure that the data as well as theoutputs of the AI and ML are analysed for the risk of discrimination.
Biases can be reinforced by AI and ML algorithms, resulting in consumer harm. For example,
models based on behaviour history are performing sub-optimally for younger customers witha limited history, in which case other explanatory variables need to be found.
Firms should have appropriate processes and controls, to identify and remove unjustifiablebiases from data sets. A few respondents to the Consultation report expressed their concernthat the approach to the measure be proportionate to the risks). Firms should consider having
specific training courses to raise awareness amongst their data scientists (and/or other relevant
staff) of potential data biases.

21

ANNEX 1
How regulators are addressing the challenges created by AI and ML
FSRA (Abu Dhabi Global Market)
The Financial Services Regulatory Authority (FSRA) issued detailed guidance in July 2019setting out the regulatory framework for firms carrying out “digital investment management”(i.e., robo-advisory activities) in Abu Dhabi Global Market (ADGM). 12
Such firms (termed “Digital Investment Managers”) are required to apply for a licence fromthe FSRA to conduct one or more regulated activities which may include (depending on theirbusiness models):•••

Advising on Investments or Credit
Arranging Deals in Investments
Managing Assets

Digital Investment Managers – being regulated entities – would therefore need to comply with
the FSRA’s regulatory requirements such as those pertaining to the conduct of business andprudential requirements. Such firms are also required to have robust frameworks and controls
in place with respect to client suitability and disclosures, algorithm governance, and technology
governance.
The limited human interaction between Digital Investment Managers and their clients
necessitates careful consideration of how suitability assessments are performed. Digital
Investment Managers typically rely heavily on an online questionnaire to collect the
information needed to perform suitability assessments (Risk Profile Questionnaire. Whendesigning a Risk Profile Questionnaire, the FSRA expects that Digital Investment Managers
ensure that the information obtained to assess suitability is proportionate with the complexity
and risk of the products being sold. In addition, the Risk Profile Questionnaire should havemechanisms for “knock out” questions that reject prospective clients whose investment
horizon, liquidity needs or other circumstances are misaligned with the products offeredthrough the platform.
A critical component of the digital investment management business model is the use ofalgorithms to automate the investment process. Accordingly, the FSRA sees a need to ensurethat Digital Investment Managers have adequate algorithm and technology governance policies
and processes in place to address the specific risks arising from such a technology-driven
business model.
From a governance perspective, the FSRA expects that Digital Investment Managers will
establish internal governance structures that enable its Board and Senior Management to haverobust oversight and control over the design, performance, deployment and security of

12

Abu Dhabi Global Market (ADGM) was established on 21 October 2015 as a broad-based
international financial centre in the Emirate of Abu Dhabi, with its own civil and commercial laws
independent from the rest of the United Arab Emirates.

22

algorithms. The roles and responsibilities of all personnel who oversee the design, performanceand integrity of algorithms must also be clearly defined.
In terms of the development and testing of the algorithm, Digital Investment Managers must
maintain proper documentation explaining the decision tree or logic of the algorithm to ensurethat the outcomes produced by the model are explainable, traceable and repeatable. The Digital
Investment Manager must also ensure the relevance of any data or assumptions upon which thealgorithm is based, and that the Risk Profile Questionnaire completed by clients takes into
account potential behavioural biases that may lower the accuracy of client responses. The
Digital Investment Manager must carry out sufficient testing to demonstrate that its algorithm
meets these principles. Where appropriate (e.g., in the case of a complex algorithm), the FSRA
may require a third-party audit to validate the performance outcomes of the algorithm as
purported.
Additionally, given their heavy dependence on collecting and processing client data and therisks of cyberattacks to their automated and largely digital mode of operations, the FSRA
requires Digital Investment Managers to implement robust data security policies and systems
to ensure compliance with all relevant data protection regulations, including the ADGM’s Data
Protection Regulations.
FCA (United Kingdom)
It is essential that key oversight functions, including compliance and risk management, keep
pace with technological advancements. In the absence of appropriate systems and controls, theincreased speed and complexity of financial markets can turn otherwise manageable errors into
extreme events with potentially wide-spread implications.
The Financial Conduct Authority (FCA) deems it good practice to review how trading
algorithms are used; develop appropriate definitions; ensure all activities are captured; identify
any changes to algorithms; and have a consistent methodology across the testing anddeployment of AI and ML. Markets in Financial Instruments Directive (MiFID II) requires
firms to develop processes to identify algorithmic trading across the business. These can beeither investment decisions or execution algorithms, which can be combined into a single
strategy. Firms are also required to have a clear methodology and audit trail across the business.
Approval and sign-off processes should ensure a separation of validation and development a
culture of collaboration and challenge and consistency of a firm’s risk appetite. Whilst thealgorithms are field-deployed, it is a requirement to maintain pre-trade and post-trade risk
controls, real-time monitoring of algorithms in deployment, with the ability to kill an algorithm
or a suite of algorithms centrally, a functionality commonly known as the kill-switch.
It is a best practice, but not a requirement, to have an independent committee to verify thecompletion of checks. However, under the SM&CR, a firm’s governing body would be
expected explicitly to approve the governance framework for algorithmic trading, and its
management body should identify the relevant Senior Management Function(s) withresponsibility for algorithmic trading.

23

CanadaWith respect to the algorithmic trading method in Canada, under National Instrument 23-103
Electronic Trading and Direct Electronic Access to Marketplaces 13 and the Investment
Industry Regulatory Organization of Canada (IIROC) Notice 12- 0364 – Guidance Respecting
Electronic Trading, firms are required to ensure that algorithmic trading systems used byparticipants to route orders to marketplace have been adequately tested prior to being“engaged”, including the ability of the firm to immediately disengage the operation of thealgorithmic trading system should the need arise. It is expected that firms ensure, at a minimum,that each algorithmic trading system has been tested under various market conditions to identify
problematic outcomes related to the operation of the algorithmic trading system.
Also, firms should have built-in features or functionality that prevent (or provide real-timealerts) certain pre-programmed order or trade parameters from being exceeded (e.g., certain
volume, order or price limits). Firms should also have an “override” functionality which either
automatically disengages the operation of the algorithmic trading system or permits the firm to
do so remotely. Additionally, as part of their supervisory policies and procedures, firms are
required to maintain a written record with sufficient details to demonstrate the testing of thealgorithmic trading system undertaken, including details of the testing conducted by thealgorithmic trading system provider.
Further, at the national level, consumer protection and personal information protection laws
are expected to apply to AI tools and services. Quebec AMF is of the view that it might become
necessary to make new stakeholders accountable for their operations or to request
certification/IT audit of the underlying systems. Except as expressly stated, existing laws andregulations administered by the Canadian Securities administrators (CSA) are technology
neutral. For example, as stated in the CSA Staff Notice 31-342 Guidance for Portfolio
Managers Regarding Online Advice, 14 the applicable rules are the same if the activities are
conducted under the traditional model of interacting with clients face-to-face and if a portfolio
manager uses an online platform.
On December 4, 2018, the Montréal Declaration for Responsible AI 15 was published, a paper
drafted by leading AI institutes, AI researchers, and other stakeholders in Canada, aimed at
helping guide individuals, organisations, companies and governments make responsible andethical choices when building and utilising AI technology. The Montréal Declaration for
Responsible AI provides effective supervisory and oversight principles for AI and ML
development and deployment.

13

CSA National Instrument 23-103 Electronic Trading and Direct Electronic Access to Marketplaces,
https://lautorite.qc.ca/fileadmin/lautorite/reglementation/valeurs-mobilieres/23available
at:103/2014-03-01/2014mars01-23-103-vofficielle-en.pdf

14

CSA Staff Notice 31-342 Guidance for Portfolio Managers Regarding Online Advice, September
2015, available at: https://lautorite.qc.ca/fileadmin/lautorite/reglementation/valeurs-mobilieres/0avis-acvm-staff/2015/2015sept24-31-342-avis-acvm-en.pdf

15Montréal Declaration Responsible AI, December 2018, available at:https://docs.wixstatic.com/ugd/ebc3a3_c5c1c196fc164756afb92466c081d7ae.pdf

24

BaFin (Germany)
BaFin published its study Big data meets artificial intelligence in 2018. The study was triggeredby the fact that the entities BaFin supervises store and analyses data in very different ways, andthat this impacts how financial services are being provided. Its objective is to provide anunderstanding of these trends as a basis for discussing the supervisory and regulatory
implications. The report stresses consumer trust and data sovereignty as key success factors of
BDAI applications and innovation and identifies the following supervisory and regulatory
implications:
1. Supervision of firmsa) BDAI Governance• When designing partially automated processes, it is important to ensure that they areembedded in an effective and appropriate manner. Ultimately, responsibility remains withthe senior management of the firm. Appropriate documentation is required to ensure this.
• It is the responsibility of firms to guarantee the explainability (the ability to determine themain factors influencing a specific individual decision that has been reached by a system)and traceability of BDAI-based decisions. The study shows how new approaches could
provide at least some insight into how models work and the reasons behind decisions, evenin the case of highly complex models, thereby preventing models from being categorised
purely as black boxes.
• Supervisory and regulatory authorities will therefore not accept any models presented as an
unexplainable black box. In addition, a better understanding of models provides anopportunity to improve the analysis process – allowing, for instance, the responsible units
in the supervised firm to identify overfitting and data bias.
• According to the study, it may be necessary to further develop established governanceconcepts, such as the principle of dual control, and to apply these to automated processes.
For example, one could think about introducing special safeguards for certain particularly
risky BDAI applications, safeguards already used in other technological applications (inaviation, speed is measured using several independent systems - the back-up would be analgorithm).
• Due to the complexity of the applications, also resulting from the type and scope of data
used, consideration should be given as to whether the process results should also bemonitored in the future, in addition to the documentation requirements described above. For
example, this could be done by evaluating the results produced by an algorithm in a test
scenario set by the supervisory authorities.
b) Fighting financial crime and preventing conduct violations:• Exploiting BDAI potential in compliance processes - BDAI can improve the detection rateof anomalies and patterns, and thus increase the efficiency and effectiveness of compliance25

processes, such as money laundering detection or fraud prevention. In addition, BDAI canalso play a role in monitoring employee conduct.
• Preventing criminals from turning to less advanced firms - Should the use of BDAI
technologies result in far more efficient ways to detect money laundering, criminals could
potentially turn to firms that are less advanced in this area. It is therefore necessary tomonitor whether this will materialise.
• Defining supervisory requirements for the explainability and effectiveness of algorithms
for detecting financial crimes - Supervisory and regulatory authorities need to discuss
whether it is necessary to define specific minimum standards for the
explainability/traceability and effectiveness of the methods used. The results of thealgorithms would have to be traceable to the extent that the supervisory authorities canmonitor them and that the competent authorities, e.g., law enforcement agencies, can makeuse of them. In addition to performing comparisons with specific minimum standards that
may need to be developed, supervisory authorities could also determine the effectivenessof algorithms by benchmarking them against the algorithms of other providers.
c) Internal models subject to supervisory approval• Possible uses of BDAI in models requiring supervisory approval - Supervised firms may
also wish to use BDAI methods in models that are subject to approval by the supervisory
authority, for example, the internal ratings-based approach for banks. This is likely to be
the case if the use of BDAI leads to more precise results than the combination of methods
and data currently used. In this way, BDAI could potentially be used to improve current
methods, for instance, those used to optimise the assessment of individual counterparty
default risks. In addition, a better portfolio view could possibly alter the weighting ofindividual risks as concentration risks could be addressed more appropriately, for instance,by analysing the degree of interconnectedness between borrowers.
• Actual effects of BDAI use on capital requirements are likely to be limited. It is worth
noting that there are currently no BDAI applications observed or authorised in thealgorithms of such models. It should also be noted that the authorised models in existence
today are highly advanced, for instance, with regard to the scope of the data used, including
external data. In the banking sector, it should also be borne in mind that the output floor
approved as part of the finalisation of the Basel III reforms would additionally limit theactual effects on capital requirements.
• Defining prerequisites for BDAI use in models requiring supervisory approval
The use of BDAI models would always be subject to a corresponding approval on a caseby-case basis. Beyond the individual case, the question could be asked as to whether all
BDAI methods are equally suited for use in models that require supervisory approval, or
whether there are methods that should be ruled out per se.
Furthermore, it is necessary to examine whether existing legal (minimum) standards forthe data used and for BDAI model transparency are sufficient or whether additional
requirements would be necessary.
26d) Handling information security risks• Information security risks increase as a result of greater use of BDAI - The growing
complexity caused by BDAI presents new challenges in managing information security
risks. Together with ever larger data volumes, the disaggregation of value chains supported
by BDAI creates a larger attack surface while simultaneously reducing each individual
provider's ability to control the data used and distributed.
• In addition, a new phenomenon has arisen: attacks on certain BDAI algorithms throughdata manipulation, e.g., in the form of adversarial or poisoning attacks. This leads to risks
that are very different to those encountered in traditional online banking, resulting in a risein both operational and reputational risks.
• BDAI and encryption systems for mitigating information security risks - Although the useof BDAI can result in an increase in information security risks, BDAI can also be used tomitigate such risks, for example, to analyse and detect danger. For instance, BDAI could
be used to identify irregularities in user behaviour in online banking, which could indicate
potential improper use. Certain encryption systems that allow BDAI methods to be useddirectly on encrypted data could also be used to reinforce resilience against such risks.
More recently the Deutsche Bundesbank also published a policy discussion paper on the useof AI/ML in the financial sector where it outlines considerations on the risks posed by the useof AI/ML and supervisory expectations for ML. In their view, successful ML applications
represent an important building block of digitalisation which can improve analysis, reaction
times and costs but it also creates new challenges, such as the characteristics of the “black box”,
issues to do with data quality and the model learning process. The paper calls for financial
market participants such as banks to maintain sound risk management processes, including
processes to identify and control relevant and material risks and suggests the main supervisory
focus should be on features of ML which are novel to current regulation and supervisory
practices. 16
China
In order to effectively cope with the challenges brought to intermediaries by new technologies
such as AI and ML, the CSRC issued the Measures for the Administration of Information
Technology Management of Licensed Corporation in December 2018 (CSRC No. 152,
hereinafter referred to as Measures).
First, this promotes the integration of information technology governance and corporate
governance, thus consolidating the foundation of information technology governance. The
Measures requires intermediaries to clarify the IT management responsibilities of senior
management. Also, it improves the distribution of power and responsibility through theestablishment of CIO and information technology governance committees.

16

https://www.bundesbank.de/resource/blob/598256/d7d26167bceb18ee7c0c296902e42162/mL/20
20-11-policy-dp-aiml-data.pdf

27

Second, the Measures requires intermediaries to ensure that the back-office management
capabilities should be consistent with the complexity of front-office application, strengthening
compliance and risk management. Taking into account the professionalism and complexity of
AI and ML, the “Measures” requires the compliance management department and the risk
management department to equip with information technology resources that are compatible
with the scale and complexity of business activities, and establish corresponding review,monitoring and inspection mechanisms.
Third, the Measures requires the enhancement of data governance. Due to the importance ofdata for AI and ML, it encourages intermediaries to exploit full value of data and explore dataapplications in business operations, risk management and internal control. On the other hand,adhering to the principle of “Tech for Social Good”, it requires intermediaries to strengthendata life-cycle management, including data security management, log recording andprohibitions on over-collect and customer data abuse.
Fourth, it requires strengthening the management of third-party providers. Given the fact thata large number of AI and ML applications are provided by third-party providers, the regulationof third-party providers is critical. Therefore, the Measures clarifies the responsibilities andbehavioural boundaries between intermediaries and third-party provider and requires thirdparty providers to file with CSRC if important information system services are provided tointermediaries.
Further improvements on regulatory measures based on CSRC’s recent practices.
First, based on traditional risk management framework, regulators should requireintermediaries to adopt technical methods to improve compliance and risk management
capabilities, ensuring those are compatible with the technology applied in the industry.
Second, regulators should require intermediaries to carefully select service providers, andcontinue to pay attention to service concentration, in order to prevent convergence of trading
strategies, triggering market sentiment resonance and extreme circumstance in which large
number of intermediaries are affected at the same time.
AMF (France)
In France, the Autorité de contrôle prudentiel et de résolution published, in conjunction withseveral other French financial regulators, including the Autorité des marchés financiers (AMF),
a guidance 17 stating that the use of artificial intelligence for the allocation of assets or internal
modelling of capital requirements may question the governance and risk management rules ofthe firm. Additionally, the reliability of algorithm, which first comes from data quality, is
instrumental in the decision-making process. Some precautions should be taken, notably
minimising the use of public external data, performing regular data quality checking andregular update of personal data with customers themselves.
17

Artificial intelligence: challenges for the financial sector, December 2018, available at:https://acpr.banquefrance.fr/sites/default/files/medias/documents/2018_12_20_intelligence_artificielle_en.pdf

28

To achieve these goals, several approaches may be considered, including:• Use of experts to validate the relevance of the variables used, eliminate those that are
unnecessary or a source of potential biases;• Use of safer and more traditional parallel processes on part of the test data;• Use of a standard dataset on the algorithms to regularly monitor both the relevanceand non-discriminatory aspect of the algorithms; and• Develop tools that would assess conceptual drift to control this specific risk of
automatic learning.
Also, there should be conditions for monitoring these algorithms to demonstrate their
explainability. The firms should be able to explain:•

Generally, the mechanisms and criteria followed by the algorithm during its analysis
process;•

For a given action such as a decision taken, or an advice provided, the objective
criteria and discriminant elements that conducted the algorithm, in the case examined,
to effect one action or propose a solution rather than another

Coupled with the explainability test, a number of tests on datasets - independent from thoseused for algorithm learning - could be considered to assess the quality of the results and themethodology for such tests should be defined.
Furthermore, some firms may consider maintaining a human intervention to verify theconsistency of the results of the algorithm. This may be the case in the sensitive areas of
information and advice provided to clients or AML-CFT. There may also have some challenges
relating to financial stability. For example, there are several algorithms coded with similarvariables, which may cause the high frequency trading programmes to converge towards thesame strategy. The resulting risk is to increase the pro-cyclicality and market volatility throughsimultaneous purchases and sales of large quantities. It would be therefore crucial to raise thebar of the risk analysis and management in such case to reduce more considerably the pricedifferentials.
MAS (Singapore)
The Monetary Authority of Singapore (MAS) issued a paper 18 setting out principles to promotefairness, ethics, accountability and transparency in the use of artificial intelligence and data
analytics in financial sector. This paper contains a set of generally accepted principles for theuse of AI and data analytics in decision-making in the distribution of financial products andservices, with a particular mention on building consumer confidence and trust in providing
their personal data for AI, such as selecting appropriate organisational governance measuresand adopting good data management practices. In developing this set of principles, MAS has
worked closely with key industry stakeholders and sought feedback from a range of financial

18

Principles to Promote FEAT in the Use of AI and Data Analytics in Singapore’s Financial Sector,
November 2018, available at:https://www.mas.gov.sg/-/media/MAS/News-and-Publications/Monographs-and-InformationPapers/FEAT-Principles-Updated-7-Feb-19.pdf

29

institutions and companies, who have offered diverse perspectives and inputs. The paper
provides some guidance on the four following principles:••••

Fairness (justifiability, accuracy and bias)
Ethics
Accountability (internal and external)
Transparency.

The paper also provides some illustration on actions to be taken by the firms to ensure they
comply with the principles. For example, the frequency of validation or tests of algorithms
depends on the materiality of the decision and the complexity of the model. For particularlymaterial and complex deep learning models, a firm requires more frequent validation andtesting. Conversely, for less material and more straightforward models, quarterly, annual oreven less frequent validations and tests suffice.
AFM (Netherlands)
The AFM believes it is vital to have key functions within the firm to safeguard the safety anduse of AI and ML techniques within the organisation. The risk and compliance functions play
an important role and should preferably be involved during the initial development process.
Senior management can set the conditions for AI and ML techniques and take responsibility
for techniques deployed in the firm. A clear governance framework will reflect the firms
control on its AI and ML techniques and on the data it feeds into the models.
Concerning AI and ML techniques and the data itself, the AFM believes it to be good practice
to check the quality and accuracy of the input data; make conscious decisions on the specific
AI and ML technique deployed; determine the level of explainability suitable to become andstay in control; be aware of the effect of using third-party providers; and have the skills in place
to test, monitor and challenge both internally and externally developed datasets and AI and ML
techniques. Clear documentation of the considerations and decisions made in the process help
to understand why firms chose a specific path.
AI and ML techniques also create new opportunities for firms to further strengthen andsafeguard the customer interest.
Securities and Exchange Commission (United States)
In October 2018, the Securities and Exchange Commission (SEC) launched theagency's Strategic Hub for Innovation and Financial Technology (FinHub) which was elevatedto a stand-alone Office of the SEC in December 2020. The FinHub coordinates the agency’s
oversight and response regarding emerging technologies in financial, regulatory, andsupervisory systems, including in the areas of artificial intelligence/machine learning,
distributed ledger technology (including digital assets), automated investment advice, anddigital marketplace financing. 19 Through internal and external engagement, FinHub helps
shape the SEC’s approach to technological advancements like those related to artificial
intelligence and machine learning.

19

For a list of relevant SEC staff speeches and statements related to AI/ML go tohttps://www.sec.gov/finhub.

30With respect to robo-advisers, the staff of the Division of Investment Management at the SEC
issued a guidance update in February 2017 that addressed certain unique considerations forrobo-advisers to keep in mind as they seek to meet their fiduciary and substantive regulatory
obligations under the Investment Advisers Act of 1940 (e.g., robo-advisers’ unique businessmodels, such as the level of overall human involvement, potentially limited face-to-faceinteraction with clients, and the use of algorithms). 20 Among other things, the guidance
identified the following considerations: (i) the substance and presentation of disclosures toclients about the robo-adviser and the investment advisory services it offers; (ii) the obligation
to obtain information from clients to support the robo-adviser’s duty to provide suitable advice;and (iii) the adoption and implementation of effective compliance programs reasonably
designed to address particular concerns relevant to automated advice. The staff also noted that
there may be a variety of means for a robo-adviser to meet its obligations to its clients underthe Advisers Act, and that not all of the issues addressed in its guidance will be applicable toevery robo-adviser.
Financial Industry Regulatory Authority (United States)
In April 2019, the Financial Industry Regulatory Authority (FINRA) formed an Office ofFinancial Innovation (OFI) that serves as the central point of coordination related to significant
financial innovations by FINRA member firms, particularly new uses of FinTech. To achievethis, OFI initiates outreach with various stakeholders, trains FINRA staff, publishes anddisseminates timely research and publications, maintains internal coordination across FINRA
departments, and collaborates with other regulators on matters related to FinTech. Key FinTech
areas monitored by FINRA that are impacting the securities industry include artificial
intelligence and related areas such as machine learning, deep learning and natural language
processing. Recent FINRA actions related to artificial intelligence are discussed below.
FINRA has solicited broker-dealers’ views on how FINRA can support FinTech innovationand helped educate firms on new regulatory issues. On July 30, 2018, FINRA issued a Special
Notice on Financial Technology Innovation in the Broker-Dealer Industry. 21 The Special
Notice requested comment on financial technology innovation in the broker-dealer industry. In
the Special Notice, FINRA specifically requested comment regarding supervision in thecontext of artificial intelligence. FINRA noted that as broker-dealers increase their use of
artificial intelligence, including chat bot-based services, they are grappling with how thesetools may fit in the current regulatory framework. FINRA pointed to examples that included
its supervision rule and previous guidance. FINRA Rule 3110 (Supervision) requires a brokerdealer to establish and maintain a system to supervise the activities of its associated persons
that is reasonably designed to achieve compliance with the applicable securities laws andregulations and FINRA rules. In addition, FINRA restated that: “[A]s the use of algorithmic
strategies has increased, the potential of such strategies to adversely impact market and firmstability has likewise grown. When assessing the risk that the use of algorithmic strategies
creates, firms should undertake a holistic review of their trading activity and consider
implementing a cross-disciplinary committee to assess and react to the evolving risks
20

Robo-Advisers, IM Guidance Update No. 2017-02,
https://www.sec.gov/investment/im-guidance-2017-02.pdf

21

FINRA, Special Notice on Financial Technology Innovation in the Broker-Dealer Industry, July 30,
2018, available at: https://www.finra.org/sites/default/files/Special-Notice-073018.pdf

31

February2017,

availableat:associated with algorithmic strategies.” In the Special Notice, FINRA sought to better
understand the challenges or issues that broker-dealers may face in deploying artificial
intelligence tools and requested comment on any measures that FINRA could take to clarify or
adapt its rules and processes in light of the evolving uses of such tools. Relatedly, FINRA
issued Regulatory Notice 15-09 to address supervision and control practices for firms thatengage in algorithmic trading strategies.
In response to the Special Notice, commenters recommended that FINRA undertake a broad
review of the use of artificial intelligence in the securities industry to better understand thevaried applications of the technology, their associated challenges, and the measures taken bybroker-dealers to address those challenges. Based on feedback received, FINRA engaged in an
active dialogue with the industry for over a year and held meetings with over two dozen market
participants, including broker-dealer firms, academics, technology vendors, and service
providers to learn more about the use of artificial intelligence in the securities industry. In June
2020, FINRA published a white paper on Artificial Intelligence (AI) in the Securities Industry 22that summarises the key findings from FINRA’s review.
With respect to educating firms on new regulatory issues, FINRA issued a report on
Technology Based Innovations for Regulatory Compliance (RegTech) in the Securities
Industry 23 in September 2018. The report was a result of a focused review to learn more about
the emergence and adoption of RegTech tools within the securities industry. The reportdiscussed the use of machine learning with respect to regulatory intelligence programs,
surveillance and monitoring, and investor risk assessments. FINRA encouraged comments on
the report, including areas where guidance or modifications to FINRA rules may be desired tosupport adoption of RegTech tools while maintaining investor protection and market integrity.
In addition to the report, FINRA hosted the 2019 FINRA RegTech Conference. The conference
brought together regulators, thought leaders and industry practitioners to learn more about theuse of RegTech tools, and related opportunities and challenges. Panelists discussed key
innovative technologies that are transforming compliance functions including artificial
intelligence.
FINRA also included regulatory technology as a highlighted item in the 2019 FINRA Risk
Monitoring and Examination Priorities Letter. FINRA recognised that broker-dealers are using
a variety of innovative RegTech tools to make their compliance efforts more efficient, effective
and risk-based. As stated in the letter, “FINRA will engage with firms to understand how they
are using such tools and addressing related risks, challenges or regulatory concerns, including
those relating to supervision and governance systems, third-party vendor management,
safeguarding customer data and cybersecurity.”
CSSF (Luxembourg)
In December 2018, the Commission de Surveillance du Sector Financier (CSSF) issued a white
paper on Artificial Intelligence describing opportunities, risks and recommendations for the22

FINRA, Artificial Intelligence in the Securities Industry, June 2020, available at:https://www.finra.org/sites/default/files/2020-06/ai-report-061020.pdf

23

FINRA, Technology Based Innovations for Regulatory Compliance (“RegTech”) in the Securities
Industry, September 2018, available at:https://www.finra.org/sites/default/files/2018_RegTech_Report.pdf

32

financial sector. 24 The paper has no binding value vis-à-vis the supervised institutions, but it
includes practical guidance regarding the risks associated with the use of AI and ML and related
recommendations to take into account.
Among the risks covered in the paper, there are data-related risks (e.g., data quality). Given theimportance of data in AI and ML, having a solid data governance in place (e.g., setting clear
roles and responsibilities for data ownership and defining processes to identify and fix data
quality issues) is paramount to ensure that the data is of good quality. In case firms use external
data sources, this requires having first verified their reliability (e.g., via due diligence of theexternal data service provider) and ensuring the data sources are adequate for their intended
use. In addition, they should carefully review the integration of the AI and ML into the businessprocess and establish controls performed by humans whenever there is a critical decision step.
In addition, the integration of the AI and ML into the business process should include controls
performed by humans whenever there is a critical decision step.
An important section of the paper focuses on ethical aspects such as bias and discrimination,
data privacy, accountability, explainability and auditability. The importance of these aspects is
proportional to the potential impact that the AI and ML solution might have on the customer.
The paper provides an overview of some existing techniques to help detect/prevent bias and to
improve explainability of black box solutions.
Institutions should assume clear responsibility and accountability for the actions and decisions
taken by automated AI and ML systems and processes. This responsibility cannot be delegatedto a machine. Ultimately, responsibility should rely on the senior management of the institution
which integrates the AI and ML logic into its business processes. When using third-party
solutions, clear liability provisions may be defined at contractual level.
Another important aspect mentioned in the paper concerns the monitoring and update of the AI
and ML solution, to promptly detect performance deviations or discriminative results andensure that results remain accurate over time.
The lack of adequate internal skills to maintain the AI solution may lead to difficulties in
updating the model or over-reliance on external parties. Therefore, firms should ensure asufficient level of internal AI skills to understand, develop and supervise the solution, andshould consider setting up multidisciplinary teams involving data scientists, IT, risk andcompliance staff. The paper also describes the main categories of security attacks affecting AI
and ML solutions and related defence techniques, recommending institutions to perform
technical watches over these continuously evolving topics, in order to remain up-to-date froma technical perspective.
Finally, another vigilance point concerns systemic risks, which may be caused, for example,
by the use of a common trading algorithm by many institutions, especially if the algorithm
contains errors. Therefore, there should be a monitoring of potential systemic effects to issue
warnings when appropriate.
24

Artificial Intelligence: opportunities, risks and recommendations for the financial sector, December
2018, available at:https://www.cssf.lu/fileadmin/files/Publications/Rapports_ponctuels/CSSF_White_Paper_Artifici
al_Intelligence_201218.pdf

33

ANNEX 2
Guidance published by supranational bodies
Financial Stability Board (FSB)
On November 2017, the FSB published a report considering the financial stability implicationsof the growing use of AI and ML in financial services. 25
The report indicates that financial institutions are increasingly using AI and ML in a range ofapplications across the financial system including front office client interactions, risk
management, trade execution optimisation, as well as regulatory compliance and fraud
detection.
The FSB’s analysis reveals a number of potential benefits and risks for financial stability thatshould be monitored as the technology is adopted and as more data becomes available. Theyare:•

The more efficient processing of information, for example in credit decisions, financial
markets, insurance contracts and customer interactions, may contribute to a more
efficient financial system. The applications of AI and ML by regulators and supervisors
can help improve regulatory compliance and increase supervisory effectiveness.

•

Applications of AI and ML could result in new and unexpected forms ofinterconnectedness between financial markets and institutions, for instance based on
the use by various institutions of previously unrelated data sources.

•

Network effects and scalability of new technologies may give rise to third-party
dependencies. This could in turn lead to the emergence of new systemically important
players that could fall outside the regulatory perimeter.

•

The lack of interpretability or auditability of AI and ML methods could become amacro-level risk. Similarly, a widespread use of opaque models may result inunintended consequences.

•

As with any new product or service, it will be important to assess uses of AI and ML
in view of their risks, including adherence to relevant protocols on data privacy, conduct
risks, and cybersecurity. Adequate testing and ‘training’ of tools with unbiased data andfeedback mechanisms is important to ensure applications do what they are intended todo.

25

Artificial intelligence and machine learning in financial services, Market developments andfinancial stability implications, November 2017, available at:https://www.fsb.org/wp-content/uploads/P011117.pdf

34

International Monetary Fund (IMF)
The IMF published Fintech and Financial Services in June 2017, 26 sets an economic
framework for considering fintech channels (including AI and ML) that might provide
solutions that respond to consumer needs for trust, security, privacy and better services, change
the competitive landscape and affect regulation. It highlights numerous potential opportunities
for AI solutions within financial services including, for example, tracking regulatory changes
and interpreting new regulations.
Additionally, the IMF report highlighted areas of its own work that could potentially utilise AI
and ML once the issues of provenance of data, and data privacy have been addressed before
basing analysis or grounding policy advice on AI and ML outputs.
Institute of Internal Auditors (IIA)
The IIA published guidance 27 introducing an AI auditing framework to help organisations
evaluate, understand, and communicate the degree to which AI negatively or positively affects
the organisation’s ability to create value in the short, medium, or long term.
Information Systems Audit and Controls Association (ISACA)ISACA published a white paper 28 discussing the information technology that auditors need toknow when auditing AI reliant businesses in the absence of a mature auditing framework andAI-specific regulations or standards.
European Commission
The European Commission released draft ethics guidelines for AI in April 2019. 29 The paper
highlights the importance of trust in ensuring a human-centric approach to AI and outlines threecomponents that should be met throughout the AI system’s life:••

It should be lawful, complying with all applicable laws and regulations;
It should be ethical, ensuring adherence to ethical principles and values; andit should be robust, both at a technical and social level.

Furthermore, the paper provides guidance on how such principles can be operationalised insocio-technical systems. The guidance is being piloted by stakeholders and feedback will bepresented to the European Commission in early 2020.

26

Fintech and Financial Services, June 2017, available at https://www.imf.org/en/Publications/StaffDiscussion-Notes/Issues/2017/06/16/Fintech-and-Financial-Services-Initial-Considerations-44985

27

Artificial Intelligence, Internal Audit’s Role, and Introducing a New Framework, 2017, available
at: Artificial Intelligence, Internal Audit’s Role, and Introducing a New Framework

28

Auditing Artificial Intelligence, 2018, available at: Auditing Artificial Intelligence29

Ethics Guidelines for Trustworthy AI, April 2019, available at: https://ec.europa.eu/futurium/en/aialliance-consultation/guidelines#Top

35

The European Parliament and the Council also published a legislative proposal for laying down
harmonised rules on artificial intelligence (artificial intelligence act) and amending certain
union legislative acts. 30 The proposed regulatory framework on artificial intelligence has thefollowing specific objectives:• ensure that AI systems placed on the Union market and used are safe and respect
existing law on fundamental rights and Union values;• ensure legal certainty to facilitate investment and innovation in AI;• enhance governance and effective enforcement of existing law on fundamental rights
and safety requirements applicable to AI systems;• facilitate the development of a single market for lawful, safe and trustworthy AI
applications and prevent market fragmentation.
The International Technology Law Association (ITechLaw)
ITechLaw 31 published a paper in May 2019 32 providing a principled framework that encourage
the responsible development, deployment, and use of artificial intelligence. The principles
relate to:
•
•
•
•
•
•
•
•

Ethical Purpose and Societal Benefit
Accountability
Transparency and explainability
Fairness and Non-discrimination
Safety and Reliability
Open Date and Fair competition
Privacy
AI Rights and Intellectual Property

Organisation for Economic Co-operation and Development (OECD) and the G20
On May 2019, OECD adopted principles along with recommendations on AI, 33 produced by agroup of leading international experts. The OECD AI principles are the first such principles
signed up by governments.
The OECD’s five principles are:•Inclusive growth, sustainable development and well-being: stakeholders should
pursue beneficial outcomes, such as augmenting human capabilities and enhancing
30

Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) andAmending Certain Union Legislative Acts, The European Parliament and the Council, April 2021,
available at:https://digital-strategy.ec.europa.eu/en/library/proposal-regulation-laying-down-harmonised-rulesartificial-intelligence-artificial-intelligence31

A multi-disciplinary group of 54 technology law experts, researchers and industry representatives
from 16 countries.

32

Responsible AI: A Global Policy Framework, May 2019, available at:https://www.itechlaw.org/sites/default/files/ResponsibleAI_Principles.pdf

33

OECD Principles on AI, May 2019, available at: http://www.oecd.org/going-digital/ai/principles/36

creativity, advancing inclusion of underrepresented populations, reducing economic,
social, gender and other inequalities, and protecting natural environments.
•

Human-centred values and fairness: AI actors should respect the applicable laws,
human rights and democratic values, such as freedom, dignity and autonomy, privacy
and data protection, non-discrimination and equality, fairness, social justice; as well asinternationally recognised labour rights.

•

Transparency and explainability: AI actors should disclose meaningful information tofoster a general understanding of AI systems, make stakeholders aware of theirinteractions with AI systems, including in the workplace, enable those affected by an
AI system to understand the outcome and those adversely affected by an AI system tochallenge the outcome•

Robustness, security and safety: AI systems should be robust, safe and securethroughout their entire lifecycles. AI actors should ensure traceability of datasets,
processes and decisions making during the AI system lifecycle to enable AI actors toanalyse AI systems’ outcomes. AI actors should also apply a systematic risk
management approach throughout AI systems’ lifecycles.

•

Accountability: AI actors should be accountable for the proper functioning of AI
systems.

The OECD makes five recommendations:
•
•
•
•
•

Invest in AI research and development;
Foster a digital ecosystem for AI;
Shape an enabling policy environment for AI;
Build human capacity and preparing for labour market transformation; andEncourage international co-operation for trustworthy AI.

The G20 endorsed these OECD AI principles and recommendation in June 2019.

37

ANNEX 3
Consultation Report Feedback Statement
The use of artificial intelligence and machine learning by market intermediaries andasset managers
IOSCO received 22 responses to the Consultation Report. All respondents recognised theimportance of a regulatory frameworks around the use of AI and ML and the importance ofhigh standards of conduct.
IOSCO amended the final report in response to the feedback, including revising and clarifying
text. This included, for example, clarifying the definition of AI and ML within the report.
Definition
Question 1: Do you agree with the proposed definition of AI and ML?
The vast majority of respondents were supportive of the need to have a clearer definition ofboth AI and ML.
Some respondents suggested that the definitions be made more succinct and precise to avoid
misinterpretation. Another respondent suggested a principles-based approach to defining
AI/ML. Respondents also recognised the differing pace of innovation across the financial
industry and suggested that the definitions should also account for future developments in thisfield.
IOSCO has taken this feedback onboard and has revised the definition of both terms within the
final report (please refer to Chapter 2).
Risks and challenges
Question 2: Do you see any risks or challenges around AI and ML which are notmentioned in the report?
Some respondents identified a number of challenges that were not mentioned in theConsultation Report, including:•••Outsourcing practices
The quality of datasets on overall performance
Ethical issues around diversity and inclusion.

IOSCO has revised the drafting in Chapter 3 to reflect this feedback.
Guidance
Question 3: Do you agree that the guidance set out in Chapter 6 of the Consultation
Report is appropriate to address the potential risks associated with the general use of AI
and ML by market intermediaries and asset managers? If not, please provide details.

38

Measure 1 – There were a contrast of views among respondents in relation to the skills requiredby senior management. Some respondents suggested that the requirement to have seniorindividual(s) to have the relevant skill set and knowledge (presumably about AI and ML) sign
off on deployments and updates was not needed and that proportionality should be consideredin implementing internal governance processes. Some other respondents agreed with themeasure stating that regulators should consider requiring firms to have designated members ofsenior management responsible for oversight of the development, testing, deployment,
monitoring and control of AI/ML systems.
Measure 2 – Most respondents supported the need for continuous testing. However, some
respondents queried whether AI and ML systems required continuous testing given the
resources involved in and benefits associated with undertaking real-time testing. The final
report clarifies proportionality should be considered when determining the frequency ofcontinuous testing.
Measure 3 – Most respondents were supportive of the need for senior management to have anadequate level of skills to oversee the different stages in the development of AI and ML. One
respondent stated that it is not practical to expect such personnel to have the same level ofunderstanding of AI/ML. This respondent suggested that compliance and risk management
personnel should help design, understand and test the overall process, the objective of the toolor strategy and how it should generally operate or behave. This feedback is reflected in the
final report.
Measure 4 – Most respondents agreed that firms should have a clear service level agreement
and contract with all third-party service providers outlining the scope and responsibility of theprovider as well as clear performance indicators. Most respondents also welcomed the
proposed measure but noted the growing complexity of the third-party/outsourcing
relationships emerging in AI/ML systems, and respondents stated that further guidance may beneeded for IOSCO members. This risk has been highlighted in the final report.
Measure 5 – Some respondents’ concerns around this measure focused on the need for anddegree of disclosure should depend on the nature of the AI/ML use case and be proportionate
to the risks posed to the stakeholders. However, the majority of respondents agreed that client
disclosures were a key part of the guidance in mitigating the risks and issues posed by AI and
ML. This feedback has been incorporated within the final report.
Measure 6 – The majority of respondents agreed that mitigating against biases is an importantrisk to manage particularly regarding the use of predictive machine learning algorithms. Other
respondents agreed that understanding the technology is fundamental to the overall AI strategy,
and it is essential to ensure ML models are accurate and consistent throughout changing
conditions. Where respondents disagreed with the proposals this centred around proportionalityof the measure and the validity of this measure when applied to differing use cases.
Disclosure of Information
Question 4: Do you disclose information to customers / clients on the use of AI and ML?
If yes, please indicate what kind of information is disclosed.
No substantive feedback was received on this question.
39

Question 5: What factors do you need to take into account when considering theappropriate level of information that should be disclosed to clients (including prospective
clients) and other relevant stakeholders around the firm’s use of AI and ML algorithms?
There was also limited feedback to this Consultation question. However, one respondent statedthat the guidance should allow firms to make a risk assessment and determine the appropriate
level, rather than mandating a single standard.
Proportionality
Question 6: How do you consider and apply proportionality to your systems and controls
over the use of AI and ML?
The majority of respondents agreed that proportionality should be applied to the measures thatwere proposed in the Consultation Report. Firms also stated strong views that that the size ofthe regulated entity should not be the only parameter, when considering proportionality andthat the activity being undertaken is more important when assessing the level of regulation.
This feedback is reflected within the final report.

40