Source: https://iclg.com/practice-areas/fintech-laws-and-regulations/1-ai-in-the-fintech-sector-the-importance-of-transparency-and-explainability
Timestamp: 2020-07-06 21:05:53
Document Index: 144065342

Matched Legal Cases: ['art 1', 'art 2', 'art 3', 'art 1', 'art 1', 'art 2', 'art 2', 'art 2', 'art 3', 'art 1']

Fintech 2020 | AI in the Fintech Sector: The Importance of Transparency and Explainability | ICLG
AI in the Fintech Sector: The Importance of Transparency and Explainability: Fintech 2020
ICLG - Fintech Laws and Regulations - covers a broad overview of common issues in fintech laws and regulations in 50 jurisdictions
ICLG.com > Practice Areas > Fintech > AI in the Fintech Sector: The Importance of Transparency and Explainability
3. How are Financial Institutions and Fintech Organisations Using AI?
4. Why Explaining AI is Important
5. How Fintech Organisations Can Explain Their Use of AI – the FCA and ICO Approach
6. Why is the GDPR Relevant?
8. What Does the Project ExplAIn Guidance Cover?
As Artificial Intelligence (or “AI”) solutions become more prevalent in the Fintech sector, customers and regulators are demanding increased information regarding what this new technology does and how it is being used. In Europe there is also a strong focus at governmental level on the ethical deployment of AI, and transparency forms an important part of this. Being able to explain AI, particularly where it is used to make decisions about people (for example, whether or not credit is given to an individual) is often seen by regulators as essential for those organisations wishing to bring their customers, regulators and supply chain along with them on their AI journey. But is it always possible (or sensible) to explain AI? In this chapter we look at how Fintech organisations are using AI, why explaining AI is important, and how (according to the UK’s financial and data regulators) organisations should go about explaining their AI use.
What is AI and what are AI-assisted decisions or outputs?
Artificial Intelligence is not a new concept, but rapid increases in computational power and volumes of data, together with reducing costs for data storage, mean that the range and complexity of AI solutions is fast developing.
AI is an umbrella term used to describe a range of technologies and approaches that try to mimic human thought to solve tasks. Examples include machine learning (a sub-set of AI) and natural language processing.
AI-based systems can be purely software based, acting in the virtual world (for example, voice assistants, search engines and speech or facial recognition), or can be embedded in hardware devices. Examples of these include autonomous cars, wearable technology and other internet of things applications.
Big data is also often intrinsically linked to AI. In its 2017 guidance on big data, AI and machine learning, the UK’s data regulator (the Information Commissioner’s Office, the “ICO”) described big data as “an asset that is difficult to exploit” and AI as “the key to unlocking” its value.
The field of AI is generally divided into two categories: (i) general AI, which has broad applicability and could solve any tasks requiring human intelligence – this is not yet a reality; and (ii) narrow AI, which is basically algorithms that are designed to solve one or more particular problems.
Decisions and outputs made using AI can also be divided into categories. Outputs can be classed as predictions (e.g. you will not default on a loan), recommendations (e.g. you will like this advert) or classifications (e.g. this is spam), whereas decisions are either fully automated or involve human intervention (often referred to as having a “human in the loop”).
AI is an area of increasing focus and development for financial institutions. Firms (both new market entrants and traditional players) are using AI to add value to, and personalise, their current customer offerings as well as to help enhance their back-office functions and regulatory compliance, particularly in relation to the fight against financial crime. In their 2019 survey on machine learning in UK financial services, the Bank of England (“BoE”) and the Financial Conduct Authority (“FCA”) found that, of respondent firms, 57% use AI for risk management and compliance purposes, 39% for customer engagement and 25% for sales. For example, funds are using AI to optimise execution of trades and better analyse market impact on large trading positions, whilst banks, such as Barclays, are using chatbots to automate client interactions and have adopted AI into their internal anti-money laundering and fraud detection processes.
Fintech organisations, and their regulators, are also starting to consider how best to explain the AI they use to their customers, supply chain and regulators.
Being able to explain the AI being used, at least at some level, is seen by many as good business sense. For example, it:
enables organisations to build consumer and regulator trust in their offering, and is in certain circumstances a regulatory requirement (e.g. it is a General Data Protection Regulation (“GDPR”) requirement in the UK and EU – see below);
improves an organisation’s internal governance. Explaining the AI to affected individuals requires those within the organisation to understand the models, choices and processes used along with any AI decisions which are made. This gives the organisation more oversight and helps it ensure the AI systems meets the organisation’s objectives; and
can lead to better outcomes, as organisations identify and mitigate discriminatory outcomes which may be present in traditional systems and human decision making.
That said, explainability does bring with it some challenges. Industry research1 in the UK has highlighted a number of issues which could limit the information organisations are willing to share regarding their use of AI. There are concerns that sharing too much information can actually lead to distrust due to the complex and sometimes opaque nature of AI. There may also be sensitivities around inadvertently disclosing commercially sensitive information about how an AI model or system works or that disclosing too much information may enable individuals to exploit the AI model, particularly where AI is used to identify wrongdoing or misconduct (such as fraud detection). In addition, trade-offs may need to be made, for example between a system’s accuracy and its transparency. In the UK, the data regulator (the ICO) considers that these challenges can be mitigated, for example by using a data protection impact assessment (see below). In its view, organisations should start with the assumption that they will be as transparent as possible about the rationale of an AI system and work back from there, justifying and documenting where they consider it necessary to limit information.
In the UK, both the FCA and the ICO are collaborating with the UK’s national institute for data science and AI (The Alan Turing Institute or “the Turing”) to look at how organisations can explain their AI use.
In February 2020, the FCA and the Turing announced that they are working on a year-long collaboration on AI and transparency in financial markets. They will look at the different types of information that are relevant – both information relating to the “inner workings” of the AI model (model transparency) and information that concerns the process of developing and using the AI system (process transparency). They will also consider who should have access to the information, acknowledging there are likely to be a broad range of stakeholders with a transparency interest. These range from stakeholders within the firm that deploy the AI, to customers, regulatory stakeholders and other third parties, including the general public. While this is still at a very early stage, the FCA blog launching the project identified three key considerations:
Rationale-dependence: the reasons that motivate stakeholder interest in transparency.
Stakeholder-specificity: the type of information provided may need to differ between stakeholder types.
Use case-dependence: the provision of different types of information may depend on the nature of the particular use case.
It suggested that firms may find it helpful to develop a “transparency matrix” that, for a particular use case, maps different types of relevant information against different types of relevant stakeholders. Applying the matrix to different use cases would then offer a way to integrate considerations of rationale-
dependence, stakeholder specificity and use-case dependence in a systematic manner.
The ICO’s work in this area is more advanced than the FCA’s. In April 2018, it was tasked (in the UK Government AI sector deal) to develop guidance with the Turing to assist in explaining AI decisions. They subsequently launched Project ExplAIn and published draft guidance for “Explaining decisions made with AI” in December 2019.2 This draft guidance was open for consultation until 24 January 2020, and the final version is expected later in 2020. Although not a statutory code of practice, it sets out good practice for explaining AI decisions to individuals and clarifies the data protection provisions associated with this.
While the guidance is primarily relevant to those Fintech organisations caught by the GDPR (which can include non-EU organisations given the GDPR’s extra-territorial scope), its practical approach means it is of interest to any Fintech organisation that decides (or is required) to explain its AI decision making processes. We therefore set out below details of why the GDPR, and the Project ExplAIn guidance, is relevant, and what the guidance covers.
Within the EU and UK, the GDPR applies whenever an AI model processes personal data (see “What is personal data?” below). While some AI models do not use personal data, it is common for those in the Fintech space to use or create personal data when being tested and when in operation.
The GDPR is drafted in a technology-neutral manner, and so does not explicitly reference AI. However, it does contain specific provisions on large-scale automated processing of personal data (including profiling), which means it will apply where AI is used to make a prediction or recommendation about someone. For example, it gives individuals:
a right to be informed of the existence of solely automated decision-making (including profiling) producing legal or similarly significant effects. An example may include where AI is used to determine whether or not an individual is granted credit. In such circumstances, the individual is entitled to receive meaningful information about the logic involved in the decision, as well as the significance and the envisaged consequences of such processing for them;
a right of access in relation to that information, which includes the right to obtain an explanation of a solely automated decision after it has been made;
a right to object to the processing of their personal data, specifically including profiling, in certain situations. For example, they have an absolute right to object to profiling for direct marketing purposes; and
a right not to be subject to a solely automated decision producing legal or similarly significant effects, subject to certain exemptions. Where an organisation is relying on one of the exemptions, that organisation must adopt suitable measures to safeguard individuals, including the rights to obtain human intervention, to express their view and to contest the decision.
Even where an AI-assisted decision is not part of a solely automated process (because there is meaningful human involvement), the GDPR imposes general requirements to provide information to individuals whose data is being processed about that processing (Articles 12–14). In addition, the main GDPR principles (Article 5) will still apply, with the principles of fairness, transparency and accountability having particular relevance to explainability:
Fairness involves considering how an individual’s interests are affected. If a decision is made using AI (whether solely automated, or merely AI-assisted) without some form of explanation about the decision, this is unlikely to be fair.
Transparency is also about being clear and open with individuals about how and why their personal data is being used.3 The ICO considers it unlikely that processing will be considered transparent if an organisation is not open with individuals about how and why an AI-assisted decision about them was made, or where their personal data is being used to train and test an AI system. Privacy notices are often used to provide some of the necessary transparency, together with the general information that must be provided whenever personal data is processed (for example, around the purpose and duration of the processing).
Accountability includes demonstrating compliance with the GDPR principles. One way to demonstrate that you have treated an individual fairly and in a transparent manner when making an AI decision about them is to provide them with an explanation of the decision and to document this.
In addition, the GDPR requires organisations to carry out a data protection impact assessment (“DPIA”) when they are processing data using new technologies (like AI) which is likely to have a high risk to individuals (Article 35). DPIAs are also required where there is any systematic and extensive profiling or other automated processing of individuals’ personal aspects which are used for decisions which produce legal or similarly significant effects. The ICO considers that carrying out a DPIA may help organisations mitigate some of the challenges (mentioned above) around explaining AI.
Personal data is defined in the GDPR as any information relating to an identified or identifiable natural person. An identifiable person is one who can be identified directly or indirectly, in particular by reference to an identifier such as a name, ID number or online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that person.
Personal data is often processed when an AI model is being trained and operated. AI can also determine whether information falls within the definition of personal data, as the ability of AI to recognise patterns in data, or link data sets, can potentially enable data that would not normally be considered personal data to become ‘identifiable’.
The guidance is set out in three parts:
Part 1 covers the basics of explaining AI and is an introductory section aimed at all stakeholders within an organisation.
Part 2 looks at explaining AI in practice. It is aimed at technical teams but may also be of interest to compliance teams and data protection officers.
Part 3 examines what explaining AI means for organisations. This is aimed at senior executives in an organisation and outlines the different roles that should be involved in providing an explanation to the relevant individuals.
Part 1: Explanation types and principles
Part 1 of the guidance explains some of the basic terminology, GDPR provisions and risks associated with AI explainability. It also lists a set of AI principles which should be applied when explaining AI and a number of different ways in which AI decisions can be explained (explanation types).
The AI principles
The following four principles underpin how organisations should explain AI-assisted decisions to individuals and should be used together with the explanation types listed below them:
Be transparent – this is an extension of the transparency aspect of the lawfulness, fairness and transparency principle in the GDPR (see above). It is about making the use of AI for decision making obvious, and explaining the decisions you make to individuals in an appropriate way and at an appropriate time.
Be accountable – again, this is linked to the GDPR principle of the same name. GDPR accountability means: (i) taking responsibility for complying with the other data protection principles and demonstrating that compliance; and (ii) implementing appropriate technical and organisational measures and data protection by design and default. In an AI context, this means ensuring appropriate oversight of AI decision systems and being answerable for the decisions made (within the organisation, but also to regulators and relevant individuals). Organisations must, for example, identify those within their organisation who manage and oversee the “explainability” requirements of an AI decisions system and assign responsibility for this. It also means showing that you have considered how to design and deploy explainable AI (and can justify this), have provided explanations to individuals and have a “capable human point of contact” to manage queries.
Consider context – while this may seem obvious, the importance of context (use case, sector, individuals involved, etc.) was one of the key findings of the Project ExplAIn research, as set out in its interim report released in June 2019.4 It also aligns with the FCA’s work in this area (see above), which highlights the need to consider the different stakeholders and use-cases involved. In addition, considering context will help deliver the explanation in a way that the recipient will find most useful.
Reflect on impacts – many decisions made by AI will previously have been made by humans. The guidance describes AI as “increasingly serving as trustees of human decision-making” but notes that “individuals cannot hold these systems directly accountable for the consequences of their outcomes and behaviours”. The principle of reflecting on impacts helps organisations to explain to individuals that the AI will not harm their wellbeing, which involves considering questions about the ethical purposes and objectives of the AI project. This aligns with the focus at UK and EU level on the ethical deployment of AI, although arguably stretches beyond the ICO’s remit of data protection compliance.
As we have seen, context is a key aspect of explaining decisions involving AI. Several factors about the decision, the individual involved, type of data and the setting will all affect what information an individual would find useful or expect to receive as part of an explanation. The guidance therefore recognises that different types of explanation are required. It sets out six different types, which can be combined into an explanation in different ways, depending on the particular decision in question. They are:
Rationale explanation – the “why” of the decision which helps people understand the reasons that led to a decision or outcome. These should be delivered in a non-technical, accessible way. Part 2 of the guidance contains detailed information for technical teams on how to do this in practice.
Responsibility explanation – this focuses more on who is involved in the development, management and implementation of an AI system and who to contact for a human review of that decision.
Data explanation – what data has been used in a particular decision, and how. For example, what data was used to train and test the AI model and how it was used.
Fairness explanation – what steps have been taken (and will continue to be taken) in the design and implementation of the AI systems to ensure decisions are generally fair and unbiased. This also gives people an understanding of whether or not they have been treated equitably.
Safety and performance explanation – what steps have been taken across the design and implementation of an AI system to maximise the security, robustness, accuracy and reliability of its decisions and behaviours.
Impact explanation – what impact will the use of an AI system and its decisions have on an individual (and what broader societal effects may it have).
These are not intended to be an exhaustive list, but rather to identify what the ICO and the Turing consider to be the key types of explanations people will need.
Part 2: Explaining in practice
As well as providing explanation types and principles to follow, the guidance provides some practical assistance on how to apply these. Part 2 is aimed primarily at technical teams, although the content is accessible and useful (albeit a long read) for compliance and risk advisors. It goes into some detail on the technical factors that should be considered when selecting an AI model, the issues which arise with black box models and the types of AI models and tools/strategies for extracting explanations that are available. It also sets out seven steps organisations should take, which aim to provide a systematic approach to selecting, extracting and delivering explanations regarding AI decisions. These are:
Step 1: Prioritise – select priority explanations by considering the domain, use case and impact on the individual. This will often involve prioritising the rationale and responsibility explanations, although all relevant explanations should be made available to the relevant individuals.
Step 2: Collect the information needed for each explanation type. This will involve both the information needed for the process-based explanation (i.e. how the general decision making is structured) and the outcome-based explanation (i.e. what happened in the case of a specific decision).
Step 3: Build the rationale explanation to provide meaningful information about the underlying logic of the AI system. This requires an understanding of the AI. Organisations should ensure that they have selected an AI model/system with an appropriate level of interpretability, which requires consideration of the specific standards, conventions and requirements of the domain/sector in which the AI system will be applied. For example, in the financial services sector, rigorous justification standards for loan and credit decisions mean fully transparent and easily understandable AI decision-support systems are required. This section of the guidance also looks at the specific issues associated with “black box” AI systems and the use of hybrid methods in regulated sectors (see “Black box issues and hybrid methods” below).
Step 4: Translate the rationale of the AI system’s results into usable and easily understandable reasons. There must be a simple way to explain the model’s statistical results to an individual. Where a decision is fully automated, the use of software may be needed to do this.
Step 5: Prepare implementers to deploy the AI systems/models – when human decision makers are involved in an AI-assisted outcome, they must be trained to use the model’s results responsibly and fairly.
Step 6: Consider contextual factors when you deliver your explanation – organisations should draw up a list of relevant factors, which may include sector and audience or may involve factors specific to an organisation/the decision in hand.
Step 7: Consider how to present the explanation. For example, should the information be layered and should certain information be provided in advance of a decision being made?
Black box issues and hybrid methods
The black box effect of some AI models or systems has traditionally been seen as a barrier to explainability. The guidance defines a black box model as “an AI system whose inner workings and rationale are opaque or inaccessible to human understanding”.
It may not always be possible to avoid black box models. For example, the most effective machine learning approaches will likely be opaque (for example, when recognising speech) as the feature spaces of these types of AI systems grow exponentially. However, such models should only be used if the potential impacts and risks have been thoroughly considered in advance, and it has been determined that the use case and organisational capacities/resources support the responsible design and implementation of these systems. In addition, when considering the use of opaque algorithmic techniques, organisation must make sure that appropriate supplementary interpretability tools are used. These must provide a “domain-appropriate level of explainability that is reasonably sufficient to mitigate its potential risks and… a solid basis for providing affected decision recipients with meaningful information about the rationale of any outcome”.
Hybrid methods – use of challenger models
The Project ExplAIn research found that, while some organisations in highly regulated sectors like banking and insurance are using interpretable models in their customer-facing AI decision-support applications, they are starting to use more opaque ‘challenger’ models in parallel. Provided this is done in a transparent and responsible manner (and is documented), it can provide useful insights and comparisons. However, if the insights from the challenger model’s processing are incorporated into the actual decision making, then they must be treated as core and held to the same explainability standards as the main model.
Organisations should keep a record of any deliberations that go into their selection of a black box or challenger model as part of the process-based aspect of the rationale explanation (step 3).
Part 3: What explaining AI means for an organisation
The final section of the guidance focuses on what this all means in practice for an organisation. It is aimed primarily at senior executives and looks at the various roles, policies, procedures and documentation that can be put in place to ensure an organisation is prepared to provide meaningful explanations to its customers and other individuals. It is also of use to compliance teams and risk advisors as it lists, albeit at a high level, what should be covered in the organisation’s relevant policies and procedures, what documentation is legally required under the GDPR and what documentation can help the organisation demonstrate the explainability of its AI systems.
The first action point for organisations is to identify everyone involved in the decision-making pipeline and where they are responsible for providing an explanation of the AI system. In terms of the role of senior management, the guidance confirms that this is the team with overall responsibility for ensuring the AI system used by their organisation (whether developed in-house or procured) is appropriately explainable to the affected individuals.
Where AI is procured, organisations are still primarily responsible for ensuring the AI system is capable of producing explanations even where it is brought from a third party. Where off-the-shelf products are procured which do not contain inherent explainability, the organisation may need to use another model in parallel.
Although the FCA and ICO explainability projects are still being developed (or, in the case of Project ExplAIn, finalised), they do provide some useful guidance for organisations looking to increase the transparency of their AI use. They are, however, not the only sources of guidance in this area. In the UK, the ICO also discusses AI transparency in its AI auditing framework and general big data and AI guidance.5 The UK’s Centre for Data Ethics and Innovation (“CDEI”) and the European Commission are also considering this point. For example, strong themes of transparency and explainability came through in responses to the CDEI’s review into bias in algorithmic decision-making (which focused on financial services as one of four key sectors), and the Ethics Guidelines for Trustworthy AI published by the Commission’s High-Level Expert Group on AI include transparency as one of seven key requirements that AI systems should meet. They state: “AI systems and their decisions should be explained in a manner adapted to the stakeholder concerned. Humans need to be aware that they are interacting with an AI system, and must be informed of the system’s capabilities and limitations.”6
1. ICO Project ExplAIn guidance (consultation draft) – part 1.
2. See (Hyperlink).
3. See also recital 60 GDPR.