Enabling an Efficient Regulatory
Environment for AI - Practical
Considerations for Generative AI

JANUARY 2024

Disclaimer
The information and opinion commentary in this Enabling an Efficient Regulatory Environment for AI - Practical
Considerations for Generative AI (Paper) was prepared by the Asia Securities Industry and Financial Markets Association(ASIFMA) to reflect the views of our members. ASIFMA believes that the information in the Paper, which has been obtained
from multiple sources believed to be reliable, is reliable as of the date of publication. As estimates by individual sources may
differ from one another, estimates for similar types of data could vary within the Paper. In no event, however, does ASIFMA
make any representation as to the accuracy or completeness of such information. ASIFMA has no obligation to update, modify
or amend the information in this Paper or to otherwise notify readers if any information in the Paper becomes outdated or
inaccurate. ASIFMA will make every effort to include updated information as it becomes available and in subsequent Papers.

ASIFMA is an independent, regional trade association with over 160 member firms comprising a diverse range of leading
financial institutions from both the buy and sell side including banks, asset managers, law firms and market infrastructureservice providers.
Together, we harness the shared interests of the financial industry to promote the development of liquid, deep and broad
capital markets in Asia. ASIFMA advocates stable, innovative and competitive Asian capital markets that are necessary tosupport the region’s economic growth. We drive consensus, advocate solutions and effect change around key issues throughthe collective strength and clarity of one industry voice. Our many initiatives include consultations with regulators andexchanges, development of uniform industry standards, advocacy for enhanced markets through policy papers, and lowering
the cost of doing business in the region. Through the GFMA alliance with SIFMA in the US and AFME in Europe, ASIFMA
also provides insights on global best practices and standards to benefit the region.
www.asifma.org
If you have any comments or questions, please reach out to Laurence Van der Loo, Managing Director - Head of Technology& Operations (lvanderloo@asifma.org), Celina Leung, Senior Manager - Technology & Operations (cleung@asifma.org), or
Leonardo Troeman, Analyst - Technology & Operations (ltroeman@asifma.org).

1

Introduction

4

2

Executive Summary

6

3

What is Generative AI

8

4

Leveraging the Benefits of Generative AI in Capital Markets

105

Review and Application of ‘Principles for AI Regulation’ From June 2021 Paper

126

Stakeholders and Deployment Models

14

7

Assessment of New and Incremental Challenges, Existing Controls and
Gap Analysis

16

A

Data-related Considerations

17

B

Model-related Considerations

21

C

Third-Party Management and Resilience26

D

Cybersecurity

27

E

Expertise

28

F

Environmental Implicatons

29

8

ASIFMA’s Proposed Recommendations for a Future-Proof, Risk-based
Regulatory Framework for Generative AI

30

9Glossary of Acronyms

36

1 | Table of Contents3

1

INTRODUCTIONASIFMA’s paper on “Enabling an Efficient
Regulatory Environment for AI”1 (“ASIFMA
2021 AI Paper”) was launched in 2021 as a
guiding set of regulatory principles for Artificial
Intelligence (AI) which would help form thebasis for building an efficient regulatory environment
whilst at the same time supporting customer and investor
protection, market integrity and financial and systemicstability. These principles recommended that regulators
follow technology-agnostic, risk-based, and principle-based
approaches to regulating AI in order to help increase crossborder regulatory standardisation and harmonisation.
Since the launch of ChatGPT, excitement as well as anxiety
have been heating up globally around the opportunities andchallenges of generative AI. Governments and regulators
across the world are starting to consider regulation andare at various stages of the drafting process regarding
generative AI regulation.
This 2023 addendum to the 2021 paper focuses on
generative AI and was developed with the support ofASIFMA’s Generative AI Taskforce and the Fintech Working
Group which consist of banks, brokers, asset managers,
professional firms, market infrastructure providers andtechnology companies. The aim of this paper is to initiate
further proactive engagement with regulators on generative
AI, to help advance the public-private dialogue and to pushforward greater collaboration so as to ensure that generative
AI is used in a secure and responsible manner in the capital
markets industry. We hope this paper will help to guidestakeholders in staying ahead of the curve and preparing
for future regulatory developments. This paper will also
serve as a useful tool in helping to educate and inform themarket and regulators on the various opportunities, gaps,

and challenges prevalent in the use of generative AI incapital markets and outline the industry’s views on how tobest seize those opportunities and address those gaps andchallenges.
We wish to thank all members of the Fintech Working Groupand Generative AI Taskforce for their valuable input andcontributions to the paper.
This Paper does not provide and should not be treated as legal
or professional advice on regulatory compliance, licensing
requirements or any other matter. It is up to readers to obtain
their own legal, tax and other professional advice. This
Paper does not endorse, approve, recommend, or make any
representation or warranty (express or implied) about, any
particular product, transaction, service, exchange, entity,
platform or service provider and other intermediary and weassume no responsibility or liability for any consequencethat may arise from your use or reliance of any of them(including the information and views they provide and the
products and services they offer).

ASIFMA, ‘Enabling an Efficient Regulatory Environment for AI’, 2021, https://www.asifma.org/wp-content/uploads/2021/06/enabling-anefficient-regulatory-environment-for-ai-report_june-2021.pdf
1

1 | INTRODUCTION

52EXECUTIVE
SUMMARY

W

hilst the upside and benefits of the useof generative AI in capital markets areundeniable, generative AI also introduces
new challenges. In this paper, we willassess the new and incremental challenges
of generative AI, explore how these are already addressedby existing regulations, tools and governance frameworks,
explore mitigants, identify any gaps and make suggestions
on how to address the gaps with the aim to ensure thesafe and responsible adoption of generative AI in thecapital markets industry so as to realise the full benefits ofgenerative AI.
As the world embarks on the journey of formulating
governance frameworks for generative AI, with somefurther ahead than others, it is important to recognisethat everyone needs to come together to have a common
consensus on making it a safe environment and yet harness
the advantages that generative AI can bring.

The paper continues to reiterate the importance of the 7principles as mentioned in our 2021 AI Paper that regulators
should consider. We have also put forward an additional
set of principles which we believe will allow for a balanced
regulatory approach for generative AI. Specifically, we
recommend that regulators should:

ASIFMA PRINCIPLES FOR GENERATIVE AI- Principle 1:
Leverage existing regulations- Principle 2:
Maintain governance and accountability- Principle 3:
Provide the appropriate level of transparency- Principle 4:Continue to adopt a risk-based approach-Principle 5:Continue to adopt a technology-agnostic regulatory
approach-Principle 6:
Address any IP Protection challenges-Principle 7:
Strive for regulatory certainty and a harmonised framework
We look forward to engaging with regulators and otherstakeholders on our suggested principles and key
findings to support an enabling regulatory frameworkfor generative AI in APAC and beyond.

2 | EXECUTIVE SUMMARY

7

3WHAT ISGENERATIVE AI

G

enerative AI is a type of AI that can generate various
types of content, including text, images, audio,
programming code and video, and more, based on
the data it has been trained on and the algorithms
used. While there is no formal definition of “generative AI,” itis generally understood to be a subset of traditional machinelearning (ML) that generates new and useful outputs from
simple inputs. In layman’s terms, generative AI allows auser to provide a ‘prompt’ and receive a newly generated
output in the desired medium. This ‘generated output’ is a
key differentiating factor between generative AI and priormodels. Generative AI has progressed ML beyond models
that analyse a set of data and find resulting patterns, tomachines capable of creating entirely new content.
ChatGPT, one of the most well-known generative AI tools,
defines generative AI as “a category of AI techniques andmodels that are designed to generate new content, data, or
information. These models are trained on large datasets andlearn to produce outputs that are similar to the examples
they were trained on. They work by learning the underlying
patterns and structures present in the training data and thenuse generalised patterns to generate new, and potentiallypreviously unseen content.”
Generative AI is powered by ML models — but specifically for
generative AI, very large models that are pre-trained on vast
amounts of data and commonly referred to as Foundation
Models (FM). FM is a conceptual term that is used todescribe incredibly powerful, general-purpose models thatcan be customised for specific use cases without having

3 | WHAT IS GENERATIVE AI

to build a model from scratch each time. Generative AI
tools are usually based on FMs like Generative pre-trainedtransformer (GPT) - a large language model (LLM) createdby OpenAI, which are trained on a broad set of data that is
adaptable to a wide range of tasks and can be fine-tuned forspecific tasks.
A class of FMs, such as the GPT models, commonly referred
to as LLMs are specifically focused on language-based tasks
such as such as summarisation, text generation, synthetic
text generation, classification, open-ended questions andanswers, and information extraction.
What makes LLMs special is that they can perform so many
more tasks because they are capable of learning advanced
concepts, potentially resulting in greater scale of adoption
across Financial Institutions (FI), either as standalone AI
systems or embedded as an AI component in previously
non-AI related software. And through their pre-training
exposure to internet-scale data in all its various forms andmyriad of patterns, LLMs learn to apply their knowledge toa wide range of contexts.
While the capabilities and resulting possibilities of pretrained FMs are impressive, their adaptability throughcustomisation to perform domain-specific functions makes
it even more exciting to businesses. As a result, businesses
can build highly efficient and productive applications with
FMs using only a small fraction of the substantial data andcompute required to train a FM from scratch.

9

4

LEVERAGING THE
BENEFITS OF
GENERATIVE AI IN
CAPITAL MARKETSG

enerative AI capabilities have advanced rapidly
in recent years as a result of new breakthroughs
in deep learning architectures (e.g., “transformer”
models for text and “diffusion” models for images).
These developments have made this technology apowerful tool for businesses to innovate, enhance customer
experiences, boost productivity and creativity, and optimisebusiness processes. Examples of specific use cases include:-

Writing assistance: Generative AI can be a useful tool tohelp prepare initial drafts of text (e.g., emails or speeches)or to review existing texts to make them more polishedor professional.

-

Summarisation: Generative LLMs can be used tosummarise and extract key information from longarticles, documents, or webpages. Summarisation canbe helpful for people who want to quickly understand themain idea of a text or for researchers who need to readmultiple papers and want to understand the key findings.

-

Software development: Generate code snippets,
comments, and documentation based on natural
language inputs. This can improve the efficiency andaccuracy of software development tasks.

-

Conversational AI: Create natural-language-based
conversational interfaces such as chatbots and virtualassistants. Conversational AI can include speech-to-text
and translation capabilities.

-

Brainstorming: Generative AI can serve as abrainstorming tool that enables a user to input. By giving
generative models a specific prompt or theme, they canquickly generate unique and interesting pieces of content.

Generative AI has the potential to create significant value forFIs, driving productivity, innovation, and economic growth.
Generative AI capabilities can transform processes such as
portfolio management, financial documentation, intelligent
advisory, fraud detection, compliance assistance, and soon. As a result, FIs can improve their internal and externalcustomer experience, increase the efficiency of knowledge
workers, automate processes, and innovate new products.
For example, an investment management firm can build
an investment analyst assistant using generative AI tools.
Rather than employees spending significant time writingcode or using software to create charts and tables, generative
AI can produce charts and tables based on natural language
query posed by the analysts. To understand challenges andtrends at a global level, investment analysts need to siftthrough massive amounts of data from disparate sources.
Generative AI can also power investment research, providing
fast and effective Q&A capabilities with capital markets
data. Capital markets organisations can also transformclient engagement through the use of chatbots and virtualassistants, and they can rely on generative AI tools toease and automate their regulatory compliance journey.
Through such use cases, capital markets firms can develop
unprecedented capabilities to translate data into insights
and impact, develop new revenue streams, and provide
better products and experiences to their customers.
Adoption of generative AI in the financial industry has beenvaried at this time. FIs are prudently evaluating safe andeffective ways of leveraging generative AI, including bybuilding enterprise-grade applications of FMs in controlledenvironments.

4 | LEVERAGING THE BENEFITS OF GENERATIVE AI IN CAPITAL MARKETS

11

5

REVIEW ANDAPPLICATION OF‘PRINCIPLES FORAI REGULATION’
FROM JUNE 2021
PAPER ‘ENABLING
AN EFFICIENTREGULATORYENVIRONMENT FOR
AI’In our 2021 AI paper, we proposed a set of regulatory principles for AI which we believe will form the basis for an efficientregulatory environment whilst at the same time supporting customer and investor protection, market integrity and financial
and systemic stability:

ASIFMA 2021 PRINCIPLES FOR AI REGULATION- Principle 1:
Support public-private collaboration- Principle 2:
Allow FIs to take a risk-based approach to AI, taking materiality of the use case and stakeholders into account- Principle 3:
Take a technology-agnostic approach to regulation- Principle 4:
Leverage existing regulatory frameworks- Principle 5:
Strive for regional and international harmonisation- Principle 6:
Promote and facilitate cross-border data flow
- Principle 7:
Engage with the industry on areas that need further discussion

We continue to support these principles and they are even more important as we consider the future regulatory approaches
to generative AI. At the same time, as we will outline in section 7 below, our challenges and controls assessment showsthat there are some incremental or new aspects in relation to generative AI. In section 8, we put forward some additional
considerations and suggestions pertaining to generative AI for regulators and industry to consider.

5 | REVIEW AND APPLICATION OF ‘PRINCIPLES FOR AI REGULATION’ FROM JUNE 2021 PAPER‘ENABLING AN EFFICIENT REGULATORY ENVIRONMENT FOR AI’

13

6

STAKEHOLDERS
AND DEPLOYMENTMODELS

T

here are two common roles in an AI system’svalue chain that FIs might play, “provider”
and “deployer/user”. Here are their example
definitions:2

Provider: A natural or legal person, publicauthority, agency, or other body that develops an AI systemor that has an AI system developed with a view to placing iton the market or putting it into service under its own nameor trademark, whether for a fee or free of charge.
Deployer/user: Any natural or legal person, public authority,
agency, or other body using an AI system under its authority
except where the AI system is used in the course of apersonal non-professional activity.
Deployment options for generative AI applications impact
the scope of FIs’ responsibilities and approaches for risk
management over the AI system lifecycle. They areclassified into two categories – buyers of generative AI
applications and builders of generative AI applications.
Buyers of generative AI applications
When FIs purchase generative AI applications, they act as a
deployer/user. One option is that they purchase the consumer
application, a public third-party generative AI service, either
at no-cost or paid. Within this scope, FIs do not own or see
the training data or the model, and FIs cannot modify oraugment it. FIs, however, can implement an application
programming interface (API) or directly use the applicationaccording to the terms of service of the provider. (e.g., an FI
employee interacts with a generative AI chat application to
generate ideas for an upcoming marketing campaign). The
other option is that FIs purchase an enterprise application,a third-party enterprise application that has generative
AI features embedded within, and a business relationshipis established between FIs and the provider. (e.g., FIs use
a third-party enterprise scheduling application that has
2

a generative AI capability embedded within to help draft
meeting agendas).
Builders of generative AI applications
When FIs build generative AI applications themselves, theytake on the role of “provider” and “deployer/user”. Firstly, FIs
can build their own applications using an existing thirdparty pre-trained generative AI FM and directly integrate
it with their workload through an API (e.g., an FI builds anapplication to create a customer support chatbot that uses
the Anthropic Claude FM through Amazon Bedrock APIs).
Secondly, FIs could build with applications using fine-tuned
models. An FI refines an existing third-party generative AI
FM by fine-tuning it with data specific to their business,
generating a new and, enhanced model that is specialised
to their workload (e.g., using an API to access a FM, an FI
builds an application for their marketing teams that enables
them to build marketing materials that are specific to theirproducts and services). Thirdly, FIs could build with selftrained models. An FI builds and trains a generative AI
model from scratch using data that they own or acquire inwhich they own every aspect of the model (e.g., an FI createsa model trained exclusively on deep, industry-specific data tolicense to companies in that industry, creating a completely
novel LLM).
“Providers” and “deployer/users” share the responsibility forrisk management associated with the use of generative AI.
This establishes accountability and ensures responsibilitiesof the generative AI service provider and its consumers
with regard to responsible use of generative AI. Whenunderstanding the risks associated with the use of generative
AI, it is critical to note that the allocation of risk responsibilities
is contingent upon the particular AI application scenario that
could originate from the generative AI service provider or theuser of the generative AI service and such allocation should
be contractually agreed between the involved parties, where
it is not specified by the regulators.

European Parliament, EU Artificial Intelligence Act, June 2023, https://www.europarl.europa.eu/doceo/document/TA-9-2023-0236_EN.html

6 | STAKEHOLDERS AND DEPLOYMENT MODELS

15

7ASSESSMENT
OF NEW AND
INCREMENTAL
CHALLENGES,EXISTING
CONTROLS AND GAPANALYSIS

I

n what follows, we evaluate whether there are
any new or incremental challenges pertaining
to generative AI. We note that whilst somechallenges as highlighted in our 2021 AI paper
are not new or incremental to generative AI,
regulators should be aware of the amplified impact as the useof generative AI accelerates. Regulators should encourage
the adoption of this new technology while encouraging
their regulated entities to develop appropriate riskbased frameworks for their generative AI adoption. Suchframeworks need to recognise that FIs could play different
roles in the value chain of generative AI applications, andguide them to identify relevant challenges,and implement
mitigants in relevant stages of the AI system lifecycle3, given
the chosen deployment option. For example, when assessing
a challenge, FIs need to determine whether they should
directly implement mitigants and/or via the AI model orservice providers’ compliance that is governed under FI’s
third party risk management framework along the lifecycle.
FIs should clearly establish the context, their roles, andthe deployment option to determine the appropriate risk
management approach.

A. Data-Related Considerations
A.1 Data Quality/Integrity
Incremental challenges of generative AI

example data poisoning. These challenges also apply togenerative AI, likely to a greater extent.
- Training data disclosure
For open-source/freely available LLMs, training data israrely disclosed, which makes it more challenging toevaluate data quality, privacy, and security in such cases.
- Data quality
Recent research has shown that data quality (as opposed toquantity) is a far more important factor in LLMs than one
might have expected. Ongoing research found that finetuning a strong pretrained language model with a small
set of carefully curated examples can produce remarkable
results on a wide range of prompts compared to otherstate-of-the-art LLMs that are fine-tuned with large-scale
instruction tuning and reinforcement learning.4- Model collapse
A new challenge of generative AI is called model collapse.
Model collapse is defined as a degenerative process
affecting generations of learned generative models, wheregenerated data end up polluting the training set of thenext generation of models. As the subsequent models arethen trained on polluted data, it would result in them misperceiving reality.5 In short, if LLMs are trained on data,i.e., synthetic data that is generated by other LLMs and notproperly labelled, over time, these LLMs would drift awayfrom accuracy and precision and possibly into falsehoods
that will inadvertently converge to a point that is far fromreality.

In our 2021 paper, we covered a range of data quality issues
that may affect the accuracy of AI models, including forOECD, ‘Recommendation of the Council on Artificial Intelligence’, November 2023, https://legalinstruments.oecd.org/en/instruments/OECDLEGAL-0449 OECD defines the AI system lifecycle as ‘design, data and models’, ‘verification and validation’, ‘deployment’, and ‘operation andmonitoring
4
Chunting Zhou et al., ‘LIMA: Less Is More for Alignment’, May 2023, https://arxiv.org/pdf/2305.11206.pdf
5Ilia Shumailov et al., ‘The Curse of Recursion: Training on Generated Data Makes Models Forget’, May 2023, https://arxiv.org/abs/2305.17493
3

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

17

Mitigants
Below are some of the suggested mitigants of Data Quality/
Integrity challenges:- Allocation of resources
Regarding the heightened importance of data quality, bothdevelopers and deployers of LLMs may need to invest
more time, effort, and resources to ensure data quality.
For example, in the Meta paper, the researchers manuallycurated 1,000 high-quality prompts and responses as
training examples.4 Such efforts are highly laborious, but
likely necessary to uphold the quality of the data, and hence
the AI models.
- Transparency
Developers will need to provide transparency anddocumentation on the source of the data which has beenused to train a model as well as the reasons for selection of
the data in order to meet the deployers’ internal oversightand governance obligations.
- Quality standards for external data might be explored
To build trust in data coming from external sources, itwould be positive for the industry and regulators to promote
initiatives that certify the quality of external data (e.g.,the origin and integrity of their data is well documented;
continuity plans to ensure the provision of the same datain the future, etc). In this respect, Legal Entity Identifier data
is an example of an open-source, free of charge data that isused as a public good.
- Watermarking
Regarding the challenges of model collapse, since it may
be inevitable that the Internet would be inundated with
AI-generated content, developers of LLMs may need to4ensure that they are able to train the models as intended,for example using only genuine human-generated
data as required. To do so, they will need solutions
to accurately distinguish between AI-generated andhuman-generated data, also known as “watermarking”.
Traditional watermarks (e.g., stamp imprinted on animage) are unsatisfactory solutions as they present
aesthetic challenges, especially for creative purposes,
and can be easily edited out with technology. DeepMind
recently released a beta version of SynthID, a tool forwatermarking and identifying AI-generated images byembedding a digital watermark directly into the pixels
of an image, making it imperceptible to the human eye,but detectable for identification. Similar technology could
potentially evolve and be applied to text, audio, and video,
subject to technical feasibility.

A.2 Data Privacy/Protection/
Security
Incremental challenges of generative AI
While the fundamental challenges associated withgenerative AI are the same as those highlighted in our 2021
AI paper, there are certain areas that require particularattention and consideration with respect to generative AI:- Data rectification/deletion
In many jurisdictions, individuals are afforded legal
rights in respect of accuracy, rectification, and deletionof their personal data (e.g. ‘the right to be forgotten’).
These obligations could be technically challenging fororganisations to comply with in the context of generative
AI as it is challenging for generative AI models to ‘detrain’or ‘forget’ the data it was trained upon.

Chunting Zhou et al., ‘LIMA: Less Is More for Alignment’, May 2023, https://arxiv.org/pdf/2305.11206.pdf

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

18

- Output data
A generative AI model may falsely generate incorrect or
flawed personal data, which is an example of hallucination,
but in the context of personal data. (See Section ‘7B.1
Hallucination’ below)- Disclosure of personal data
If generative AI models “memorise” specific data records,
there is a possibility that the models will then exactly
replicate those records (as opposed to “generalising” fromthe broad data set) when responding to a user query, which
could lead to the inadvertent disclosure of personal data,
as well as the revelation of sensitive and/or confidential
information.6- Basis for data processing
One of the challenges, for example, is where the use of anindividual’s personal data for the purposes of training or
inputting into a generative AI system is beyond the limitsof the original collection consent obtained from that
individual.
- Balancing fairness and data minimisation
Privacy laws often limit the collection of personal data towhat is strictly necessary (data minimisation). However,there is trade-off and balance to be struck between data
minimisation and fairness, as generative AI developers
and deployers may need data, including sensitive personal
data such as race and ethnicity, to conduct fairness
assessments.

Mitigants
Below are some of the suggested mitigants of Data Privacy/
Protection/Security challenges7:- Consent
Where practical and necessary, FIs should take into
account which jurisdictions’ privacy laws apply to therelevant data flow, obtain clear and unambiguous consent
from users, or otherwise ensure that there is a clear, valid,
legal basis for the collection of data. FIs should also provide
clear and easily understandable information about how
their data will be used and the purpose of the generative AI
applications. This would apply to Intellectual Property (IP)Protection as well. (See Section ‘7A.3 Intellectual Property
Protection’).
- Security measures
Implement robust security measures to protect user datafrom unauthorised access, loss, or breaches and ensurethat regular assessments of security protocols take place.
Where a third party is hosting the generative AI system andprocessing personal data, ensure technical due diligenceis performed to understand the third party’s security
measures, and that contractual protections are in place inthe event of a security breach.
- Frozen foundational models
Most providers of enterprise applications of generative
AI offer FIs a “frozen foundation model”, i.e. a FM that is
not further modified. FIs can use their own confidential
data, including prompts, as input to this frozen FM andeven fine-tune the model using an “adapter layer”. The
data and adapter layer are unique to the FI, secured within
the FI’s own cloud project, and only accessible by the FI.
Most importantly, when the FI makes an inference, theFM receives the adapter layer, runs through the request,
and returns the results, without modifying the FM or
storing the request, which is why the FM is “frozen”. Suchenterprise-grade security measures help to ensure that FIs’
data can remain confidential when using generative AI.

IMDA, ’Generative AI: Implications For Trust And Governance’, Jun 2023, https://aiverifyfoundation.sg/downloads/Discussion_Paper.pdf
Justin Antonipillai, ‘The Intersection of Generative AI, Data Privacy, and GDPR: Unlocking Marketing Opportunities Responsibly’, July 2023,
https://wirewheel.io/blog/the-intersection-of-generative-ai-data-privacy-and-gdpr-unlocking-marketing-opportunities-responsibly/
6
7

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

19

A.3 Intellectual Property
Protection
Incremental challenges of generative AI
Below are some of the incremental challenges of IP
Protection:- Training data challenges
Generative AI models train on large amounts of data.
The models, particularly FMs, can scrape the web andother sources for data which can be copyrighted andtherefore, the output of generative AI models could breach
copyrights.8 It is unclear whether fair use exceptions would
apply. However, if the regulators prohibit a FM from being
deployed in a jurisdiction, it could put critical businessservices (including customer facing ones) at risk.
- IP leakage and trade secret challenges
Some generative AI models may memorise and output
significant parts of the data that they are trained on, as
well as the data provided in the prompts. This means that
subsequent outputs, generated by the model for other users,
contain or are derived from those input data and prompts.
One recent example of this is when Samsung employees
unintentionally leaked copyrighted source code when they
were leveraging ChatGPT to optimise the code.9- Software licensing challenges
Software code written by generative AI may incorporate
third-party code used to train the model, including opensource code. In this event, the generated code will haveto comply with any licences applicable to the originalcode. In particular, certain licences may require new (andproprietary) software to be made open-source.

generated by generative AI. However, a United States(US) federal district court had ruled that AI-generated
artwork is not eligible for copyright protection under the
US law, explaining that human authorship is a “bedrock
requirement of copyright”.10 At the same time, jurisdictions
such as Japan apply copyright protection to works that are
produced using a computer as a tool if there is a human’s
creative intention and creative contribution.

Mitigants
Below are some suggested mitigants of IP Protection
challenges:- Data source
Organisations need to understand the source of data andhave a mechanism to distinguish the originality of content
produced.
- Transparency
Controls and transparency requirements will have tobe embedded throughout the generative AI lifecycle, bydevelopers and users alike. However, there also may be
an imbalance of power between large market vendors
and FIs, resulting in an inability of FIs to obtain adequatetransparency.
- Contractual requirements
FIs should embed protections into the contracts withthird-party AI developers to obtain appropriate contractual
protections.
- Consent
See Section ‘7A.2 Data Privacy/Protection/Security’. above.
- Frozen foundational models
See Section ‘7A.2 Data Privacy/Protection/Security’ above.

- IP ownership challenges
There is uncertainty surrounding IP rights of outputs

Jonathan Stempel, ‘NY Times sues Open AI, Microsoft for infringing copyright works’, December 2023, https://www.reuters.com/legal/transactional/ny-times-sues-openai-microsoft-infringing-copyrighted-work-2023-12-27/
9
Siladitya Ray, ‘Samsung Bans ChatGPT Among Employees After Sensitive Code Leak’, May 2023, https://www.forbes.com/sites/
siladityaray/2023/05/02/samsung-bans-chatgpt-and-other-chatbots-for-employees-after-sensitive-code-leak/?sh=3d00ab9a6078
10
Josh Gold-Quiros et al., ‘Court Finds AI-Generated Work Not Copyrightable for Failure to Meet “Human Authorship’ Requirement—But
Questions Remain’, August 2023, https://www.jdsupra.com/legalnews/court-finds-ai-generated-work-not-2083236/
8

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

20

B. Model-Related Considerations
B.1 Hallucination
Incremental challenges of generative AI
AI hallucination is a phenomenon wherein generative AI
generates results with falsehoods, a lack of informativenessand a lack of relevance to the context.

a. Leveraging thecharacteristics
of
Text-to-text generative AI models are systems that can
produce natural language
based
on some input
blockchain
astexts
thebasisfortexts(or prompts). These models are evaluated on how well they
can avoidfinancial
hallucination – thephenomenon of generating
newservices

texts that are not grounded in reality, factual, or relevantto the input – as a key criterion. To measure the degree of
hallucination, three criteria can be used:11- Truthfulness
The generated text does not contain any false or misleading
information. This is core to the concept of hallucination.
- InformativenessThe generated text provides useful and pertinent
information that answers the user’s query or satisfies
the user’s goal. This is the completeness and relevanceof information provided and there can be other reasonsleading to a lack of informativeness, for example,
erroneous, inconsistent and/or outdated data. By itself, it is
not a necessary indicator of hallucination. However, a lackof truthfulness, together with a lack of informativeness,become sufficient conditions for hallucination.
- CompletenessThe generated text provides partial, yet correct,
information. This is a common issue due to FMs’ inability
to retain context beyond a certain size (also known as

11context window). While it is noted that this is not exactly
hallucination, it can be an issue when the outcomes ofgenerative AI applications are expected to be complete andused as a basis for decision making.
In the instances when hallucination or a lack of truthfulness
happens, it is not a necessarily completely black-box
phenomenon but rather a combined result of various factorsthat have affected the model’s behaviour. At the time ofwriting, there is no clear explanation to the cause of thehallucination or established techniques that can preventthe hallucination from happening with certainty. Some ofthese factors depend on the choices and actions of humans
and organisations, which may either reduce or amplify thechallenges of hallucination. Below are some common factorsthat can influence the challenges of hallucination.
- Data quality
One of the factors that exacerbates hallucination isineffective or lack of quality training data – which meansthat the model is not trained on enough or relevant data
to produce more accurate outputs for the domain that it isintended for use. For example, there could have been limited
quality data, irrelevant data, intentional negative biases inthe training data, and “noisy” – missing, contradictoryor ambiguous – input data used in its training, andany of these would lead to incorrect results that can beconsidered as hallucinated results. Such inaccuracies canbe compounded by historical data that the FM has been
built on, and itself could contain errors and inconsistencies.
Depending again on the context of application, if the model
is not effectively combined with updated live data for it
to draw from to form its responses to user queries, thechallenges of hallucination would be higher. Moreover,the deployment domain of the model should match its
training domain, as a model trained on wholesale financial
industry data and terms may perform poorly and exhibit
higher personal bias when applied to retail banking. (See
Section ‘7A.1 Data Quality’ above)

Stephanie Lin et al., ‘TruthfulQA: Measuring How Models Mimic Human Falsehoods’, September 2021, 2109.07958.pdf (arxiv.org)

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

21

- Model training
A related factor is ineffective model training, for example,
the effects of inadequate “overfitting” where a model
learns from the training data too well that it would failto generalise to new data – meaning that its results are
consistent with the training data but not with reality.
Another factor that can influence the level of hallucinationin text-to-text generative AI models is the phenomenon of“forgetfulness,” which refers to the loss of some abilities orknowledge by the model due to its data manipulation skills
being distorted (as a consequence of being “overfitted”)
during the training process. This means that the model
becomes too specialised or adapted to a specific task ordataset and fails to generalise well to new or different
inputs or scenarios.
- Development process and feedback
The development process can also contribute to a model’stendency to hallucinate. As a significant departure fromtraditional technology software development lifecycle(SDLC), where domain users are primarily involved in
specific requirements definition and test stages, generative
AI development process requires such domain expertiseto be involved – and stay involved – right from the startand in every stage of its development. Domain experts
are required to define expected training results, evaluatethem, and provide feedback; acting together and in parallelwith technical model adjustments by data engineers andscientists.

Mitigants
Below are some suggested mitigants of Hallucination
challenges:- Upskilling users
An important first step is to educate users about how
generative AI actually works, so there is no expectationthat the citations or news-like stories produced are always
genuine or factually correct. Users should also be educatedon the particular generative AI system’s capabilities andlimitations to prevent any misuse.
- Verification
The specific case of hallucinated citations could be
mitigated by augmenting LLMs with independent, verified
citation databases and similar sources, using approaches
such as retrieval-augmented generation.12 Other generative
AI applications are linking to search engines. For instance,
Google’s Bard chatbot recently updated its “Google it”
feature, which allows users to easily double-check Bard’s
responses by reading the response and evaluating whether
there is content across the web to substantiate it. Whena statement can be evaluated, the user can click thehighlighted phrases and learn more about supporting orcontradicting information found by Google Search.
- Data attribution
Another nascent but intriguing approach is to develop
methods for attributing generated outputs to particularpieces of training data, allowing users to assess the validityof those sources. This could help with explainability as
well. (See Section ‘7B.3 Transparency, Explainability,
Traceability’ below)- Implementing boundaries
Some users choose to intentionally limit the scope of a FM
to a specific domain by implementing boundaries within
their models to prevent models from hallucination.
- Adopt an AI-centric SDLC
Especially for LLMs, allow for continuous user feedback
and model tuning including after the initial launch.
- Data quality and training model
Ensuring the trained model is deployed into the context it
is meant for; and ground it with updated high-quality datathat is relevant to the use case as well as regular model
review and maintenance to reduce overfitting.

Amog Kamsetty, ‘Retrieval Augmented Generation with Huggingface Transformers and Ray’, February 2021, https://huggingface.co/blog/
ray-rag
12

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

22

- Expertise
Ensuring relevant expertise in both data science anddomain fields are available for model training and feedback.
- Model evaluation
A measurement of an LLM model can provide valuable
insights into the reliability of its responses, facilitatingrisk assessment and error, and reducing hallucinationsin natural language generation tasks.13 Industry standard
measurements of LLM confidence will need to be further
developed. (See Section ‘7B.2 Foundational Model Selection’
below)

B.2 Foundational Model
Selection
Incremental challenges of generative AI
When using generative AI, selecting a FM is one of the firstand most critical steps. Recall that a FM is a large ML model
pre-trained on a vast quantity of data at scale resulting in amodel that can be adapted to a wide range of downstreamtasks. FM selection has strategic implications for how a use
case gets built. Models vary across several factors, including
level of customisation, model size, inference options14,
licensing agreements, context windows15, and latency16.
- Open-source vs proprietary
There are many FMs in the market, ranging fromproprietary to open-source models, each having their own
trade-offs. The proliferation of open-source FMs is giving
rise to thriving ecosystems, such as Hugging Face. For
Open-source FMs – such as OpenAI - the API is released,
but not the model or training data, are also emerging.
- Hosting
Another key consideration in model selection is how themodel can be served. Open-source models, as well as
self-managed proprietary models, grant the flexibility tocustomise how and where the models are hosted. Directly
controlling a model’s infrastructure can help companies

ensure reliability of their applications with best practices
like autoscaling and redundancy. Managing the hosting
infrastructure also helps to ensure that all data generated
and consumed by a model is contained to dedicatedenvironments which can adhere to security requirements
set by the company.
- LifespanFIs should also consider the lifespan, including support,of the FM.
Given that FMs are the foundational architecture on which
many generative AI applications are built, selecting the right
FM for a particular use case is key.

Mitigants
Below are some suggested mitigants of Foundation Model
Selection challenges:- Existing regulations
The existence of well-established regulations in the areasof outsourcing, third-party risk management, technology
risk management, cybersecurity and operational resilienceshould be considered and adhered to in the selection of a
FM and its underlying provider.
- Use benchmarks to assist model selection
Generalised benchmarks (such as Stanford’s Holistic
Evaluation of Language Models)17 are a great starting
point because they help prioritise which FMs to startexperimenting with. Custom benchmarks are useful for
scenarios where generalised benchmarks are insufficient,such as use cases that are focused on building for a specificcustomer base. Custom benchmarking may include
techniques such as calculating BiLingual Evaluation
Understudy and Recall-Oriented Understudy for Gisting
Evaluation scores18 . These are two metrics that help
quantify the number of corrections that are necessary to
AI-generated text before giving it final approval for humanin-the-loop applications.

Yijun Xiao and Wiliam Yang Wang, ‘On Hallucination and Predictive Uncertainty in Conditional Language Generation’, March 2021, https://
arxiv.org/pdf/2103.15025.pdf
14
Inference options - Different methods of generating output including speculation, next word predictions and stopping criteria.
15
Context windows - The textual range around a token that can be processed at generation.
16
Latency - The overall time it takes to generate a full response.
17
Stanford University, ‘A holistic framework for evaluating foundation models’, https://crfm.stanford.edu/helm/
18
Sthanikam Santhosh, ‘Understanding BLEU and ROUGE score for NLP evaluation’, April 2023, https://medium.com/@sthanikamsanthosh1994/understanding-bleu-and-rouge-score-for-nlp-evaluation-1ab334ecadcb
13

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

23

- Consider narrow models for specific tasks
Companies can also experiment with using narrowmodels meant for specific tasks, like following instructions
or summarisation. These purpose-built models cansignificantly reduce a model’s parameter count while
maintaining its ability to perform domain-specific tasks.
- Model customisation
Open-source FMs empower companies to further
customise and fine-tune their systems with their own
datasets. For example, Parameter-Efficient Fine-Tuning
solutions from Hugging Face have shown how adjusting asmall number of model parameters, while freezing mostother parameters of the pre-trained LLMs, can greatly
decrease the computational and storage costs.19- Vendor
The importance of third-party risk management remains
critical, especially as a firm considers the challenges
that the FM may not perform as intended, particularly
in edge scenarios. Typical third-party risk management
assessments and governance will help to manage
these model challenges. One such example is outlining
responsibilities associated with the model provider/developer embedded in a contract.

B.3 Transparency,
Explainability, and Traceability
Incremental challenges of generative AI
Generative AI systems often leverage LLMs, which are morechallenging to explain or provide transparency on how themodel works, as compared to for example a regression modelor a decision-tree. This potentially leads to operational issues(such as not being able to explain how a business decision
was made) and regulatory compliance issues (such as not
being transparent regarding the usage or not being able toexplain how decisions affecting individuals were made, as
required under data protection laws).
It is also important to adapt the explanation to the audience,
which – in the case of generative AI – can be broad andtherefore adaptation to existing explanation techniques
might be required.

Mitigants
Below are some suggested mitigants of Transparency,
Explainability, and Traceability challenges:- References
Providing further information on the training data set(s)that have been used to train the model, with a particularattention to additional information used to supplement
pre-trained models.
- Citations
Whilst further research and investment will be required,
there are benefits to augmented LLMs that are able to cite
their data sources, particularly when they are referencing
a smaller company specific sub-set of documents, e.g.
policies.
- Relevance
Consideration should be given to whether generative AI
and the chosen model are appropriate after taking into
account the materiality/critical nature of a given use case.
(e.g., analysing log files of a system and summarising theissues vs. providing investment advice to a customer).
More importantly, generative AI is not a silver bullet andshould not be viewed as the solution to every problem. If
intuitive explainability is very important, generative AImight not be the appropriate technology; FIs may wish
to consider other forms of AI that produce more intuitive
explainations.

B.4 Fairness
Incremental challenges of generative AI
In the 2021 paper, we stated that fairness is integral toprevent challenges that can arise when algorithms andlearning models receive data that presents ingrained humanflaws and biases, or when human decisions could make analgorithm discriminate unfairly. Fairness in AI protects FIsfrom deploying technology that undermines their codes ofconduct and ethical values and conflicts with social values.
This continues to be true. Unfair bias can be introduced into
AI systems in a number of ways, for example low quality of,or inadequately representative, training data.

19
Sourab Mangrulkar and Sayak Paul, ‘PEFT: Parameter-Efficient Fine-Tuning of Billion-Scale Models on Low-Resource Hardware’, February
2023, https://huggingface.co/blog/peft

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

24

-

-

-

The challenges of unfair bias are particularly applicableto generative AI given its accessibility to a wide range of
public data, as well as broad public interest, and the factthat many users may not necessarily be able to discern
fact from fiction.
Generative AI models can potentially generate harmful
content, represent and amplify biases, and be at odds withgenerally shared values. Ethical failures in generative
AI systems can also create reputational challenges by
violating social norms and values.
For example, FIs face growing challenges to meet theMonetary Authority of Singapore (MAS) “Fairness” FEAT
principles because of the growing difficulty of identifying
and responding to bias or prejudice in the inputs andoutputs characteristic of generative AI systems.20 This isdue to the fact that it is challenging to assess for bias insets of unstructured data, images etc, which are includedin the datasets of generative AI systems.

Mitigants
Below are some suggested mitigants of Fairness challenges:- The mitigation of unnecessary biased data
The removal or remediation of biased data from thedataset used to train the models results in an output that isprovided only upon the meeting of specifically pre-defined
supporting conditions.
- Standards
A defined standard for fairness with a robust process for
identifying at-risk groups and potential proxies.
- Tried and true
Recommendations of trusted algorithmic methods toidentify and reduce unfairness.
- Measurement
Fairness metrics to be put in place which will help toquantitatively assess fairness in the data set as well as
AI outputs.
- Anonymity
Using anonymised data so as not to use distinguishing
characteristics that could be used to identify and trigger
bias upon output (e.g., replacement of Chairman with
Chairperson).
- Weighting of parameters
During the pre-training of the model phase, assigninghigher or lower weight to parameters related to majority
or minority groups.
- Fair representation
Ensure that the dataset used to train the models is acomprehensive and fair representation of the population.
- Human oversightHave humans interjection in during review process andto remove any bias and reinforcement learning fromhuman feedback. These humans will need adequate
training and expertise to identify discrepancies and beaware of generative AI’s limitations. The people designingor implementing the models also need to evaluate andvalidate the systems to correct for bias and potential harm.
- Right to refusal
Some types of biases can also be mitigated via training
for refusals, by getting the model to refuse responding
to certain questions. However, these refusals can inpractice also amplify biases and there are often publicised
workarounds that can reduce their effectiveness.
- Collaboration
Regulator-industry-academic partnerships to create
guidance on testing methodologies for fairness would be
useful (e.g. the Hong Kong Monetary Authority guidanceon testing for fairness in its white paper on “Reshaping
Banking with AI”, the MAS-led Veritas consortium).

MAS, ‘Emerging Risks and Opportunities of Generative AI for Banks: A Singapore Perspective’, November 2023, https://www.mas.gov.sg/-/
media/mas/news/media-releases/2023/executive-summary---emerging-risks-and-opportunities-of-generative-ai-for-banks.pdf
20

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

25

C. Third-Party Risk
Management and Resilience
Incremental challenges of generative AI
Our 2021 paper notes that “Third-party risk management
and the management of interdependencies is closely linked
to the topic of (operational) resilience”. Given the potential
for generative AI to be used in different use cases in thefinancial industry, combined with the limited transparencyof the FMs, this might lead to higher resilience challenges.
This is potentially exacerbated by limited substitutability
due to the fact that it is less straightforward to replicate
training resources.
- Increased reliance on third parties developing the FMs
With generative AI, reliance by FIs on third parties isincreasing, because FMs are generally developed by third
parties. There is a distinction between accountability andresponsibility, where FIs must remain accountable, but
third parties may be responsible for certain parts of the AI
value chain.
- Concentration challenges
Given the need for large volumes of data to develop a LLM
or FM, the ability to create and run such a model is going to
be limited to organisations with the performance, data, andstorage capacity to operate this, leading to concentration
challenges.

Mitigants
Below are some suggested mitigants of Third-Party Risk
Management and Resilience challenges21:- Minimise shadow AI22
It is easier to manage what you can measure. In order
to effectively mitigate the risks of third-party AI tools,
organisations need a clear view and inventory of uses ofAI within their operations.
- Ensure that third-party risk mitigation is part of yourResponsible AI (RAI) program
Full awareness of AI is not enough. In order to effectively
detect and mitigate third-party risks, an FI’s RAI program

should extend to all uses of AI across the organisations,
whether internally or externally built or developed,
including through any relevant procurement and postprocurement policies- Uplift the organisation’s third-party supply chain
management process
Enhance third-party sourcing processes, AI and generative
AI threats and controls assessment and required contract
clauses, bridging attestation frameworks gaps.
- Continually update and iterate to address new risks
As with all other components of RAI, third-party risk
mitigation is not a one-off exercise. Whether due toadvancements in AI or legal and regulatory developments,
FIs should plan to regularly revisit their approach to thirdparty risk mitigation in their RAI program. There should
also be clear contractual arrangements between FIs
and third-party providers, including obligations aroundtransparency and explainability, at least for criticalactivities.
- Establish shared responsibility modelFIs should negotiate a shared responsibility model in thirdparty agreements, considering the service materiality andthe service reliance on the third-party generative AI models
and reviewing of models not covered under attestation
frameworks. In the absence of any regulatory guidelines
or industry framework on shared responsibility between
AI developers and deployers, several generative AI service
providers have proactively developed measures to addressconcerns by AI deployers. As an example, Google Cloud
indemnifies its customers against liabilities arising froman infringement by its services of a third party’s IP rights.23
Microsoft is also committed to defend its customers from IP
infringement claims arising from the customer’s use anddistribution of the output content generated by Microsoft’s
Copilot services.24 FI deployers can also review the risk
assessment and AI governance frameworks of the thirdparty developers and any gaps can be addressed in thecontractual clauses. The responsibility of the various
controls will depend on the deployment model of thegenerative AI tool and use case. We recommend thatdevelopers, deployers and the regulatory community cometogether to establish a mutual understanding of a shared
responsibility framework.

Elizabeth M. Renieris et al., ‘Responsible AI at Risk: Understanding and Overcoming the Risks of Third-Party AI’, April 2023, https://sloanreview.mit.edu/article/responsible-ai-at-risk-understanding-and-overcoming-the-risks-of-third-party-ai/\
22
Shadow AI - the unauthorised use or implementation of AI that is not controlled by, or visible to, an organisation’s IT department.
23
Neal Suggs and Phil Veneables, ‘Shared fate: Protecting customers with generative AI indemnification’, October 2023, https://cloud.google.
com/blog/products/ai-machine-learning/protecting-customers-with-generative-ai-indemnification
24
Microsoft, ‘Introducing the Microsoft Copilot Copyright Commitment’, September 2023, https://www.microsoft.com/en-us/licensing/news/
microsoft-copilot-copyright-commitment
21

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

26

Incremental challenges of generative AI

be updated to address the scope of AI and specificallygenerative AI challenges. This increases the complexity
on both the third party and the FIs in supply chain
cybersecurity risk management.

The cybersecurity challenges associated with the use ofgenerative AI are a mixture of new threats such as malicious
content generation and ad-hoc exploit creation, as well as
increased exposure to existing cybersecurity challenges
such as the adversarial attacks, data and model poisoning,and data and access control management. Cybersecurity
professionals will be required to upskill their ability andadapt newer strategies as needed.

- Deep Fake and impersonation attempts
FIs rely on biometric identification and authentication
mechanisms (e.g. voice, images, video) for customer
identification and authentication. Generative AI models
could allow malicious actors to quickly train and generatehigh quality avatars based on publicly available data toimpersonate and even deceive security controls resulting
in an increase of fraudulent activities.26

D. Cybersecurity

There are certain areas that require particular attention andconsideration with respect to generative AI:- Standardisation of development lifecycle
The lack of a standardised development lifecycle forgenerative AI due to its relative novelty, potentially leaves
gaps on applying security standards and principles in thedesign phase.
- Lack of commercial / open-source for generative AI
security testing
Security tools, to test against generative AI challenges, toprovide verification and assurance, are still maturing.
- Security controls bypass via generative AI inputs/outputs
manipulation
Use of prompt engineering by skilled adversaries to bypass
model controls built by the model or FM developers, toobtain information and/or perform activities that would
otherwise be unauthorised.25- Third-parties attestation framework gaps
Assessor and attestation frameworks (such as International
Organisations for Standards (ISO)/ International
Electrotechnical Commission (IEC) 27001, ISO 28000, ISO/IEC 15408 and SOC 2 type 2) for third parties have yet to

Mitigants
Below are some suggested mitigants of Cybersecurity
challenges:- Uplift the cybersecurity risk management framework
Assess the impact that AI and generative AI threats haveon policies, technical standards, processes, and control
implementation.
- Cybersecurity assessment against AI and generative AI
use cases
As part of the use case evaluation process, qualify use
cases with the agreed risk appetite and establish effective
governance mechanisms as part of the organisational
cybersecurity risk framework.
- Uplift the organisation’s SDLC
Extending processes to cater to AI and generative AI
specific concerns including data life cycle management(data sourcing, cleaning, maintaining linage between data
model and training data).
- Develop and/or source security training and awarenessprograms
This should be considered for all users, prioritisingdevelopers, security testing and assurance personnel.

Andy Zou et al., ‘Universal and Transferable Adversarial Attacks on Aligned Language Models’, July 2023, https://arxiv.org/pdf/2307.15043.
pdf
26(1) Thomas Brewster, ‘Fraudsters Cloned Company Director’s Voice In $35 Million Heist, Police Find’, October 2021, https://www.forbes.com/
sites/thomasbrewster/2021/10/14/huge-bank-fraud-uses-deep-fake-voice-tech-to-steal-millions/ ; (2) Bloomberg, ‘Deepfake Imposter Scams
Are Driving a New Wave of Fraud’, August 2023, https://www.bloomberg.com/news/articles/2023-08-21/money-scams-deepfakes-ai-willdrive-10-trillion-in-financial-fraud-and-crime?leadSource=uverify%20wall
25

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

27

- Raising awareness of the potential benefits and risks fromthe growing adoption of generative AI
As more FIs leverage generative AI to enhance their systems
and business processes, there is an increasing need toguard against potential challenges, including leakage ofsensitive information and data poisoning. FIs would needto manage these challenges by establishing guardrails,
for example, by raising employee awareness on the safeuse of generative AI and establishing comprehensive data
handling policies.27

NB: Use of Generative AI as additional
approach to mitigate cybersecurity
threats
Generative AI can also serve as an additional tool in the
toolbox for FIs to defend against cybersecurity threats inareas such as:---

Performing initial incident analysis.
Intelligent threat hunting to identify security
vulnerabilities.
Dissect malware code samples to detect trends inmalware software, such as polymorphic code, thatmakes such code more easily identifiable even whenmalware is constantly changing.
Identiy communication patterns to detect a shift insentiment that may suggest social engineering.
Deploying AI-enabled solutions in areas such as securecode development, security monitoring, threat hunting
and red-teaming to improve the effectiveness androbustness of their cyber defence.
Generate synthetic data for cybersecurity training.

E. Expertise
Incremental challenges of generative AI
While the fundamental challenges associated with theuse of generative AI by employees are the same as outlinedin the 2021 ASIFMA AI paper, there are certain areas that
require particular attention and consideration with respect
to generative AI:-

Increased dissemination of false (e.g., deepfakes),
harmful, or sensitive information due to lack ofawareness and expertise.

--

Misuse of generative AI applications due to lack ofcompetency.
Lack of AI experts within FIs risk and compliance teams
to truly understand the capabilities and limitationsof generative AI, and who are then able to effectively
oversee and challenge generative AI implementation.
Understanding that the flexibility/adaptability ofgenerative AI means it is not possible to anticipate or
prevent all possible misuse.

Mitigants
Below are some suggested mitigants of Expertise challenges:- Technical Training
Providing technical staff with a deep technicalunderstanding and education in generative AI. For nontechnical staff, provide education and training to deepen
understanding and raise awareness, which can include
regular updates and internal assessments. Training
should be focused on real world scenarios, where AI cango wrong, the implications, how to improve governance,risk assessment and validation report being reviewed bya diverse group of people.
- Education
Wider education and awareness raising for FI industry (andits users) on the applications and impacts of generative AI
is needed, including various risk factors and regulations.
For example, even where there is a human in the loop, thisentails upskilling and training across employees to provide
a consistent human review of output from the models ,which is also referenced as a mitigant of hallucination
as mentioned earlier in the paper. (See Section ‘7B.1
Hallucination’ above).
- Engage people from the user community to understandharm
Provide instructions for use or a user manual that has theright level of information to interpret the model’s outcomes.
Simple and easy language will help a broad range of users
to understand.
- Oversight
Ensuring AI experts are involved in the process of
onboarding/using generative AI and the relevant
governance committees to ensure appropriate oversight.

MAS, ‘MAS Cyber Security Advisory Panel Proposes Ways to Tackle Mobile Malware Scams and Generative AI Risks for the Financial
Sector’, October 2023, https://www.mas.gov.sg/news/media-releases/2023/csap-2023
27

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

28

F. Environmental Implications
Incremental challenges of generative AI
New research suggests that AI systems can have serious
environmental impacts. 28 This includes its energy
consumption, water consumption for cooling, carbon
footprint, and contribution to e-waste as a result of theincreased use of hardware as compared to other types ofcomputing. As the generative AI models become larger, morecomputational power is needed and as such, more resources
are required.

Mitigants
Below are some suggested mitigants of Environmental
Implications:- ReduceFIs should assess the sustainability of their own andtheir third-party data centers and their use of renewableenergy and incorporate into relevant internal and externalreporting.
- Reuse
Reuse models and resources, particularly by leveraging
global and vendor investments in FMs.29

7 | ASSESSMENT OF NEW AND INCREMENTAL CHALLENGES, EXISTING CONTROLS AND GAP ANALYSIS

29

8

ASIFMA PROPOSEDRECOMMENDATIONSFOR A FUTUREPROOF, RISK-BASEDREGULATORYFRAMEWORK FOR
GENERATIVE AI

Backdrop: Evolving regulatoryenvironment
With the explosive growth of AI applications and theassociated challenges coming to light, governments andregulators across the world, including in Asia, have become
more proactive in developing requirements to govern theuse of generative AI recently. In China, the Cybersecurity
Administration of China (CAC) published the “Interim
Measures for the Management of Generative AI Services”
in July 2023 with more concrete AI regulatory requirements
compared to the previous “Provision of AI Services” issued.
This was followed by the proposed technical standards
on generative AI issued by China’s National Information
Security Standardisation Committee, which provides
specifics on complying with the CAC measures.30 In India,
the Ministry of Electronics and Information Technology
indicated in May 2023 that AI regulations will be coming
after earlier stating that they had no intention to issue AI
regulations not long before. In South Korea, the government
had announced the “Digital Bill of Rights” in September 2023,
creating a set of guidelines and a base standard for future
AI legislation.31 In Hong Kong, the Chief Executive Officerof the Securities and Futures Commission highlighted theimportance of addressing challenges associated with AI
models in her recent speech at the Hong Kong Investment
Funds Association in June 2023. 32 In Australia, thegovernment issued a consultation in June 2023 exploring
the governance approach of responsible AI.33 In Singapore,
the government published the “Singapore National AI
Strategy 2.0”, which is an update from the 2019 AI strategy,
highlighting the need to take a pragmatic approach bysupporting experimentation and innovation, while still
ensuring that AI is developed and used responsibly, in linewith the rule of law and the safeguards they have put inplace.34 In the European Union (EU), the European Council
and the Parliament have reached an agreement on theprovisional rules on the EU AI Act in December 2023, whichaims to ensure that fundamental rights, democracy, therule of law and environmental sustainability are protectedfrom high risk AI, while boosting innovation and making
Europe a leader in the field.35 At the global level, the G7 haveestablished the Hiroshima Process International Guiding
Principles for Organisations Developing Advanced AI
Systems and the Hiroshima Process International Code of
Conduct for Organisations Developing Advanced AI Systems
in September 2023, which was done in partnership with theOrganisation for Economic Co-operation and Development(OECD) and Global Partnership on Artificial Intelligence,
which seeks to promote the development of secure andtrustworthy AI systems internationally and address thechallenges associated with the technology.36 In addition,
the OECD had also published a paper on Generative Artificial
Intelligence in Finance in December 2023, presenting
the recent evolutions in generative AI and its slowpaced deployment in finance, highlighting the potential
challengerisks from a wider use of generative AI tools byfinancial market participants, and discusses the potential
policy implications.

Recommendations for regulator yapproaches towards generative AI
As mentioned in our “Enabling an Efficient Regulatory
Environment for AI” published in June 2021, werecommended that regulators take a principles-and riskbased approach to AI. The principles include supporting
public-private collaboration, allowing FIs to take a risk-based
approach to manage AI-related risks, taking materialityof the use case and stakeholders into account, leveraging
existing regulatory frameworks, striving for regional andinternational harmonisation, promoting, and facilitating
cross-border data flow, and engaging with the industry on
areas that need further discussion.
Generative AI can potentially be used in a whole range offunctions across financial markets, to augment existing
activities, to replace them, or to perform complex andintensive tasks that were not previously feasible.

TC260, ‘Basic security requirements for generative artificial intelligence service’, October 2023, https://www.tc260.org.cn/
upload/2023-10-11/1697008495851003865.pdf
31
Ministry of Science and ICT, ‘South Korea presents a new digital world order to the world!’, September 2023, https://www.msit.go.kr/eng/
bbs/view.do?sCode=eng&mId=4&mPid=2&pageIndex=&bbsSeqNo=42&nttSeqNo=878&searchOpt=ALL&searchTxt=
32
SFC, ‘Reflect, Reset and Refocus: Game Plan for Hong Kong as an Asset Management Hub’, June 2023, https://www.sfc.hk/-/media/EN/files/
COM/Speech/HKIFA-Keynote---Eng_20230605.pdf?rev=e6f1d40a460049f8a3db13c4a0d34ac4&hash=9D7136A93B2E75DAC7DA336B0F3B1B7B
33
Australian Government: Department of Industry, Science and Resources, ‘Responsible AI in Australia: have your say’, June 2023, https://www.industry.gov.au/news/responsible-ai-australia-have-your-say
34
Singapore Government, ‘Singapore National AI Strategy 2.0’, December 2023, https://file.go.gov.sg/nais2023.pdf
35European Parliament, ‘Artificial Intelligence Act: deal on comprehensive rules for trustworthy AI’, December 2023, https://www.europarl.
europa.eu/news/en/press-room/20231206IPR15699/artificial-intelligence-act-deal-on-comprehensive-rules-for-trustworthy-ai
36
Ministry of Foreign Affairs Japan, ‘G7 Leaders’ Statement on the Hiroshima AI Process’, October 2023, https://www.mofa.go.jp/
files/100573466.pdf
37OECD, ‘General artificial Intelligence in Finance’. December 2023, https://www.oecd-ilibrary.org/docserver/ac7149cc-en.
pdf?expires=1704356849&id=id&accname=guest&checksum=394C7A5213BF1D9F34ADA3B42FEA8F37
30

8 | ASIFMA’S PROPOSED RECOMMENDATIONS FOR A FUTURE-PROOF RISKED BASEDREGULATORY FRAMEWORK FOR GENERATIVE AI

31

\

Each use case will have its own risk profile and key
stakeholders. FIs should be allowed to adopt a risk-based
approach that is proportionate to the risk of the particular use
case and its potential impact on stakeholders and measured
against the performance of comparable current processes (ifexisting) or an available human-powered alternative.
We continue to promote this approach and put forward the
additional recommendations and nuances for a balanced
regulatory and industry approach for generative AI:

Principle 1
Leverage existing regulations
The development and use of AI and generative AI are coveredby a myriad of existing regulations in the financial sector
in the areas of cybersecurity, governance, third-party
management, privacy etc. In addition, regulators aroundthe world have been providing guidance – and in somecases, drafting regulation – around the use of AI. Whenassessing the incremental challenges of generative AI,
we urge regulators to leverage these existing frameworks
and mitigants where appropriate, conduct a gap analysis
to identify any challenging areas pertaining to the useof generative AI that are not addressed by the existing
regulatory framework, and focus any additional regulatory
guidance on addressing these gaps.
As identified in section 7.C above, the importance of robust
third-party risk management will continue to grow with the
adoption of generative AI. Regulators and FIs should leverage
existing outsourcing, operational resilience and technology
risk management guidelines and emerging frameworks
on critical third parties. Whilst the ultimate responsibility
for the use of third-party developed generative AI tools will
lie with the regulated FIs, regulators are exploring how theexisting frameworks might have to be amended or expanded
to address challenges across the (generative) AI lifecycle.
As mentioned, accountability of the various controls in thegenerative AI lifecycle will depend on the deployment model,
and we suggest that responsibility (and liability) should lie
with the party who has control over the specific elementof the lifecycle. We suggest regulators, developers and FIs

come together to agree on an appropriate and enforceable
shared responsibility model for generative AI. ASIFMA andits members look forward to inputting into this important
policy debate.

\

Principle 2

Maintain governance and accountability
Organisations must establish a comprehensive governanceframework that reflects legal, compliance and ethicalconsiderations, and transparency to harness the governanceand accountability pertaining to the use of generative
AI. We recognise that many organisations have alreadyinvested in these frameworks to manage the challenges of“traditional” AI, whether as new structures or within existing
frameworks. Hence, we suggest that such frameworks
should be validated against generative AI specific challenges
to validate coverage, but do not need rebuilding from scratch.
In line with the Securities Industry and Financial
Markets Association (SIFMA) response to the National
Telecommunications and Information Administration
request for public comment on AI system accountability
measures and policies, we suggest that a risk-based AI
governance framework should include the following
components:37
1. Scoping: Companies should determine which AI
applications are in scope of the framework when building
their governance programs.
2. Inventory: Companies should prepare and maintain an
inventory of their AI applications with sufficient detail
to allow them to be risk rated.
3. Risk Rating: Companies should have a process for
identifying their highest-risk AI applications. The
risks considered would include legal and regulatory
risks, including operational, reputational, contractual,discrimination, cybersecurity, privacy, consumer harm,lack of transparency, and confidentiality risks.

37
SIFMA, ‘Response to Request for Comments on AI Accountability Measures andPolicies’, June 2023, https://www.sifma.org/wp-content/uploads/2023/06/Response-to-Request-for-Comments-on-AI-Accountability-

8 | ASIFMA’S PROPOSED RECOMMENDATIONS FOR A FUTURE-PROOF RISKED BASEDREGULATORY FRAMEWORK FOR GENERATIVE AI

32

4. Responsible Persons or Committees: Companies should
designate one or more individuals or committees who
are responsible for identifying and assessing theirhighest-risk AI applications, and either accepting thoserisks, mitigating them, or abandoning the particular AI
application because the risks are too high.
5. Training: Companies should develop training programs
to ensure that stakeholders are able to identify thechallenges associated with their AI use and the various
options for reducing risk.
6. Documentation: Companies should maintaindocumentation sufficient for an audit of the risk
assessment program.
7. Audit: Companies should conduct periodic auditsthat focus on the effectiveness of the risk assessment
program, rather than on individual AI applications.
Companies should be permitted to determine how andwhen audits should be conducted, and who can conduct
those audits.
8. Third-Party Risk Management: Companies should
use the same risk-based principles that are applied
to in-house AI applications to evaluate third-party AI
applications, and mitigate those risks through diligence,
audits, contractual terms, and internal testing. There
should be referenced to adapting the risk-based principles
to the context of third-party models and applications. (See
Section ‘7B.3 Transparency, Explainability, Traceability’
above)

accessibility and democratisation of generative AI, its
potential widespread use, and the limited substitutabilityof the FMs.

\

Principle 3

Provide the appropriate level oftransparency
Transparency in AI refers to the level and quality of
disclosure provided regarding the application of AI inservices and/or products, including the challenges that may
be involved in AI usage. Explainability typically refers to theextent to which workings of a model can be understood. Inline with our 2021 paper, we suggest the regulatory focus
should be on transparency of generative AI models insteadof explainability which will allow a firm to demonstrate
how the AI application has been developed, how it will be
used and monitored, and how it can stand up to scrutiny andchallenge. Within these broad themes, transparency shouldmeet the varied needs of individual types of stakeholders,both inside and outside the firm.
Trustworthy generative AI starts with transparency on twolevels, to supplement our existing guidance: transparency
from the developer towards the deployer, and transparency
from the deployer towards the consumer.

This proposed framework could be incorporated into existing
governance and compliance programs in related areas suchas model risk, data governance, privacy, cybersecurity,
vendor management, and product development, withfurther guidance from applicable sectoral regulators asneeded. Further, having qualified persons identify, assess,and mitigate the challenges associated with the highestrisk AI uses improves accountability, appropriate resource
allocation, and employee buy-in through clearly defined andfair processes.

Transparency from developers towards deployers tomanage risks from AI-generated systems:

Such risk-based approach would provide a valuable,
flexible framework through which FIs and their sectoral
regulators can build tailored AI governance and complianceprograms that ensure accountability and trust without
stifling innovation or wasting time or resources on lowrisk generative AI applications.

--

Transparency from deployers towards consumers:-

These AI governance principles also apply to the use ofgenerative AI. Robust governance and accountability
frameworks are more crucial than ever given theTransparency on type of data sets used.
Transparency around model development, limits, andtesting.
Transparency around the collection and use of
personal data.
Transparency on ownership of data and IP with aparticular focus on firm-specific data sources rather
than base LLMs.
Clear guidelines are needed to determine theresponsibility of platforms and content creators to label
AI-generated content.

Labelling AI-generated content will allow consumers
to make more informed decisions and is of particular
relevance to generative AI as it will help inform futureuses of this data, including by other generative AI models.

8 | ASIFMA’S PROPOSED RECOMMENDATIONS FOR A FUTURE-PROOF RISKED BASEDREGULATORY FRAMEWORK FOR GENERATIVE AI

33

\

Principle 4
Continue to adopt a risk-based approach
Policymakers should adopt a governance framework forgenerative AI that is risk-based and not overly prescriptive.
A generative AI governance framework should treat AI
models, algorithms, applications, and systems (collectively,“AI applications”) differently depending on the likelihood orseverity of the potential harm they might cause. The conceptof same business, same risks, same rules should continue toapply in the context of generative AI.
Such risk-based approach to generative AI governanceprovides the necessary flexibility to balance the potential
challenges with the potential benefits and opportunities in
deploying generative AI. There are several considerations
associated with a risk-based governance framework for AI,
including (1) which specific challenges a company should
consider when deciding which AI applications are highrisk, (2) how best to mitigate the challenges associated withhigh-risk AI applications, and (3) which AI applications
carry unacceptable challenges and should not be pursued.
The granular determinations of such considerations are
best made by the company’s management, with guidancefrom its applicable sectoral regulators. Regulators therefore
should primarily guide companies to focus their efforts on
identifying their highest-risk AI uses, and on mitigating
the challenges they present, as well as define what “highrisk” means for firms to focus on instead of bucketing “highrisk” use cases without clear criteria. Such determinations
and governance will then already exist for other high-risk
models that use traditional AI, or other forms of models.

Principle 5
Continue to adopt a technologyagnostic regulatory approach
We urge regulators to continue to adopt a technologyagnostic approach when assessing and addressing any
incremental risks associated with generative AI. The
technology is fast-evolving and is therefore important toavoid designing regulation based on a particular technology
and to avoid directly or indirectly dictating the use of any
one type of technology over another. Such technologyagnostic approach to regulation will accommodate future
innovation without requiring regulatory reforms eachtime new technology is implemented and will allow themarket to innovate and capture risk appropriately. In
contrast, a technology-specific regulation may run therisk of subsuming technology used in traditional financial
activities into incongruous regulatory perimeters. Moreover,existing regulations that only permit certain technologies
to be used should be updated to accommodate the changing
technology landscape.
Generative AI and FMs are in their early development stage
and continue to evolve. There are no universally agreed
definitions which we believe is appropriate given the
evolving nature of the technology and to ensure regulations
remain future proof. Any definitions of these concepts in any
future regulation should be broad enough to be adaptable.
Regulators should focus on regulating the input and output,
rather than the technology itself and should not favour one
technology over another.

When assessing the challenges and deciding on theappropriate control framework of a particular use case, firms
should take into account factors such as the potential impact
of its use on the firm’s resilience, on financial stability, andon end-consumers, whether the generative AI application/use case uses public data or proprietary data, and whether
it is an enterprise vs a consumer application.

CourtListener, THALER v. PERLMUTTER, 1:22-cv-01564, (D.D.C.), August 2023, https://www.courtlistener.com/docket/63356475/thaler-vperlmutter/Measures-and-Policies.pdf
38

8 | ASIFMA’S PROPOSED RECOMMENDATIONS FOR A FUTURE-PROOF RISKED BASEDREGULATORY FRAMEWORK FOR GENERATIVE AI

34

Principle 6

Principle 7

Address any IP Protection challenges

Strive for regulatory certainty and aharmonised frameworkFIs using generative AI are at risk of unintentional IP
infringement given the limited control over the input andoutput of the FM. The existing legal framework may need beclarified, broadened, or strengthened to reflect the increased
use of generative AI. For example, clarity is needed onwhether fair use exceptions apply to training data.
It is an open debate as to whether current IP and copyrightlaws are still fit for purposes in the context of generative
AI and its related content. This is both in terms of the input
data, as well as the ownership of the content generated by
AI. In August 2023, a US federal judge has ruled that works
produced entirely by generative AI cannot be copyrighted.38
Policymakers need to provide clarifications on copyrightin their respective regulations.
There is also need for greater international alignment
given the existing different approaches being taken. E.g.
AI training requires the extraction of information fromdatasets which may contain copyright-protected works– which may be copyright infringement unless this falls
under an exception (transient and temporary copying ortext and datamining exceptions in the EU or fair use in the
US. We also note that Japan currently takes the approachthat datasets for training AI models do not violate copyrightlaw. Singapore amended its Copyright Act to introduce a
new exception to copyright infringement for use of works
for text and data mining, which is expected to significantly
increase the availability of training data for AI. In addition,
China’s recent proposed technical standard on generative
AI devoted some attention to IP protection (e.g. operators
to establish mechanisms to address complaints from IP
owners). According to the guidance, the process should
enable complainants to query the entire training corpus
to learn whether it includes proprietary material etc.
On the other hand, the United Kingdom did not follow
this approach and the EU AI Act requires publication of
copyright data used for training.

Leveraging the potential of generative AI and addressing
the challenges demands a coordinated response. There is aneed for collaboration between all stakeholders to achievea harmonised approach which will ensure that challenges
can be sensibly mitigated, and the full potential achieved.
This includes close collaboration between regulators and FIs
including through consultations before any new or amended
regulations or regulatory guidance is issued. Likewise,
regulators should promote consistency across jurisdictions
through globally recognised standards. Financial services
regulators should also consider collaboration with regulatorsin other areas of the economy. There is a risk that standard
setting in the wider economy could impose undue
constraints on financial services innovation and inclusion.
Evolving technology (and hence challenges) as well as
evolving regulatory frameworks make it challenging forthe financial services industry to adopt generative AI (given
the investment it takes to research, develop, govern, and thenand roll it out) and the appropriate governance structure
to go with it. Therefore, we need a risk-based, transparent
regulatory framework to allow adoption that should offer
wider benefits in efficiency and inclusion. Regulatory
fragmentation exacerbates the cost and therefore we neednational and international coordination and alignment.
Sectoral regulators overseeing AI should cooperate to ensurethat firms are not subject to conflicting obligations. Any new
specific AI regulations and guidelines should be appliedconsistently across bank and non-bank FIs to ensure that
consumers remain protected wherever they choose toreceive their financial services.

8 | ASIFMA’S PROPOSED RECOMMENDATIONS FOR A FUTURE-PROOF RISKED BASEDREGULATORY FRAMEWORK FOR GENERATIVE AI

35

9

GLOSSARY OF
ACRONYMS

AI

Artificial Intelligence

API

Application Programming Interface

ASIFMA

Asia Securities Industry and Financial Markets Association

CAC

Cybersecurity Administration of China

EU

European Union

FI

Financial Institution

FM

Foundation Model

GPT

Generative Pre-trained Transformer

IEC

International Electrotechnical Commission

IPIntellectual Property

ISO

International Organisation for Standardisation

LLM

Large Language Model

MAS

Monetary Authority of Singapore

ML

Machine Learning

OECD

Organisation for Economic Co-operation and Development

RAI

Responsible Artificial Intelligence

SDLC

Software Development Lifecycle

SIFMA

Securities Industry and Financial Markets Association

US

United States

9 | GLOSSARY OF ACRONYMS

37