Advancing AI Audits forEnhanced AI Governance
AI Audit Study Group, AI Governance ProjectTechnology Governance Policy Research Unit
Institute for Future Initiatives, The University of Tokyo

The opinions expressed in this presentation are the opinions of this study group
and not those of the entities to which members belong.

Agenda1. The Need for Summarizing AI Audit Issues

4

2. Issues Related to AI Audits

6

3. Factors that make AI Audits Difficult

17

4. Future issues and recommendations on AI auditing

21

2

Our target is Audit for AI. Not AI used in audit
The premise of wording “AI Audit” in our policy recommendation

AI Audit - Topics in the context of AI and Audit
AI Used in Audit

Audit for AI- Perform Audit proceduresfor AI services and systems- Using AI service or system in our current
audit procedures

Input
DecisionTargetCompany/
Organization Analysis

AI

Auditors

Analyzing

Audit
procedure

Our target

Business
TargetCompany/
SystemOrganization
Suspicious
trade

Auditors

Sales, Accounting
data etc.

Input

Analyzing Analysis AI owned
by auditor

Out of our scope3

1. The Need for Summarizing
AI Audit Issues

4Without common understanding, misunderstanding occurs
1. The Need for Summarizing AI Audit Issues
AI audit is to perform audit procedures efficiently andeffectively with using AI technologyAlex

I want to discuss internal audit for our sensor using deeplearning technology to confirm excess or not certain threshold.
Our bank client introduced machine learning model for their loan
execution judgement. We plan to external audit for their internal controls.

Bob

Chris

• If they discuss AI audit, including audit procedures and timing, it doesn’t work
properly. In order to have effective discussion, we have to be on the same pageand clarify what we are discussing.
• This report is intended to summarize the issues related to AI auditing and toestablish a common foundation for discussing critical issues among theparties concerned.

52. Issues Related to AI Audits

6

Introduce several issues related to AI audits for discussion
2. Issues Related to AI Audits

The necessity for AI auditing

Currently, there is no regulation to mandatory require AI audit
in Japan. However, people have much concerns for AI audit to
use AI service safely

AI Audit Proof Propositions

We use the term "proof propositions" to describe what should beconsidered audit topics when conducting an audit. Considering AI specificconcerns, it’s hard to address them with only current standards / criteria.

AI Audit Scope

The AI systems and services’ audit scope can be classified into two
categories: auditing individual AI services and systems and auditing theinternal controls implemented in organizations that provides AI services

Timing of AI Audits

We classifies the AI lifecycle into four phase: (1)new development,(2)functional change or additional development, (3)operation and(4)disposal. Identified audit timing based on audit categories.

AI Audit Practitioner Requirement

The AI audit practitioner requirements are presented from the perspectivesof (1) expertise requirements, (2) independence requirements,(3)organizational requirements, and (4) legal responsibilities of the auditor.

AI auditing Parties andOrganizations Involved

Auditors and audited companies are not the only parties
involved in auditing AI services and systems. We organized
relevant parties with charts.

7

Traditional audit proof propositions are not enough to cover all
AI audit specific needs
2. Issues Related to AI Audits – AI Audit Proof Propositions #1
Examples of audit proof propositions of traditional service and system audit
Proof Propositions

Standard / Criteria
exampleIT audit in part offinancial statement
audit

Risks arising from the use of IT are appropriately
reduced to reflect their financial status on
financial statements accurately and completely(Accuracy and Completeness). (*)

International Standard on
Auditing (ISA) 315

SOC2 reportassurance

Security / Availability / Process Integrity /Confidentiality / Privacy

Trust Service Criteria

Governance process, risk management andcontrol validity and effectivenessInternational Standards for theProfessional Practice of Internal
Auditing(IPPF)
Japanese system audit standard

Internal audit

Part of them can be used for AI audit but not enough to cover all AI audit
specific needs(*)Management assertions like existence and completeness are applicable for entire financial statement audit proof propositions.
However, we listed accuracy and completeness of the process as a representative of automated control assertion in this table.

8

Common understanding are needed because our discussion aremuch different based on AI audit specific proof propositions
2. Issues Related to AI Audits – AI Audit Proof Propositions #2
Proof Propositions that are unique to AI audit (Example)
AI Principle

Items frequently
appear in various
guidelines

Consumer fears inusing AI service
etc.

Proof
Propositions

Explanation

Fairness••

Is there any inappropriate bias in the output results of the AI system, etc.?
It is required to have a common understanding of the definition of fairness
beforehand.

Transparency•

Can the output results of the AI system be reproduced, and can the
training data and feature values be explained, etc.?

•

Is there any possibility that the AI system may harm the user? If a problem
occurs, does the system properly transition to a halted state etc.?
Hardware in which AI systems are embedded should also be considered.

Safety•Security••

Can attacks on training data be prevented or detected?
Can production input data that intentionally induce inappropriate
output be prevented etc.?

Privacy••

Can individuals refuse to attribute data they do not wish to share?
Can erroneous personal assessments be corrected in a timely andappropriate manner etc.?

• Audit procedures and criteria are much different depending on each proof
propositions. Hence, Common understanding are needed in auditing and discussion.
• Only current standards / criteria are not enough to cover all proof propositions.

9Expected AI audit scope are different depending on the
background
2. Issues Related to AI Audits – AI Audit Scope #1

AI Audit - Topics in the context of AI and Audit
Audit for AI - Perform Audit procedures for AI services and systems
Auditing AI services andsystems

Audits of internal controls inorganizations that provide AI
services and systems

Mixed approach isacceptable- Using AI service or system in our current audit procedures

AI Used in - (Example) Detecting suspicious trades from numerous
trade data
Audit
-Out of our scope

10

Auditing AI services and systems
2. Issues Related to AI Audits – AI Audit Scope #2
AI Service
AI System operating server and client terminal
AI system software(Application, OS, DB)AI model program(Application)
Trainedmodel

Training
data

Need to expand traditional
application audit such as
audit for data governance,
algorithm auditing etc.

Hardware

AI Service
providingframework

Serverperformance
DeviceOS- AIサービスやシステムに対して監査手続を実施する
performanceDB

Softwareinfrastructure

Middleware
Self-driving car
Cameraetc.

In addition to traditional
issues like server hardware
performance and physicalsecurity, need to consider
device performance which
is close to AI output
accuracy and performance.

• Appropriate disclosure about
using AI for users• Appropriate explanation
about collected information
Need to consider service
providing framework too.

11

Audits of internal controls in organizations that provide AI
services and systems
2. Issues Related to AI Audits – AI Audit Scope #3
AI service andsystemOrganization that provide AI services and systems- Management, User division and system developing or operation team

Rule
Guideline

Structure
PolicyEntity level control for entire organization

AI utilize policy
Education structure etc.

Daily operation
Following rule

Individual work level control

Test and approval before release
Confirmation for training data validity
etc.

Design and operate each control
Framework like COSO and COBIT(*)COSO : Internal control framework from by Committee of Sponsoring Organizations of the Treadway Commission
COBIT : Entity information and technology framework by ISACA and ITGI

12

Need to AI specific consideration when determine audit timing
2. Issues Related to AI Audits – Timing of AI Audits
AI life cycle
Planning

PoC

Develop

Testing

New development

Release Planning

PoC ・・・ Release ・・・ Operation Disposal

Functional changes orAdditional development

Issues around timing of AI audits
Performance updates with continuous training• Performance and accuracy may differ between the AI output results at the timing of
the audit procedure and AI output results when such results are used.
Timing difference based on audit scope and auditor• Timing differs depending on audit scope (AI services / systems or internal control) andauditor (internal audit or external audit).
• In providing AI services, the appropriateness of the service itself and the appropriatenessof decisions on whether or not development is necessary are also important issues.
Therefore, external audits that include the planning and PoC phases may be considerednecessary.
13

In addition to AI specific knowledge as requirement, AI auditor
certification and organization accreditation should be considered
2. Issues Related to AI Audits – AI Audit Practitioner Requirements
Expertise requirements•

Internal
audit
External
audit

•

•

An understanding of audit theory, industry knowledge
related to the audited company and the services it
provides, and knowledge and experience in IT areas not
limited to AI.
Deep AI specific technical knowledge, and ethical,
cultural, legal, and regulatory expertise. New
certification program should be considered too.
The requirements for those conducting AI audits arevery high, and an individual is unlikely to possess all theskills and experience required for AI auditing.

Independence requirements•

As with traditional audits, AI audits must be conductedby independent practitioners and organizations thathave no conflict of interest with the audited company
or department.

•

Independence requirements are defined for both
external and internal audits.

Organizational Requirements•

External
audit

•

•

The organization conducting the audit must meetcertain quality and independence standards.
The certification and accreditation system for theorganization, as well as the organization and its role in
conducting monitoring, must be considered.
Notably, the audit results conducted by anorganization that does not meet these standards may
not credibly reflect the actual situation.

Legal responsibility of the auditor•

The scope of AI audit practitioner’s legal liability withdue care.

•If the responsibility of AI audit practitioners is too
burdensome, very few would be willing to conduct theaudits.

•The need for exemption requirement or insurance
system to protect AI audit practitioners

14

Many internal and external parties are relevant to audit,
assurance and certification for AI
2. Issues Related to AI Audits – AI auditing Parties and Organizations Involved

AI service user, organization

Outsourcing
company

Library
providerOutsourcing
company

DataproviderAI service user organization
Audit
committee

ServiceprovideAI service

Board ofdirector
Management
First line
(User department)
Second line
(Risk management
department)Internal
audit

External audit(Statutory/
Voluntary)

AI service provider
organization
Audit
committee
Board ofdirector

Certification
First line
(User department)
Second line
(Risk management
department)Internal
audit

Non-business
user

Third line
(Internal audit
department)

External
auditorStandardization bodyManagement

Third line
(Internal audit
department)

External audit(Statutory/
Voluntary)

Certification
Certification
body
Accreditation
bodyStandard
15

Many internal and external parties are relevant to audit,
assurance and certification for AI
2. Issues Related to AI Audits – AI auditing Parties and Organizations Involved
Society andCommunity
ServiceprovideAI service user, organization
AI service user organization

Outsourcing
company

Library
providerOutsourcing
company

DataproviderAI service provider
organization

External
auditorAI service
External audit(Statutory/
Voluntary)

Non-business
user

External audit(Statutory/
Voluntary)

Standardization body

Certification
Certification

Certification
bodyStandard /Criterion
Guideline

Other party
Industry
association

Insurance
Company

Academia

Educational
institution

Legislation(Regulation /Punishment)

Public institution(Domestic /Oversea)
Private organization(Domestic /Oversea)

Legislation(Institutional
design)Standard /Criterion
Guideline163. Factors that make AI Audits Difficult

17

Complicated technical, institutional, and social factors makeauditing AI services and systems difficult
3. Factors that make AI Audits Difficult••

Difficulties in Setting PerformanceStandards for AI Audits

Complexity of
AI Technology

Technical
factors

Institutional
factors•

Underdevelopment ofinstitutional design for AI audits•

Complexity arising from thescope of audited entities•

Imbalance between demandand supply for AI audits

Socialfactors
18

Steady discussion for resolving each factor is necessary to realize
AI audit
3. Factors that make AI Audits Difficult◼ Unlike current IT systems, AI systems are not designed to output pre-defined uniquevalues.
Complexity of
AI Technology◼ In the development of AI systems, the logic of decision-making is often black boxed,and it is difficult to verify the logic.
◼ Some AI services and systems conduct continuous learning. In this case, an evaluation
at the time of the audit may differ from an evaluation when the audit results are used.
◼ AI-specific general standards had not yet been established for audit practitioner
requirements and quality management systems.

Underdevelopmentof institutional
design for AI audits◼ No standardized criteria or standards of practice (performance standards) had yet been
established for audit procedures, leaving auditors to design their own procedures andjudge the audit results.
◼ There are no unified rules concerning with which domestic and international rules,
management, auditors, certification bodies, and other parties must comply.

Difficulties in Setting
PerformanceStandards for AI
Audits◼ Some proof propositions, including fairness, are difficult to define precisely anddifficult to verify.
◼ Trade-offs can be assumed between the audit’s proof propositions. It’s difficult tocover multiple audit perspective with one audit procedure.
◼ When considering the training data governance effectiveness, it’s difficult to set
sufficiency indicators and potential biases in such data.
19

Steady discussion for resolving each factor is necessary to realize
AI audit
3. Factors that make AI Audits Difficult

Complexity arisingfrom the scope ofaudited entities◼ Because there are many cases that lots of companies are involved in development as oursourcing companies, it’s difficult to set audit scope organization. In addition, contractual
obligation and audit fee burden issues also should be considered.
◼ If they prepare models with external library, the correct operation of AI systems isseemed to rely on these function. The audit scope becomes even broader, without theassumption that the functions provided by these libraries are functioning properly.
◼ If they use third party data collectors and public data providers for training data, theseparties should be considered in audit scope.
◼ A huge gap may develop between the requirements of AI audits from society andactual audit result.

Imbalance between
demand and supply
for AI audits◼ Considering the efforts for performing AI audit, legal responsibility of the auditor andaudit fee the auditors receive, they can be imbalanced. In such case, there is little
incentive for external auditor to undertake such work.
◼ Lack of incentives for audited companies to undergo AI audits because there are noneor little legally binding regulation with penalty.

To perform AI audits is difficult because of complicated factors. However,considering AI audit demand from expanding AI use cases, discussion in oversea
and activities for international rule setting, it’s better to proceed discussion as
much as we can20

4. Future issues andrecommendations on AI auditing21

We hope that this paper will contribute to realizing responsible AI development and operation by
AI system and service companies, and a society in which AI can be used safely and trustfully

4. Future issues and recommendations on AI auditing

1
2
3

Development of institutional design for AI audits

Institutional design for AI audits to meet the demands has not been prepared.
Considering increasing international demands, further discussion is needed.

Training human resources for AI audits

Conducting an AI audit requires diverse expertise with a wide range of skills andexperience. To meet the demands, training AI auditors and their new certification
should be discussed.

Updating AI audits in accordance with technological progressTechnical research related to AI is flourishing around the world, and new services andsystems are emerging daily. the AI audit institutional design, standards and methods
must be updated to avoid obsolescence.

22

Our policy recommendation consists of lots of topics
The contents of our paper
Chapter 1

Toward AI Audits that Contribute to AI Governance

Chapter 2

Issues Related to AI Audits

Introduction

Issues

Chapter 3

Factors that make AI Audits Difficult

Chapter 4

Assumed AI audit case study: Recruitment AI

Chapter 5

Future issues and recommendations on AI auditing

Practice

Summary

Chapter 6

Toward a society that can use AI safely and trustfully

Appendix

AI Auditing and Generative AI

Appendix23

Reference URL

AI監査ガバナンスに資するAI監査の実践に向けて
https://ifi.u-tokyo.ac.jp/news/16814/
Advancing AI Audits for Enhanced AI Governancehttps://ifi.u-tokyo.ac.jp/en/news/12034/

24

About the AI audit study group
This policy recommendation is the result report of the AI Audit Study Group, a study
group of the AI Governance Project, a project of the Technology Governance Research
Unit of the Institute for Future Initiatives, The University of Tokyo. The AI Audit Study
Group started its research activities in June 2022 and consists of the following members.
• Arisa Ema（Associate Professor, Institute for Future Initiatives, The University of Tokyo）• Ryo Sato（Visiting Researcher, Institute for Future Initiatives, The University of Tokyo /Deloitte Touche Tohmatsu LLC）
• Tomoharu Hase（Visiting Researcher, Institute for Future Initiatives,
The University of Tokyo / Deloitte Touche Tohmatsu LLC）
• Masafumi Nakano（Visiting Researcher, Institute for Future Initiatives,
The University of Tokyo / The University of Toyo）• Shinji Kamimura（Deloitte Touche Tohmatsu LLC）
• Hiromu Kitamura（CDLE, AI Legal (NEC) / IRCA (International Register of Certified
Auditors) “Japan” Members Supporter）

The opinions expressed in this presentation are the opinions of this study group and not those of the entities to which members
belong.

25

Acknowledgements
Many people provided valuable advice in the writing of this policy recommendation.
Due to time constraints and organizational affiliations, we are unable to include all
their names, but acknowledge and thank them for the useful feedback.
• Takashi Akoshima（Member of Japan Society for Systems Audits, Certified Internal Auditor）• Naoto Ichihara（Ernst & Young ShinNihon LLC, AI Leader）• Koichi Ito（PricewaterhouseCoopers Aarata LLC Assurance Innovation & Technology, Partner /Deputy Chief of AI Audit Lab）
• Teppei Usuki（KPMG AZSA LLC, Digital Innovation, Partner）• Nobuhiko Kato（Ernst & Young ShinNihon LLC, Digital Leader）• Yuji Shimada（Guest Researcher, Research Institute of Industrial Technology, Toyo University）• Kiriko Shimizu（PricewaterhouseCoopers Aarata LLC Assurance Innovation & Technology,
Senior Manager）• Hiroshi Taki（Professor, College of Business Administration, Ritsumeikan University）• Hiroshi Naka（Professor, Institute for Future Initiatives, The University of Tokyo / 2023–24
Director-Research for the Board of Trustees (BOT) on the Global Board of Directors of The
Institute of Internal Auditors (IIA) ）The opinions expressed in this presentation are the opinions of this study group and not those of the entities to which members
belong.

26