International Symposium on AI Standards:
towards trustworthy AI through standards

Human-Machine Co-Evolution
System Guidebook
4th/July/2024
Akihisa Morikawa
Project Leader of HMCES Project
Director of Imaginary Corporation
Corporate Officer of Service Design Business Area, Witz Corporation

My self-introduction
I specialize in developing highly safety
and reliable embedded systems.
• Biography• Mainly experienced in new development of embedded software for informationhome appliances, in-vehicle software PF development, etc.
• Started functional safety development in 2006. Successful acquisition of IEC
Published: 2022/11/14
61508 process certification in 2010 for the first time in Japan. Moreover in（from Amazon）
2012, ISO 26262 process certification is the first time in the world.
• Leading business centered on functional safety / product safety / AI safety(supporting more than 100 companies in Japan)• Organize methods for conforming to functional safety standards for AI and makeinternational technical proposals (publish technical paper)https://arxiv.org/abs/2008.01263• Contributed to the formulation of the AI functional safety standard(ISO/IEC TR 5469) at ISO/IEC JTC1/SC42 WG3Published: 2022/7/1• November 2022: Publish a book that explains the points of safety assurance（only PDF from JSA）
measures for AI systems in a story style2

INDUSTRY-ACADEMIA-GOVERNMENT JOINT RESEARCH:RESEARCH AND DEVELOPMENT OF SAFETY CASE TECHNOLOGIESFOR AI SYSTEMS SUCH AS AUTONOMOUS DRIVINGSEAMS Project https://www.seams-p.jp/
Ministry of Economy, Trade and Industry (METI) Support Project (2017 to 2019)<Main R&D results>• Functional safety design/evaluation patterns for AI systems• "Safety design concepts for statistical machine learning components toward
accordance with functional safety standards“• Akihisa Morikawa (Witz Corp.), Yutaka Matsubara (Nagoya Univ.), https://arxiv.org/abs/2008.01263• How to quantify uncertainty in AI systems• based on IEC/TS 62998-1• Building a reliable ML development process (procedures, templatesand checklists)• based on 8.5 of UL 4600, Annex B of ISO/TR 4804, Automotive SPICE v4, etc• Comprehensive verification method for complex conditions (using virtual simulation)
3• Several concrete developments of AI systems

Agenda1.Technical issues for the future society we envision
2.Our R&D Project “HMCES Project”
3.Human-Machine Co-Evolution System Guidebook
4.On-going tasks

4The future society we envision
Serious problemsin Japan・Decrease inproductivity (aging
society, shortened
working hours, etc.)
・Low sense ofhappiness (47th inworld happinessranking)Im
provement

Advanced symbiotic
society of humans andmachines (with AI)• improves well-being by•

optimizing for eachindividual
continues to improve
productivity, convenience,safety, and happiness“Co-Evolution” ofhumans, machines (AI), and society
1. Not only AI’s improvement
2. Changes in human values and behavior
3. Changes in laws, infrastructure, lifestyles

5Issues of continuous evolving social system
Technical Issues

Case of conventional systems

Case of co-evolving systems(1) Verification
timing of continuouschanging systems(2) Quality andsafety assurance of
unknown systems

Thoroughly verify beforedeploying.

(3) Human trackingof machine andenvironmental
changesCompliance with safety
Humans also need to be
operation rules for the assumed flexible to keep up withsystemchange.

Since it is constantly changing,
there is no timing forverification.
Extract unknown risks to theIt is not convenient to detectassumed system as much asdeviations from the assumed
possible (using Guideword) and system (concept drift) and shut
implement risk mitigation
down the system.
measures (SOTIF)

6

Guidebook against issues of continuous evolving social system<Co-evolution Guidebook>
Supporting the Safety and Security of continuous
evolving social system••

Establish new design and verification methods
Guides to enable human flexibility and adaptability

Based on＜Key Existing Technologies＞
・Human-Machine Teaming（HMT）
・Human behavior analysis
・Resilience Engineering (Safety II)・Collaborative Safety・Other several AI standards (by SC42)

7

Our guidebook vs Key existing practices
Goals to be achieved
through co-evolution

Key practices

SafetyImprove
mentAchievementof projectobjectives(e.g.,increasedproductivity)

Improved
wellbeing ofstakehol
dersFunctional Safety－－SOTIF－

Collaborative Safety
Human behavior analysis

－

HMT

－

Monitored for "change"
toward goal attainment

Uncertainty about "future change"

Entire system*The speed of change differs from each other.

Known
Unknowns

Unknown
knownsUnkownUnkowns（Knowingwhat Iknow）（Knowing
what I don’tknow）（I don’tknow what Iknow）（I don’tknow what Idon’t know）－－－

Surrounding
environment(physicalinfrastructure)

Regulations
and rules(logicalinfrastructure)

Stakeholders,
including users－－－－－－－－－－－－－－－－－－－－－－

Machine(mechanical/electronicsystems/AI)

Known
knowns－－

HMCES Guidebook
Measures

Conventional issues：• There is nocomprehensive practicethat takes into accountother factors than safety.

Measures

Conventional Issues：• No good practices exist to adapt to comprehensive changes in
the overall system.
• Safety assurance methods have not yet been established foran overall system that changes frequently.
New risks：• Complex changes increase risks that cannot be predicted orforeseen by humans.
8

Position of our HMCES Project【Industries】••

Application

AI system alone: Development is
becoming more active. Interest in quality
and safety is increasing.
PreconCo-Evolution/symbiotic society:ditionunexamined
Contribution: Points to note/PF/Application
examplesReducing costs/improving safety【Standardization】•••

AI system alone: Compliant with internationalstandards such as ISO/IEC JTC1/SC42, EU AI Act, NISTAI RMF, etc.
Human-machine collaboration: Developing
collaborative safety standards
Co-Evolution/symbiotic society: Standard development
not yet started

References

Contributing to thestandardization of guidebook

HMCES Project https://www.hmces-p.jp/「R&D of co-evolution guidelines
to continuously improve the potential of machines」
Ministry of Economy, Trade and Industry (METI) Support Project (2022 to 2025)

References【Researchers】••

Raising various issues
No specific solution presented(unestablished)【Existing technologies】

・Resilience engineering (Safety II)
・Human-Machine Teaming(HMT) / Human-AI Teaming
・behavior analysis ・Human-Machine collaboration practices
9Organization of HMCES Project[R&D Budget]
Ministry of Economy, Tradeand Industry (METI)[Management]
The Public Foundation of Chubu Science andTechnology Center (CSTC)[R&D members]• WITZ Group• Imaginary [Project Leader]• Witz• Atelier• Nagoya University [Sub Leader]• National Institute of Advanced
Industrial Science and Technology（AIST）• Gomes Company[Advisors/Observers]• Japanese members of ISO/IEC
JTC1/SC42• Big maker companies• Mitsubishi• Suzuki• Aisin• Kobelco• Meiden• Hitachi• Japan Automobile ResearchInstitute (JARI)• Certification bodies• UL Japan
• DNV

10

Requirements forhuman-machine
cooperative systemdevelopment forsafetyArrangement of requirements to ensure the safety[Theme-1] and security of human-machine cooperative systems

Standardization activities
Contribution
System operation
requirements/social
rule requirements forsafety[Theme-3]
Advanced Pilot
APPLICATION

Development of integrated control
robot for hands, feet, eyes, ears,
and mouth[Theme-2]

Feedback(for AI system, Collaborative
Safety, etc)

Guidelines / laws

"Advanced symbiotic society"of humans and machinessocialimplementation＜Examples of application＞
・Intelligent collaborative robot（in factory）
・Housekeeper robot（in-house）
・Secretary robot（in the office）
・etc
11Co-evolutionary pilot system development and evaluation•

Coexistence/collaborative robot development••••

System configuration: Hands (arm robot), feet (AMR), eyes (camera), ears & mouth (AIconcierge app)

Humans and robots share tasks in a common space
Each behavior evolves in response to mutual changes, such as improvements inhuman skills and robot performance.
＜Real Target＞
By building a similar system (including a human model) in virtual space,flexible operation verification is possible.

12Positioning of co-evolution necessary for future society
GOAL(Issues we want tosolve for futuresociety)Problem solving
concept

Well-being

Productivity

Human-Machine
Co-Evolution①Measures topromote coevolution
Human-Machine
Methodology(How to appropriatelyCo-Evolution
realize co-evolution)system life cycle

SafetyAddressing risks
unresolved by existing
guides for AI systems

New risksarising due toco-evolution②Measures toreduce new risks

13

Table of contents of HMCES Guidebook
1. About This Guidebook

3. Co-evolution System Life Cycle

1.1. Positioning and Intended Readers of This Guidebook
1.2. Future Society of Co-evolution between Humans and Machines
1.2.1. Challenges in Modern Society and Vision for an Improved Future Society
1.2.2. Definition of Co-evolution between Humans and Machines
1.3. Necessity of Guides for Co-evolution Systems
1.3.1. Unresolved Issues in Existing AI System Guidelines
1.3.2. Summary of Unresolved Issues and Existing Practices
1.3.3. Role of This Guidebook
1.4. Scope of Application
1.5. Glossary
1.6. References

3.1. Overview of the Life Cycle
3.2. [A] Inception
3.2.1. Recommendations from the Perspective of Co-evolution
Systems
3.2.2. Requirements for Machine (AI System) Developers
3.2.3. Requirements for Machine (AI System) Users
3.3. [B] Design and Development
3.4. [C] Verification and Validation
3.5. [D] Deployment
3.6. [E] Operation and Monitoring
3.7. [F] Re-evaluate
3.8. [G] Retirement
3.9. [P] Risk Management and Governance3.10. [Q] Security and Privacy
3.11. [R] Accountability, Transparency, and Explainability
3.12. [S] Safety, Resilience, Robustness, and Fairness
3.13. [T] Others

2. Co-evolution Systems between Humans and Machines

2.1. Co-evolution System Configuration
2.2. Co-evolution Levels
2.3. Details of Unresolved Issues in Co-evolution Systems
2.3.1. Unresolved Issue 1: Quality and Safety Assurance Methods for Continually Updated Systems
2.3.2. Unresolved Issue 2: Quality and Safety Assurance Methods for Unknown Systems
2.3.3. Unresolved Issue 3: Difficulty in Managing Operations for Continuously Evolving Systems
2.3.4. Unresolved Issue 4: Difficulty in Managing Operations for Unknown Systems
2.4. Major Corresponding Actions for Co-evolution Systems
2.5. Co-evolution Framework

14

Summary of HMCES Guidebook (1/2)
Defining Co-evolving Systems and Stakeholders• Definition of the co-evolving system life cycle (based on Figure.3 of ISO/IEC
22989:2022)••

Organize the requirements for human-machine co-evolution for each phase<Concept of Co-evolution system><Life cycle>

15Summary of HMCES Guidebook (2/2)

162 main roles of virtual space•

Monitoring the real space••••

events and changes that are actually occurring in the field
Known-known events: sufficiently predictable
Known-unknown events: not easy but are somewhat predictable

Simulation for unknown conditions•••

providing an opportunity to take countermeasures in advance
Unknown-known events: not clearly known at this time but could
occur in the future
Unknown-unknown events: not expected at all17

SafeComp2023 presentation and poster• Presented the Co-Evolution Guidebook at an international conference(SafeComp2023, September 2023) with a presentation and poster booth andreceived some feedback comments from international experts.
• Toward Human-centered AI Framework: An Introduction to AI2X Co-evolution
Project,• Yutaka Matsubara, Akihisa Morikawa, Daichi Mizuguchi and Kiyoshi Fujiwara• https://safecomp2023.cnrs.fr/position-papers/

18

Indicators related to human-machine cooperation• Evaluation indicators are essential to promote, analyze, and improve the co-evolutionary system＜Base indicators＞➢ HMT: Common Metrics to Benchmark Human-Machine Teams (HMT): A Review(https://ieeexplore.ieee.org/document/8404030)➢ Resilience engineering: Resilience Engineering Indicators and Safety Management: A SystematicReview (https://www.sciencedirect.com/science/article/pii/S2093791120302663)

138 indicators

Human/Machine measurement

applicability

19

Development of verification method for Co-Evolution systems•

Technical study conducted with European safety experts in September 2023•

Examining methods for verifying the safety of ever-changing and unknown
systems using specific applications.

•

Acquisition technology•

Tips on modeling, analysis, and evaluation methods for analyzing and evaluating CoEvolution systems•

Tips for Formal Verification of Safety of Co-Evolution systems

20

On-going tasks•

Brush up on the Guidebook•••Technology building••••Reflecting comments from various experts
Feedback from the demonstration experiment (Theme 3)

How to use HMT indicators
Analysis, verification, and evaluation methods for systems that
continue to change or behave in unknown ways
Human models on virtual space

Publication of guidebooks•

Contribution to standardization activities

21

THANKS FOR LISTENING!
Akihisa Morikawa
Director
Imaginary Corporation+81 50 5211 5282
morikawa@imaginary-inc.jp
www.imaginary-inc.jp