14:00

Data Spaces
Symposium

AI Testing and Data Spaces
Petra Dalunde, Kateryna Mischenko,
Nishat Mowla, Giovanni Leoni, Daniel Sáez-Domingo & Yunus BulutDATA SPACE SYMPOSIUM DARMSTADT 2024

AI Testing &Data spaces
Petra Dalunde – Coordinator AI Testing at RISE

RISE RESEARCH INSTITUTES OF SWEDEN

•

RTO

•

3 TEFs•Test & Demo•

Notified body•i-space in BDVAEDIHs – TEFs – Cyber Security – Industry - Medicine

AI Testing in this session”Testing AI systems towards
legislation and function for humancentric AI in Europe”

Todays session
Exploring AI Testing: Introductionand Methodology
Katya Mishchenko & Nishat Mowlat atRISE
AI Governance in organizations:
Data, Process & People to make it
happen in practice
Giovanni Leoni, at Credo AI
i-Spaces and Sandbox
Daniel Sáez Domingo at ITI
AI testing in practice: WhatValidator has learnt from its
customers
Yunus Emrah Bulut at Validator
Panel & Q&A

Exploring AI Testing:Introduction andMethodology
March 12th, 2024
Kateryna Mishchenko & Nishat I MowlaRISE

Agenda:• Introduction to AI testing:o What, why and how?
o About AI Act and standards• Challenges in AI testing• AI testing methodology
o Assessment list of Trustworthy AIo AI Testing standards
o Application domains and subfields of AI• Performing AI testingo AI testing at RISE
RISE — Research Institutes of Sweden

Introduction to AI testing

RISE — Research Institutes of Sweden
8About AI testing
What:

Testing AI systems is s a vital part of thedevelopment and deployment of AI systems
since ensures their accuracy, reliability,safety, efficiency and effectivenessWhy:

AI testing builds trust and confidence inreal-world applications and helps inidentifying and rectifying potential
issues early, thereby improving the qualityof software releases.

How:

One instrument to emphasize theimportance and ensure the safety andreliability of AI systems is the AI Act(enters into force 2024-2026).

9It lays down harmonized rules on AI,
aiming to balance the socio-economic
benefits and potential risks of AI
technologies placed on the Europeanmarket.

RISE — Research Institutes of Sweden

https://eur-lex.europa.eu/legalcontent/EN/TXT/?uri=celex%3A52021PC0206

AI Act: risk-based approach

Source: https://www.iasonltd.com/doc/jit/2021/
European_Commission_Regulation_on_AI.pdf

10RISE — Research Institutes of Sweden

About the Standards related to AI Act•

ISO/IEC TR 29119-11:2020 Software and systems engineering — Software testing — Part 11:
Guidelines on the testing of AI-based systems
Provides an introduction to AI-based systems, new challenges and opportunities for testing them.
This document explains those characteristics which are specific to AI-based systems and explains thecorresponding difficulties of specifying the acceptance criteria for such systems.

•ISO/IEC AWI TS 29119-11 Software and systems engineering — Software testing — Part 11:
Testing of AI systems
Describes testing techniques applicable for AI systems in the context of the AI system life cycle modelstages
Shows how AI and ML assessment metrics can be used in the context of those testing techniques. It also
maps testing processes to the verification and validation stages in the AI system life cycle.

•ISO/IEC 25059 Software engineering. Systems and software Quality Requirements andEvaluation (SQuaRE)
Outlines a quality model for AI systems and provide guidelines for measuring and evaluating the quality ofAI systems, focusing on characteristics like accuracy, interpretability, robustness, fairness, privacy, andsecurity.

11RISE — Research Institutes of Sweden

Challenges in AI Testing

12

RISE — Research Institutes of Sweden

Some challenges related to AI testing​​• Testing AI systems comes with unique challenges, such as the unpredictability of AI behaviour, the difficulty
in defining the right metrics for success, and the complexity of creating diverse and representative test
cases.

• ISO/IEC AWI TS 29119-11 "Software and systems engineering — Software testing — Part 11: Testing of AIsystems" describes testing techniques and metrics for AI systems in the context of the AI system life cycle
model stages. According to it, some of challenges are:Data testing:

Explainability:

Continuous Learning:issues with data quality,
diversity,
privacy, labeling,
temporal sequencing,data drift, and potential
biases.

Arises from “black box”,
nature, making it
difficult to understand
why they make certain
decisions.

often learn and adapt
over time, whichmeans they need to becontinuously tested andmonitored

Transparency:

Trustworthiness:arises “black box” nature, sensitivity of
training data, dynamic learning, potential
for bias, and the trade-off between model
accuracy and explainability.

arises from the “black box” nature, the need for
security against manipulation, the requirementfor data privacy, the necessity for accountability,
and the complexity of ensuring fairness and nondiscrimination.

13

RISE — Research Institutes of SwedenAI testing methodology

14

RISE — Research Institutes of Sweden

The AI Testing Elephant

15

RISE — Research Institutes of Sweden

Assessment list of Trustworthy AI (ALTAI)RISE — Research Institutes of Sweden

Ethics guidelines for Trustworthy AI: https://digital-strategy.ec.europa.eu/en/library/ethicsguidelines-trustworthy-ai
Image source: https://www.sciencedirect.com/science/article/pii/S1566253523002129

AI Testing Standards
Classific
ation
andevaluati
onAI
Softwarequality

Security,trustworthiness,
privacy

ISO/IEC
29119
seriesISO/IEC
23053

ISO/IEC
4213

ISO/IEC
25059

SafetyDataquality& biasRobustness
andreliability

Ethicalandsocietal
concernsManagement &Lifecycl
eRisk
management

ISO/IEC
22989

ISO/IEC
22989

ISO/IEC
5259

ISO/IEC
27001

ISO/IEC24368

ISO/IEC
42001

ISO/IEC
23894

ISO/IEC
24028

ISO/IEC
20547

ISO/IEC
5469

ISO/IEC
24027

ISO/IEC
24029

ISO/IEC
42006

ISO/IEC
31000

ISO/IEC
25000

ISO/IEC
24028

ISO/IEC
38507

ISO/IEC
42102

Functional

17

RISE — Research Institutes of Sweden

ISO/IEC
5338

Non-functional

Application domains and subfields of AI
Manufactur
ing

Healthcare
Subfields of AI:
1. Machine learning
2. Deep learning (DNN)
3. Natural language processing(LLM)
4. Computer vision (image, video,voice)
5. Reinforcement learning (agents)
6. Multi-agent systems
7. Robotics (autonomous)
8. Expert systems (reasoning)
9. Speech processing (speechrecognition)
10. Planning and scheduling (planactions)
11. Knowledge representation andreasoning
12. Evolutionary computing(genetic algorithm)
13. Affective computing (recognizefeelings)
18

RISE — Research Institutes of Sweden

Finance

R&D

Transporta
tion

Environment

Customer
service

EnergyLegal &Law

Education

Entertainm
ent

Cybersecur
ityAgriculture

Robotics

RetailPerforming AI Testing

19

RISE — Research Institutes of Sweden

Performing AI Explainability Testing

Hampus Lundberg, Nishat I Mowla, Sarder Fakhrul Abedin, Kyi Thar, Aamir Mahmood, Mikael Gidlund,Shahid Raza, “Experimental Analysis of Trustworthy In-Vehicle Intrusion Detection System Using
eXplainable Artificial Intelligence (XAI),” IEEE Access, vol. 10, September 2022. (Link)
20

RISE — Research Institutes of Sweden

Performing AI Explainability Testing

Microsoft Research
21

RISE — Research Institutes of Sweden

Quality of AI
Quality AI requires quality data
But quality AI is more than data

22

RISE — Research Institutes of Sweden•

Cybersecurity•

Transparency•

Robustness•moreThanks!
Katya picture

23

RISE — Research Institutes of Swedenkateryna.mishchenko@ri.se
nishat.mowla@ri.sei-SPACES ANDSANDBOXESITI EXPERIENCEIntelligence and Technology Transfer Director in ITIDaniel Sáez-Domingo •• Strategic
Coordinator of ITI Innovation Space (i-Space Platinum BDVA)(dsaez@iti.es)• Member of Board of Directors of BDVA and GAIA-X

Data Driven economy needs

Tech. Convergence

Awareness

Experimentation
Regulations &Standards

i-Spaces, making Data Strategy
happeni-Spaces concept was defined a long time ago(2014), but it is totally alive and needed
nowadays. i-Spaces are ecosystems withpowerful infrastructures, knowledge,tools, data, … ready to provide services forthe experimentation and innovation with
Data and AI.

They have a close contact and are wellknown and recognized in their local
ecosystems and are also very wellconnected globally, as stars of animpressive constellation around Data andAI.

EU AI Act

linked to the Data
Governance Act ,The New Legislative Framework (NLF) legislation (e.g. machinery, medicaldevices, toys), will integrate a conformity assessment procedures for AI risk
classification

Sandboxes
A Sandbox is an isolated framework toallow innovators, whether start-ups orlarge firms, to conduct live experiments
in a controlled environment
Tech. Challenges
Legal Challenges

Business Challenges

Ethics andRegulatory
Challenges

WHY SANDBOXES: EXAMPLE SPAIN

i-Spaces: Trusted environments to experiment• Enriched node providing:• Data Quality• Data Governance• Data Ingestión
Knowledge for onboarding in thedigitization andextraction of value
from data

Model, tools
and knowledge
to deploy data& Ai stacks

Own
infrastructure
and tools tohostexperiments• Data Storage• Data Sharing• Data Processing• Access control and security• A Data Innovation Lab

Access to personalized resources•Infrastructures•Datasets•ToolsAccess to services•Data Processing•Data Management•Business•Legal•EthicsSkills• Technical• Infrastructure deployment• Data governance• BusinessHow can i-Spaces help the regulators
SCENARIO
1• Infra• Services• Business

SCENARIO
2SCENARIO
3• Infra• Services• Business• Infra• Services• Business

AGILE AND PERSONALIZED SCENARIO CONFIGURATION
ON DEMAND

How can i-Spaces help the regulators

Analyse the AI
system, its
intended use,
potential risks,and mitigation
strategies.

Prepare the scenario
to test and deploy
the AI systems inreal-world settings,
but in controlledenvironment. Limitedtime.

Report to the regulator
on the performance,risks, and anyincidents related to theAI systems

And i-Spaces are sharing best practices

i-SPACES HAVE FEDERATED THEMSELVESWITH THE VISION OF BEING THE
REFERENCE FOR EXPERIMENTATION ANDINNOVATION WITH INDUSTRIAL, PUBLIC
AND PERSONAL DATA AND AITECHNOLOGIES FOLLOWING THE
EUROPEAN, NATIONAL AND REGIONALVALUES AND PRINCIPLES. SOLIDCOLLABORATION. COMMON VISION.
TANGIBLE IMPLEMENTATION

4 main pillars

304260k€ in cash70k€ in-kind

247 services
230 datasets
80 courses

Mission
The Federation’s mission is to accelerate theevolution and adoption of Data driven
innovation and AI Technologies in Europe by
facilitating a safe, trustworthy and regulatorycompliant environment for cross-border andcross-sector data-driven experimentation. The
Federation links relevant European initiatives on
Data and AI in a single ecosystem providing asustainable high-quality and global European
federated catalogue of data sources, datadriven services, courses and solutions deployed
locally by the i-Spaces.

AI testing in practice
What Validaitor has learnt from its customers?
Yunus Bulut , Founder & CEO
yunus.bulut@validaitor.com

Safety & Trust for AI

Validaitor Platform
Assessments & Collaboration
Validaitor offers many AI assessment
and risk management templates andenables collaboration between AI
developers and auditors.

Automated Certification

Out-of-the-box Testing
Validaitor provides many tests andmetrics out-of-the-box. You don’t haveto write knowledge intensive testingcode for AI audits.

52

Safety & Trust for AI

Validaitor automates testing andassessment so that companies
can get AI certification from theplatform every time they ship anew AI.

Forensic and Incidents
Validaitor keeps your AI assets andaudits for forensic purposes. It alsoenables you to keep track of incidents
and provides collaboration functionality
on incident management.

MoGanit
theorrthetheTraPumo
datint indel Te atheus
stmoe
the delmodel

AI Lifecyle
Requirements:

Human in the loop

Risk management system

Data Governance

Record keeping

Data and data governanceTechnical documentation

Risk management

Transparency

Record-keeping
Transparency and provision of information to usersTesting

Human oversight

Documentation

53

Safety & Trust for AI

Accuracy, robustness and cybersecurity

1. AI Testing is beyond performance testing• AI developers are familiar withperformance testing:• The educational system enforcesit.
• It is easy to understand andimplement.
• There’s more than performance inan AI!
54

Safety & Trust for AI

Security• Evasion Attacks: Models can befooled during inference time.
• Poisoning Attacks: Models can befed with backdoors that can betriggered during inference.
• Model Stealing Attacks: Intellectualproperty is at risk.

55

Safety & Trust for AI

Privacy• It’s shown that large models tend tomemorize training data.
• The larger the models, the higher thechances to memorize sensitive info.
• Models can leak this info during
inference:- Membership Inference Attacks• The only known solution that iseffective is Differential Privacy.
- It’s really hard to scale!
56

Safety & Trust for AI

Fairness• Bias is a central concern that is directlyrelated with human rights• Categories can be age, gender,nationality, religion …
• Bias can be checked on:• Datasets• Model predictions57

Safety & Trust for AI

2. There‘s a trust issue between AI and its users• The trust issue causes
underutilization of AI.
• AI developers care about external
certification even without any
regulatory purposes.

58

AI developers are unsure
about the quality of theirmodels and shy away frominnovation e.g. AI
applications in healthcare

Safety & Trust for AI

Consumers arehesitant to interact
with AI based products
and services e.g.
autonomous driving.

3. Testing should result in „better“ AI• AI developers expect actionableinsights after comprehensive AItesting.
• The aim of testing should be to
discover weak spots and come up
with suggestions to act upon.

59

Safety & Trust for AI

4. AI failure modes are subtle• It’s not obvious to understand thefailure modes of AI.
• The larger the models, the harder tofind out weak spots.

60

Safety & Trust for AI

5. Better AI means a lot of trade-offs
•
•
•
•

61

Security vs performance
Bias vs performance
Robustness vs performance
Privacy vs performance

Safety & Trust for AI

6. Pre-trained models spread the vulnerabilities• The problems cascades with finetuning.
• The direction of AI development ismore and more fine-tuning of pretrained large models.

62

Safety & Trust for AI

7. General purpose AI is hard to test
• The larger the model the harder totest and figure out new insights.
• If a model is intended to be “generalpurpose”, defining scope for testingis the only way forward.

63

Safety & Trust for AI

8. Testing is use case specific• The tests are meaningful if they’rejustified in a use case scope.
• Different thresholds apply fordifferent use cases.

64

Healthcare

AutomotiveManufacturingFinance&Insurance

Safety & Trust for AI

9. Testing requires quantifiable metrics• The communication can only bedone using quantifiable metrics.
• The lack of metrics means no tests
most of the time.

65

Safety & Trust for AI

10. Transparency is the goal of compliance testing• There’s no perfect testing that
discovers all the vulnerabilities.
• The goal of a regulation should be toenforce transparency and best
effort.

66

Safety & Trust for AI

THANK YOU!
Yunus Bulut , Founder & CEO
yunus.bulut@validaitor.com

Safety & Trust for AI