Elevating Data Quality: A Paradigm
Shift for Data Spaces and AI Needs

12/03/2024 12:00–13:00 (CET)
Daniel Alonso, Håkan Burden, Susanne Stenberg,Mark Dietrich, Kuldar Aas, Tobias Guggenberger

Objectives of the session• To approach data quality from a new perspective,
addressing specific requirements coming fromnew applications and in view of AI Act and otherregulations• To consider data quality as a dynamic feature,different metrics to respond to final application
and use cases needs (fit for purpose)• Data spaces to support data quality, as acontrolled and trusted environment that ensures
governance, provenance, traceability,description, business needs, …

Agenda of the session
Speaker
Daniel Alonso

Intro and setting-up the scene

Hakan Burden / SusanneStenberg

Data quality from a policy perspective – from law to standard. Concrete
examples from the AI Act, EHDS and emerging standards. Examples ofpolicies related to stakeholders involved in the assessment of dataquality. Implications for policy development .

Mark Dietrich

Quality from data product paradigm. Importance of semantics. View
from research, EOSC, GREAT.

Kuldar Ass

Estonian data infrastructure. New layers for secure data sharing. Fit forpurposes, use cases

Tobias Guggenberger

Data quality from a business perspective. Impact of data quality oncosts, resources and KPIs

AllQ&ABDVA discussion paper (to be published by the end ofMarch)
AI Act andotherregulations

Data
provenance

DatasetData quality
characteristics,
properties,
metrics

Data quality
ensurance​

Data quality
documentation

Data quality
description​

Trustworthiness

Use case, AIand otherapplications

Specific
purposes / usecases

Data Space

New specificcharacteristics,
properties,
metrics

ResearchPublicadministrationIndustry

Hakan Burden / SusanneStenberg
RISEA policy perspective
on data qualityHåkan Burden & Susanne Stenberg

What signifies dataquality?
And who should care?

By Göteborgs konstmuseum (Gothenburg Museum of Art). Photo: Hossein Sehatlou,
CC BY 4.0, https://commons.wikimedia.org/w/index.php?curid=112971129

Fit for purpose: Data quality
implies human judgement
European Health Data Space:

AI Act:”‘data quality’ means the degree
to which the elements ofelectronic health data areassessed and considered suitable
for their intended primary andsecondary use”

”Training, validation and testingdata sets [for high risk AI systems]
shall be relevant, sufficiently
representative, and to the best
extent possible, free of errors andcomplete in view of theintended purpose”

Fit for purpose?
Standards help us
take the right decisions

By Göteborgs konstmuseum (Gothenburg Museum of Art). Photo: Hossein Sehatlou,
CC BY 4.0, https://commons.wikimedia.org/w/index.php?curid=112971211

Data quality: ISO 8000-150:2022
Syntactic quality
Well-formed, right type, ...
Semantic qualityInterpretation of labels and values
Pragmatic quality
Is the data fit for purpose?

job_application = applicant:”JaneDoe”, ”123456”, job:”courseassistant”, cv:””}
job_application = {applicant:”JaneDoe”, date:010203, job:”teacher”,cv:”cv.pdf”}
job_application = {applicant:”JaneDoe”, date:010203, job:”chef”,cv:”cv.pdf”}

Data quality: ISO 5259-2 for ML

AI Act: data used for training, testing and validation should be ”relevant [...] andcomplete”

Completeness:

Relevance:

Ratio of data items of no presenceof null data values in a dataset

Ratio of features or records in thedataset that are relevant to the given
context

Ratio of unlabelled orincompletely labelled samples in adataset
job_application = {applicant:”JohnDoe”, date:010203, job:”chef”,cv:”cv.pdf”}
 Syntactic quality Pragmatic quality
 Is ”chef” fit for purpose?
Requires the applicant’s intention,
which might not be inferred byprovenance nor the metamodelData quality
according to whom?
Points of view according to the AI
Act:
As a provider in your market
assessment
As an AI competent authority in theregulatory sandbox
As a notified body in the compliance
procedure
As a deployer fulfilling user obligations
By Göteborgs konstmuseum (Gothenburg Museum of Art). Photo:
Hossein Sehatlou,
CC BY 4.0, https://commons.wikimedia.org/w/index.php?
curid=112974394

As an AI competent authority
conducting market surveillanceOur conclusions
Data quality is assessed on a case-by-casebasis and implies human judgement
There are currently gaps in when data
quality is a legal requirement, but also how
it should be applied across domains andapplications
There are opportunities to define theimplementation of data quality (i.e.
relevant threshold levels – quality metrics)
and applications (what is good enough formy market?)This opens for differing governanceand

By Göteborgs konstmuseum (Gothenburg Museum of Art). Photo:
Hossein Sehatlou, CC BY 4.0,
https://commons.wikimedia.org/w/index.php?curid=110310163

Thanks!
hakan.burden@ri.se
susanne.stenberg@ri.se
All pictures taken fromhttps://commons.wikimedia.org/wiki/Category:Media_contributed_by_Göteborgs_konstmuseumMark Dietrich
EGIElevating Data Quality:
A Paradigm Shift for Data Spaces and AI Needs
DataWeek 2024
Darmstadt, Germany

Mark Dietrich, Senior Advisor, EGI.eu

March 2024 | DataWeek 202418

www.egi.eu |Elevating Data Quality

Is Quality In theEye of the Beholder?
EGI.eu 2024
19

Breaking Quality Down
Some Measures of Quality

Quality of Single Data Points• Declared Precision, Accuracy• Declared Data Collection/Processing Standards
Dataset Statistics• Missing, outdated data
• Coding errors• Inter-/Intra Dataset Consistency• Apparent Bias
• Representativeness• BalancedMetadata• Incomplete, erroneous• Vocabulary errors• FAIR metrics

March 2024 | DataWeek 2024

“Subjective”(wrt Use Case)

In EU, only ~30% of data
is FAIRwww.egi.eu | 20

Data Curation/QC Processes
Sample: European Strategy Forum on Research Infrastructures (ESFRI), 2021 Roadmap

63 ESFRI
Infrastructures

March 2024 | DataWeek 2024

23 Provide
Data

10 Publish
Curation &Quality
ControlProcesses

www.egi.eu | 21

What Does “Fit for Purpose” Mean?

Research, PublicSector
Purpose:● Advancement ofKnowledge● Scientific Method● Reproducibility

Quality required depends on
specific study● Mismatches caught inpeer review

March 2024 | DataWeek 2024

Sensitive Data

Business, Industry

Purpose: must be declared
every time you want toaccess sensitive data.

Purpose motivates the firstuse case● Data sovereignty
enables later use cases

Fit for Purpose● Ethics Review● Original reason for datacollection must becompatible withproposed use● Even if compatible,
quality may not be
sufficientFit for Purpose?:● No review: ends justifythe means● Competitive advantage● Bias, Opacity● Disinformation,collusion?
www.egi.eu | 22

How Can We Operate at Scale?
Data Space
Data ProductUnknownUsers

Source: Adam Bellemare, “Practical Data Mesh: Building Decentralized Data Architectures with EventStreams”, 2022 Confluent Inc.

March 2024 | DataWeek 2024

www.egi.eu | 23

Conclusion
Quality CAN be defined and measured along multiple dimensions• For individual data points• Across and between datasets• For the metadata that describes the dataset
But these dimensions are independent of each other (not necessarily correlated)
So Quality LABELS (high, medium, low) CANNOT be universally defined.
• But labelling schemes might be set up for particular use cases

March 2024 | DataWeek 2024

www.egi.eu | 24

Thank you!
Contact: mark.dietrich@egi.eu
About Mark
•
•
•
•
•
•

Data Spaces Support Centre (DSSC)
EOSC-Future, EOSC-Focus, EOSC EU Node Implementation
GREAT (preparatory action for the Green Deal Data Space)
HealthyCloud.eu (Five Safes)
EGI-ACEHCloud.eu (Horizon Cloud)•••

EUCloudEdgeIoT.eu
Gaia-X
Supercomputing, HPC (green computing, economic models)•First analysed The European Data Strategy, June 2020

Icons created by Dinosoftlabs, Eucalyp, Freepik, Icon Mania, Inkubators, Parzival’ 1997:
https://www.flaticon.com/free-icons/code

Fundingprovided by:
GREAT: Green Deal Data

Space: EC grant agreement
101083927

March 2024 | DataWeek 2024

EGI-ACE: EC grant
agreement 101017567

EOSC Focus: EC grant
agreement 101058432

EOSC Future: EC grant
agreement 101017536

HealthyCloud: EC grant
agreement 965345

www.egi.eu | 25

EGI Support for Data SpacesTechnological Services• EGI provides several services and technology solutions for the development and implementation of Data
Spaces. These solutions, for example, enable the sharing and collaboration of data in a secure and trustedenvironment.
Policy Development Support• EGI’s policy development support activities help to ensure that data spaces are established and managed in away that promotes trust, transparency, and accountability.
Data Space Landscape Harmonisation• EGI helps to create synergies between different projects and initiatives, identifying common goals, aligning
activities and standards, avoiding duplication of effort and maximizing the impact of funding and resources,
thus harmonising the landscape of different Data Space related initiatives.
Contributions to Projects• EGI leads or contributes to several projects aimed at developing thematic data spaces or contributing to thedevelopment of overarching, multidisciplinary data spaces.

March 2024 | DataWeek 2024

www.egi.eu | 26

Contributions to Data Space Projects
IMPLEMENTATIONRESEARCH SECTOR

IMPLEMENTATIONCOMMERCIAL SECTOR

Data space among Digital
Innovation Hubs, serving
industry

Technologies for data
monetising and data
economySupport for climate (ENES),marine (SeaDataNet),
biodiversity (GBIF), marine(EMSO), radioastronomy(LOFAR)Big Data for
Next Generation
eneRGyImprove privacy and IP
protection for citizens

FinancialMobility

Public Administration
March 2024 | DataWeek 2024

Agriculture

EUCAIM

Manufacturing
Skills

E-science centre withdatasets and applications forionosphere, thermosphereand plasmasphere

Preparatory action forGreen Deal Data Space

EUropean Federation
for CAncer IMages

EUreka3D

Cultural Heritage DataSpaceSECTORAL DATA SPACES

HealthMedia
Language

EnergyTourism
www.egi.eu | 27

Kuldar Ass
Ministry of Economic Affairs andCommunications (Estonia)

Practical issues in data reuse• Low quality of data descriptions

Can I find relevant data?

• Data quality unknown Is the data actually useful for me?
• Data service quality undefinedWill I get the data when I need it?

• Limited responsiveness to issues

Will my problems get solved in atimely manner?

• Limited knowledge on DP

Will I go to prison for breaching GDPR?

Institutional context

Gather or
createdataData
steward

DQ
implementation

Set andmanage
data quality
rules

DQ requirements

Pre-definedinternal usecases

DQ requirements
Data sharing context…

DDDMusecase

Dashboarduse
case

Dataanalytics use
case

AI usecase

Way forward• Making it easier to assess data quality• „public sector agencies to establish formal data quality procedures“ (01.2024)• „public sector agencies to publish their data quality procedures“ (2025?)• Making it easier to achieve data quality• Providing data quality guidelines and training (continuous)• Establishing a support centre on data quality (2022)• Setting minimal requirements for assessing the value and ensuring the quality of data assets (2025?)• Supporting agencies in acquiring / developing data quality and profiling tools (and publishing according data quality
reports) (2025)• Last but not least• Let’s join the DSSC SSF and learn from others! (December 2023)

Tobias Guggenberger
Fraunhofer ISST

Legislation and BusinessLevels of Data Quality

Thank you!