H2020-EUJ-02-2016
H2020 Grant Agreement Number 723076
NICT Management Number 18302

Deliverable D6.3

Holistic Data Management:
Initial Concepts and Models
Version V1.2
January 31st, 2018
ABSTRACTThis document contains the initial concepts and models developed in the projectthat serve as a basis to provide a holistic data management view onheterogeneous data. On one hand holistic data management includes providing
a unified view on the variety of data ranging from IoT sensor data, over social
media data to open governmental data. On the other hand, this documents
particularly covers the challenge of managing personal data across distributed
City IoT Platforms such as CPaaS.io. We outline our approach for addressing thesemantic interoperability problem in particular with respect to personal data
managed by technologically diverse platform instances in different regions of theworld. Our focus is in particular on managing and exchanging personal data
between the European FIWARE-based CPaaS.io instance and the Japanese U2based instance.

This work is licensed under the Creative Commons Attribution 4.0 International License.
To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

Disclaimer
This document has been produced in the context of the CPaaS.io project which is jointly funded by theEuropean Commission (grant agreement n° 723076) and NICT from Japan (management number 18302).
All information provided in this document is provided "as is" and no guarantee or warranty is given thatthe information is fit for any particular purpose. The user thereof uses the information at its sole risk andliability. For the avoidance of all doubts, the European Commission and NICT have no liability in respectof this document, which is merely representing the view of the project consortium. This document issubject to change without notice.

Document Information
Editors
Authors
Reviewers
Delivery Type
Dissemination
Level
Contractual
Delivery Date
ActualDelivery Date
Keywords

H2020 EUJ-02-2016

Martin Strohbach (AGT), Katsunori Shindo (YRP)
Martin Strohbach (AGT), Adrian Gschwend (BFH), Eduard Klein (BFH),
Katsunori Shindo (YRP), Jan Frecè (BFH)
Stefan Gessler (NEC), Satoshi Asano (UCT)
ReportPublic31/01/2018
31/01/2018
Semantic models, ontologies, semantic integration layer, data quality,
linked data, heterogeneous data, privacy

CPaaS.io

Page 2 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

Revision History
Rev.
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8

Date
17/11/2017
08/12/2017
13/12/2017
15/12/2017
20/12/2017
21/12/2017
03/01/2018
14/01/2018

0.9
0.10
0.11
0.12
1.0
1.1
1.2

16/01/2018
22/01/2018
23/01/2018
23/01/2018
25/01/2018
29/01/2018
30/01/2018

Description
Initial TOC
Section 4: Problem description and design optionsFirst complete draft for section 4.2.2
First draft for sections 6 and 7
Finalized section 4.2. Initial input for section 4.3
First Draft of section 4 and section 5
Section 6 complete (draft status)
Appended documents about u2 architecture in section
4.1.1, 4.2.2, and 4.3.2.1
Correct section numbers
Corrections section 6 and completion section 7
Added abstract and introduction section
Full pass over the deliverable, added summary
Additions to section 2, wording
Release for Review
Review
Addressed review comments

Contributors
Martin Strohbach (AGT)
Martin Strohbach (AGT)
Martin Strohbach (AGT)Jan Frecè (BFH)
Martin Strohbach (AGT)
Martin Strohbach (AGT)
Katsunori SHINDO (YRP)Jan Frecè (BFH)
Martin Strohbach (AGT)
Martin Strohbach (AGT)
Adrian Gschwend (BFH)
Martin Strohbach (AGT)
Stefan Gessler (NECLE)
Martin Strohbach (AGT)

List of Acronyms
Acronym
DCAT
ICAT
IoT
LOD
M2M
NGSI
OBDA
OGD
OWL
RDF
SDR
SPARQL
SSN

Definition
Data Catalog Vocabulary
IoT Catalog
Internet of Things
Linked Open Data
Machine to Machine
Next Generation Service Interface
Ontology Based Data Access
Open Governmental Data
Web Ontology Language
Resource Description Framework
Semantic Data Repository
SPARQL Protocol and RDF Query Language
Semantic Sensor Network

H2020 EUJ-02-2016

CPaaS.io

Page 3 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

Table of Contents
1

Introduction............................................................................................................................................................................... 52

Model for assessing data quality ..................................................................................................................................... 6

3

4

5

6

2.1

Problem Description ..................................................................................................................................................... 6

2.2

Description of Challenges ........................................................................................................................................... 7

2.3

Design Options ............................................................................................................................................................... 9

2.4

Approach and Implementation .............................................................................................................................. 10

Linking Data from various sources ................................................................................................................................. 11
3.1

Problem Description ................................................................................................................................................... 12

3.2

Design Options ............................................................................................................................................................. 14

3.3

Approach and Implementation .............................................................................................................................. 17

3.4

Summary ......................................................................................................................................................................... 21

Deployment Knowledge Base .......................................................................................................................................... 22
4.1

Problem Description ................................................................................................................................................... 22

4.2

Approach and Implementation .............................................................................................................................. 23

Privacy Support Mechanisms............................................................................................................................................ 27
5.1

Problem Description ................................................................................................................................................... 27

5.2

Relation to Requirements ......................................................................................................................................... 28

5.3

Design Options ............................................................................................................................................................. 28

5.4

Approach and Implementation .............................................................................................................................. 30

User empowered master data management .............................................................................................................. 33
6.1

Problem Description ................................................................................................................................................... 33

6.2

Design Options ............................................................................................................................................................. 33

6.3

Approach and Implementation .............................................................................................................................. 33

7

Summary................................................................................................................................................................................... 34

8

References................................................................................................................................................................................ 35

H2020 EUJ-02-2016

CPaaS.io

Page 4 of 35

Deliverable D6.3

1

Holistic Data Management: Initial Concepts and Models

Introduction

This devlierable provides a first version on the initial concepts and models for a holistic data management
approach. We specifically address two of the main CPaaS.io project objectives: (1) managing personal dataand providing full control to citizens about their data, and (2) exchanging data between platforms
developed, deployed and operated in different world regions (here EU and Japan). Achieving theseobjectives requires a range of activities that look at the data and the platforms from a data management
and interoperability perspective. But it also requires to take the perspective on addressing legal and privacy
related topics. These activities are reflected in WP6 as complementing tasks. The initial results related todata management are presented here across all the WP6. Here we provide an overview of the expectedtask output with respect to this deliverable:•

Task 6.1 Data Quality. An initial model for data quality assessment and respective ontology fordata annotation.

•

Task 6.2 Semantically Enriched Data Catalogues. Concept for linking of heterogeneous data
sources•

Task 6.3 Privacy Governance. Governance mechanism for the base privacy mechanisms•

Task 6.4 User empowered master data management. Data model and mechanisms for managing
personal data by users•

Task 6.5 Deplyoment Knowledge Base. Semantic sensor deployment ontology, theimplementation of a deployment knowledge based providing a SPARQL endpoint

This deliverable is roughly structured according to the output of the tasks. Section 2 describes a model forassessing data quality, section 3 describes our approach for linking data from various sources, section 4
describes the implementation of the deployment knowledge base, section 5 describes our privacy support
mechanisms, section 6 describes our model and mechanisms for empowering users to manage theirpersonal data, and finally section 7 summarizes this deliverable. For each section we describe the problem
and associated challenges and explain the design option at hand. We conclude each section by describing
the general approach and detail the current status of the implementation work associated to the described
problem.
Note that all the sections and contributions described in this deliverable relate to the semantic integration
layer of the CPaas.io architecture as depicted in Figure 1. As the initial concepts and models are the samefor both platform instances, we do not explicitly distinguish between the European and Japanese platform
instances, but rather describe the common concepts and early implementation examples.

H2020 EUJ-02-2016

CPaaS.io

Page 5 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

Figure 1: Overview of the CPaaS.io architecture2

Model for assessing data quality

In this section CPaaS.io’s initial model for assessing data quality is described. Since sensor data isinherently unreliable, the quality of data sources is described through metadata for assessment or otheruse in higher layers of the CPaaS.io architecture or in end user applications. Since appropriate ontologies
for description still must be developed (see chapter 4), this chapter contains mostly a conceptual andgeneric view, which will be refined and implemented later during the project.

2.1

Problem Description

In Deliverable D6.1 (Fraefel et al., 2016), the state of the art of data quality wrt. Smart City solutions has
been addressed. Research on data quality assessment and design options for the CPaaS.io project will be
described in this section. Moreover, relations to platform and application requirements will be considered.
Smart City applications use data from heterogeneous sources such as IoT data, data from social networks,
crowd-sourced data and open (government) data. In this multi-modal information approach, data quality
is crucial for the quality of smart city applications (Karkouch et al., 2016). While for OGD a certain degreeof data quality could be established, IoT data such as weather, location, pollution and noise data fromthermal, environmental and geolocation sensors rely on the quality of sensor devices and on local
conditions. They are inherently unreliable regarding accuracy, timeliness and provenance. According to(Barnaghi, Bermudez-Edo and Tönjes, 2015), the quality of a data source depends on three factors: errors
H2020 EUJ-02-2016

CPaaS.io

Page 6 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

in measurement/precision, noise in the environment and data communication/processing, andspatial/temporal granularity of observations.
Since poor data quality limits the acceptance of (smart city) applications in general, according to (Zaveri etal., 2014), there are many methodologies and frameworks available for data quality assessment to enablesolid decisions about the appropriate use of data. To name a few, in (Timms et al., 2011) marine sensor
data are automatically accessed based on rate of change values and node differences. (Heuel, 2015)
describes live data quality assessment of temperature, humidity and air quality sensor data within ageographic information system. In (Debattista, Lange and Auer, 2015) a framework for linked data quality
assessment is presented which is validated on large DBpedia (Bizer et al., 2009) data sets.

2.2

Description of Challenges

In general, quality assessment is based on a set of quality attributes which is delivered through the IoT
data stream, and which is processed in specific architectural layers according to specific assessment rules.
In the context of the CPaaS.io project, we see the following challenges:•

Provide an information model containing elements for data quality, based on semantic
descriptions and a unified mechanism for data access (see section 4).

•

Provide a model for assessment of data quality. Although the basic focus within the CPaaS.io
project is on quality of IoT data, the generic view is data-source-type-agnostic. Arbitrary data
sources can be considered such as data from social networks, crowd-sourced data and open
(government) data.

•

The model must be mapped to concrete architectures, specifically the European and Japaneseplatform instances.

2.2.1

Relation to Requirements

In this section, the requirements are listed that are related to the model for data quality assessment, andits relationship is discussed. This condensed list (Table 1) is derived from the current version of the Volere
Template from Deliverable 3.1 (Requirement Collection Analysis).
Table 1: Requirements related to model for data quality assessment

UNI ID

Description

Challenge

It SHOULD be possible to set-up data quality
constraints for data producers to comply with. In the
NFREQ.28
same way, it SHOULD be possible to monitor/verify
data quality

Contributes to trust and reputation. Data
quality constraints are based on to be
defined quality measures.

Any data at the platform level shall have data quality
annotations

the semantic description of data sourcesmust contain appropriate meta data

FREQ.10

H2020 EUJ-02-2016

CPaaS.io

Page 7 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

Data search and retrieval shall be achieved upon acollection of criteria that includes location (of the
FREQ.11 Physical entity concerned), characteristics of theunderlying IoT Resource reputation and the quality of
information.

the semantic description of data sourcesmust contain appropriate meta data

The CPaaS.io platform MUST provide publish /subscribe functionality and criteria for filtering in/out
FREQ.90
data based (at least) on geographical area, data typeand data source

the semantic description of data sourcesmust contain appropriate meta data

FREQ.113

The p/f SHOULD offer means for synchronizing clocks the semantic description of data sources
between IoT devices
must contain appropriate meta data

FREQ.116

It SHOULD be possible to publish data about
actuation

the semantic description of data sourcesmust contain appropriate meta data

As a prerequisite, implementations of the requirements concerning the semantic layer (Table 2) must be
available:
Table 2: Prerequisite Requirements

UNI ID

Description

Data MUST be interoperable across the different parties anddomains (data/service providers)
The architecture MUST favor the reuse of wide-spread standards
NFREQ.36
for Data and API
NFREQ.27

FREQ.13

Any data at the platform level shall be following an accepted set ofstandard schemas (ontologies) for annotation

The platform shall be able to aggregate data from various sources
using LOD principles
The p/f MUST provide mechanisms for collecting, publishing and
FREQ.110storing semantic data (historical data)
FREQ.14

Among others, the requirements in Table 3 will profit from the integration of the assessment model into
the CPaaS.io platform.
Table 3: Benefitting Requirements

UNI ID

Description

FREQ.80

CPaaS.io platform MUST be able to perform analytics, andevent detection/prediction using “live” and historical data

The CPaaS.io platform MAY provide data validation
services
The p/f SHOUD be able to collect and manage user 's
FREQ.118
feedback
FREQ.92

H2020 EUJ-02-2016

CPaaS.io

Page 8 of 35

Deliverable D6.3

2.3

Holistic Data Management: Initial Concepts and Models

Design Options

Data quality assessment is based on data and metadata attributes in a data stream and on subsequent
processing based on assessment criteria. In general, models for data quality assessment consist of thefollowing elements:a) An ontology for describing sensor (meta) data, typically an extension of the Semantic Sensor
Network (SSN) Ontology.
b) an annotation model with meta data as quality attributes e.g. for accuracy (correctness, precision,
completeness) or timeliness (age, volatility, frequency). Sets of quality attributes for semantic
annotation are described e.g. in (Kolozali et al., 2013).
c) Attribute measures such as network consumption (“10bps”) or time between measurement andpublication (“90sec”).
d) Assessment instances which evaluate the data quality as “fit for use” depending on the respective
context. These assessment instances could be established at different layers of the CPaaS.io
architecture (see Fig. 1).
Raw sensor data is put into a semantic context while entering the CPaaS platform (Data Entity
ontology based), by adding additional metadata and rules one can semantically enrich this data
(Semant. Enriched Data Entity). Applications can access this information and interpret it accordingly.

Figure 1: data quality assessment instances at different architectural layers
Data quality assessment can be performed at different stages of the data stream:• Sensor/Edge: “my temperature lost 5 degrees within 10 seconds” -> possible outlier• Computing node/Close to the Edge: “8 out of 10 sensors within 1sqkm show
derivations”(Reputation Instance)• Computing node/Central: “humans report delay of train compared to (OGD/OD) train time table”• Application/End User: Chooses appropriate data;rule-based, e.g.: if "many" crowdsourced data report delay, then use delayed data, else use time
table data

H2020 EUJ-02-2016

CPaaS.io

Page 9 of 35

Deliverable D6.3

2.4

Holistic Data Management: Initial Concepts and Models

Approach and Implementation

Our approach for validating data quality constraints is based on describing data quality attributes withexisting and, where necessary, new vocabularies and implementing validation pipelines on top of SPARQL.
For that, CPaaS will use the newly established Shapes Constraint Language (SHACL) defined in 2017 by the
W3C. With SHACL one can describe so called "shapes" of data in RDF. These shapes define constraints on
RDF data, for example which attributes are mandatory on a graph, which classes should be assigned or
how literal values can be validated. Multiple shapes can be combined to describe complex validation
schemes for any RDF data.
SHACL only defines a basic vocabulary to describe constraints on a graph. The constraints themselves are
represented in whatever vocabulary the data itself is using. This makes it extremely flexible to integrate inany kind of domain model. In the past OWL was used or more often abused for that matter, with all thenecessary complexity OWL offers.
As an open standard, SHACL is implemented by multiple vendors and organisations, some of them as
Open Source Software. It is also possible to implement SHACL on top of a SPARQL query processor, whichmakes it relatively easy to integrate into existing environments.
In 2018, CPaaS will integrate SHACL processing into its data pipelining systems for specific use cases.
Appropriate SHACL implementations will be evaluated and integrated into the use cases and data quality
constraints will be defined on top of the data. Data quality vocabularies will be defined where necessary
and published.
The implementation will be inspired by similar approaches by consortium members. An example of such adata creation and validation pipeline is a project with City of Zurich, one of the participating CPaaS cities.
In that particular project statistical data from Zurich is made available as RDF. The data creation andvalidation pipeline is implemented on top of Gitlab pipelines 1.

1 https://docs.gitlab.com/ee/ci/pipelines.html

H2020 EUJ-02-2016

CPaaS.io

Page 10 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

Figure 2: Gitlab based RDF validation pipeline for City of Zurich

Pipelines are split into jobs with a defined input and output. Through configuration, jobs are chained into
a pipeline. These pipelines can be triggered by various events and will execute each job accordingly. Oncea job fails, notifications are sent out and the pipeline is aborted. Jobs can execute in parallel as well where
it makes sense to do so.
A particular SHACL validation job might look as follows:

Figure 3: SHACL validation job

An RDF data stream is sent to the SHACL validation job component. This component is configured by/with
specific SHACL shapes. The output of the job can contain the original RDF data stream plus the validation
results in RDF. If the validation job is implemented within a SPARQL endpoint itself, the stream could
directly be fed from a named graph and the results could be written to an additional named graph.

3

Linking Data from various sources

In this section, we describe how CPaas.io enables the linking of data from various sources. The mechanisms
described in the following section are particularly important from a cross-border and cross-region
perspective as it describe the main concept of how data accessible via the European and the Japaneseplatform instance can be holistically accessed and vice versa.

H2020 EUJ-02-2016

CPaaS.io

Page 11 of 35

Deliverable D6.3

3.1

Holistic Data Management: Initial Concepts and Models

Problem Description

The main problem of combining data from various sources is related to its heterogeneity: almost each data
source has its own format and representation. More importantly however is the problem of understanding
the semantics of the data: what entities are the data related to and how do they relate to data in the sameor other data sources? Consider for example the CPaaS.io Rainproof Amsterdam use case in which waterlevels of rain buffers are provided. Let’s assume that we have two independent deployments in the sameurban area but with completely different hardware and software technology stacks. Even if the two
deployments would provide their data in JSON format, they are very likely to have a different structure.
Without additional information, it may be extremely hard, to link both data sources in the sense that they
are essentially providing the same information (water level of rain buffers) in the same urban area.
Generally, this is known as solving the problem of semantic interoperability (Bassbouss, et al. 2016).
The problem of achieving semantic interoperability is relevant for cross-border use cases and in particular
for cross-region use cases in which semantic interoperability between whole platform instances is required.
3.1.1
Description of Challenges
As the notion of Linked Data 2 3 and in particular, the use of RDF 4 as the lingua franca for linking data has
become increasingly popular both in Europe and Japan, RDF provide a common ground for achieving
interoperability between the European and Japanese platform instance. However, RDF is only a partial
solution:••••

There are hardly any Smart City or IoT platforms available that use RDF as prime data model
While RDF provides a basic data model for linking data, it does not solve the problem of defining
the semantics of individual data items (or statements or triples)
There is an increasing amount of semantic data models in the form of OWL 5 ontologies and RDF
vocabularies available, however it is still hard to apply existing semantic data models toconcrete technical system (complex to understand, missing concepts for the domain at hand,alternative and competing models)
Many data sources are not available in RDF and do not have an explicit description of theirsemantic. This holds in particular for sensor measurements and observations

We therefore have to solve the following three challenges:
1. Provide a semantic description of both RDF and non-RDF based data sources
2. Provide a unified mechanism for accessing all data known to CPaaS.io using a semantic datamodel
3. Provide guidelines on how to use and extend existing ontologies and vocabularies
For the FIWARE-based European platform, we specifically have to solve the challenges of••Defining mappings between the FIWARE NGSI data model and a semantic data model
Define semantic access components that provide accesso to data accessible via FIWARE components (IoTBroker, ContextBroker)

2 https://www.w3.org/DesignIssues/LinkedData.html
3 http://lod-cloud.net/

4 https://www.w3.org/TR/rdf11-primer/
5 https://www.w3.org/TR/owl2-primer/

H2020 EUJ-02-2016

CPaaS.io

Page 12 of 35

Deliverable D6.3

oHolistic Data Management: Initial Concepts and Models

to data available in the Japanese U2-based platform instance

Likewise for the U2-based Japanese platform, we specifically have to solve the challenges of••Defining mappings between the uCR (uCode Relation) model and a semantic data model
Define semantic access components that provide accesso to data accessible via common API
o to data available in the FIWARE platform instance

3.1.2
Relation to Requirements
In this section, we list the platform requirements related to linking data from various sources. Platform
requirements represent a synthesized list of requirements derived from the application requirements andthus drive requirements for the WP6 components. For this first version of the deliverable we focus on highpriority requirements. The requirements in Table 2 list for each relevant platform requirement the associatechallenge related to linking heterogeneous data sources. The requirements are grouped by the related
Challenge.
Table 2: General requirements related to linking data from various sources

UNI IDNFREQ.27
FREQ.3
FREQ.9
FREQ.11

FREQ.14
FREQ.100
FREQ.110
FREQ.80
NFREQ.36
FREQ.12
FREQ.13

Description
Data MUST be interoperable across the different parties anddomains (data/service providers)
The IoT system MUST enable the dynamic discovery of IoT
Resources, Virtual Entities and all services (VE Service and IoT
Service) based on respective specifications.
Object (Virtual Entities), VE- & IoT-services and resources shallbe semantically described and registered
Data search and retrieval shall be achieved upon a collection of
criteria that includes location (of the Physical entity
concerned), characteristics of the underlying IoT Resource
reputation and the quality of information.
The platform shall be able to aggregate data from various
sources using LOD principles
It MUST be able to describe data sets (content-wise andaccess-wise) in order to allow for discovery and lookup
The p/f MUST provide mechanisms for collecting, publishingand storing semantic data (historical data)
CPaaS.io platform MUST be able to perform analytics, andevent detection/prediction using “live” and historical data
The architecture MUST favour the reuse of wide-spreadstandards for Data and API
The platform shall provide a SPARQL end-point for accessing
semantic data (either annotated data or semantic descriptions)
Any data at the platform level shall be following an accepted
set of standard schemas (ontologies) for annotation

H2020 EUJ-02-2016

CPaaS.io

Challenge
GeneralSemantic Descriptions
Semantic Descriptions
Semantic Description

Semantic Descriptions
Semantic Descriptions
Semantic Descriptions
Semantic Descriptions,
Holistic Data Access
Semantic Descriptions,
Holistic Data Access
Holistic Data Access
Guidelines on using
ontologies

Page 13 of 35

Deliverable D6.3

3.2

Holistic Data Management: Initial Concepts and Models

Design Options

Here we describe the design options for providing semantic descriptions, holistic data access andmanagement and options for using ontologies.
3.2.1
Semantic Descriptions
The design options for the pure description of data sources are primarily related to••••

Basic Description Language (RDF, OWL, XML, etc.)
Reuse of existing vocabularies (e.g. SSN, DCAT 6, M2M Ontology work, …)
Modelling aspects
Method of linking

In principle any language could be used for using semantic descriptions. However, good W3C practices
suggest that linked data should be provided in some RDF serialization, it is a natural choice to use RDF
for describing RDF and non-RDF data sources alike.
In principle ontology-based service description languages such as OWL-S could be used for describing
data sources in the sense of describing a data source as a data service. However, this approach is somewhat
heavyweight for describing data sources. Similar arguments apply to the ontology work of OneM2M 7 thatis focused on describing devices and M2M services associated with these devices and requires
implementing the M2M standards. DCAT is a suitable candidate for describing data sets and can easily be
combined with PROV-O 8 adding provenance related information. However, DCAT is primarily focusing on
static data sets and data catalogues and does not have specific provisions for describing measurement
data. This aspect in contrast is covered by SSN that provide a general and comprehensive framework fordescribing sensor devices, their capabilities, deployment aspects and observations. However, it is not well
suited for describing other data than sensor observations and measurements.
The modelling aspects are primarily dictated by the requirements, which include the description of RDF
and non-RDF data sources, distinguishing between streaming data and static data sets, and support for
both sensor data and other data sources.
With respect to the method of linking we see the following options that have partly been reflected in thecurrent platform architecture:
1. Data sources provide linked data with full descriptions
2. Data sources provide “data as is”
While the first method in principle enables full semantic integration, it also comes at a potentially high costfor data providers that have to go through the task of providing a full semantic description of their data
set. On the other hand if data is provided as is, it may in principle be possible to infer the semantics of thedata, but this is a research challenge in itself and may be very unreliable.

6 https://www.dcat.org/

7 http://www.onem2m.org/

8 https://www.w3.org/TR/prov-o/

H2020 EUJ-02-2016

CPaaS.io

Page 14 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

3.2.2
Holistic Data Access and Management
In the initial requirement analysis phase we have identified that we need a SPARQL endpoint in order toexpose data in such a way that semantic interoperability between the two platform instances can beachieved. Furthermore, we have made the design decision that the platform instances should be built
based on established IoT software platforms such as FIWARE and U2. While in a way it limits our design
options, the design options for linking data sources so that semantic interoperability can be achieved boil
down to the following question:
Should (1) data sources be exclusively managed with FIWARE and U2 or(2) should there be a combined approach in which semantically described data sources are managedwith state of the art triple stores?
Note that the first option of managing linked data with FIWARE and U2 is orthogonal to design options
for semantic descriptions described above: if we decide for RDF as the description language, in principle
we could provide even the actual data in RDF using FIWARE and U2. For instance, one could define an“RDF serialization” scheme based on the FIWARE NGSI data model as sketched by Bauer 9. This would
provide a simple approach that would benefit from the proven scalability of the respective IoT platform. In
addition, subscriptions provide some basic streaming support (Requirement FREQ.80). However, there areseveral disadvantages due to the limitations of the data models:
1. No support for linking or graph-based structures
2. Enforcement of linked data required
3. Specific Integration Layer between FIWARE and U2 required
The NGSI data model is purely entity/attribute based and does not provide any support for linking entities
or querying graph-based structures. It is up to the applications to define the semantics of attribute values
that constitute links. Even if such semantics is defined and agreed upon between applications graph based
queries, e.g. retrieving subgraphs or graph traversal queries, can only be implemented by sending multiple,
potentially a large number of queries. If such a linking scheme would be implemented, there would still be
the challenge of how to enforce or deal with that is not linked to any other data sources. Even if we had alldata managed by FIWARE as linked data, we would still need an integration layer between FIWARE andU2. This layer would be comparatively thin if both platforms support the linking and use common
infrastructures. However, for connecting platforms that use n different data model we would n2-n mapping
components.
The benefit of using RDF as lingua franca and managing the data with state of the art triple stores or graphdatabases has the advantage that data linking and graph-based queries are natively supported and thatthe integration between any platform can be achieved by directly exchanging RDF data. Consequently,
only 2n mapping components would be required. However, there are certain disadvantages if using graphbased databases
1. Limited Scalability
2. Unsuitable for streaming applications

9

https://github.com/telefonicaid/fiware-orion/files/39167/NGSI.Context.Interfaces.and.Semantics.pptx

H2020 EUJ-02-2016

CPaaS.io

Page 15 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

3. Performance vs. costs for replication
While graph-based databases and in particular triple stores can scale only to a limited degree there isconsiderable progress in the graph database community and several benchmarking efforts that keep trackof the performance improvements over time (e.g. LDBC 10 , HOBBIT 11 , LITMUS 12 ). In particular theannouncement of Neptune 13, a fully managed AWS service offering a graph database supporting the RDF
model and SPARQL shows that triple stores are achieving mainstream adoption. However, graph databases
are at the time of writing not suitable for streaming application. If there are hard real-time requirements,
one would need to combine graph databased with RDF Stream Processing Systems (e.g. CQELS 14 or CSPARQL 15, RSP). While there is considerable effort and progress (e.g. cf. Dagstuhl Seminar on Big Stream
Processing 16 or Stream Reasoning Workshop 2018 17), this area is not yet mature and would require someeffort to develop a suitable solution. Finally, if data is replicated, between FIWARE/U2 and the graphdatabase there is the disadvantage of cost for replication (storage costs) and managing potential
inconsistencies between the data representations. This can be mitigated by using ontology-based data
access approaches in which only metadata about the data stored in FIWARE is managed in the graphdatabase and queries are mapped to FIWARE/U2 queries. However, the overhead for query rewriting must
be carefully considered.
In principle the same argument applies for U2. U2 architecture provides both SPARQL-based and RESTbased API to access data repository. Functions provided by SPARQL-based API are same as these of
FIWARE. Moreover, u2 architecture provides REST-based API. This REST-based interface returns data
based on JSON-LD [Manu Sporny 2014] format. This interface is required by the following reasons:
1. The specification of SPARQL is rather difficult; knowledge of RDF is needed to understand thisinterface. This REST-based interface requires knowledge of web API and JSON format. Therefore,
more programmers and engineers can access RDF data by using this interface.
2. SPARQL interface is more complex than this REST-based interface. Some sensors or actuators are
too small to implement modules for SPARQL interface. Moreover, some data are generated in realtime; it is too frequent to process SPARQL-based query.
3. SPARQL interface requires RDF-based database. However, some existing data resources(especially real time data resources) cannot deal with SPARQL interface.

3.2.3
Using Ontologies
When using vocabularies and ontologies for solving the interoperability issue there are two extreme
choices:

10 http://www.ldbcouncil.org/
11 http://project-hobbit.eu/

12 https://litmus-benchmark-suite.github.io/#about
13 https://aws.amazon.com/neptune/

14 https://code.google.com/archive/p/cqels/wikis/CQELS_language.wiki
15 http://streamreasoning.org/resources/c-sparql

16 http://www.dagstuhl.de/de/programm/kalender/semhp/?semnr=17441
17 http://www.ifi.uzh.ch/en/ddis/events/streamreasoning2018.html

H2020 EUJ-02-2016

CPaaS.io

Page 16 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

1. Pre-scribe the use of concrete vocabularies and ontologies
2. Support any vocabulary and ontology
The first approach has the advantage that there is a clear semantic model of the data facilitating thesemantic interoperability. On the downside it requires extensive works in order to assess and possibly
extend and interlink the various models (cf. for instance the efforts required for defining the Smart
Appliances REFerence SAREF ontology 18 ). For data providers it also requires strong ontologicalcommitment making it harder to provide data to the platform.
The second approach has the advantage that data providers can easily integrate their data. On thedownside this may result in many complex transformations. Here the risk is that data providers just provide
RDF versions of their data without linking to any vocabulary or ontology. This effectively means that wewould have very weak semantic data with which it is very hard to achieve interoperability.

3.3

Approach and Implementation

In this subsection we describe our approach and detail the current implementation status.
3.3.1
Semantic Description of Data Sources
Our approach for describing data sources is based on extending the DCAT vocabulary 19 and reusing
existing extensions as described in Deliverable D6.1 Holistic Data Management State of the Art. This has
the benefit that OGD can easily be used by the platform. We started working on a new ontology called the
ICAT ontology that extends DCAT to better describing IoT sensor data. As outlined in Figure 2 we propose
to extend the distribution concept of DCAT with new subclasses that allow to distinguish between
streaming and traditional batch data sets as well as discriminate between linked and non-linked data.

Figure 4: Initial ICAT Class hierarchy

A data stream distribution would provide a client to stream data, e.g. via MQTT or the FIWARE/NGSI datainterfaces. A linked data distribution is a data distribution in which the format (dcat:format) or
mediaType (dcat:mediaType) is described as an RDF serialization (e.g. JSON-LD, N-Triples, RDF/XML,

18 https://sites.google.com/site/smartappliancesproject/ontologies/reference-ontology
19 https://www.w3.org/TR/vocab-dcat/

H2020 EUJ-02-2016

CPaaS.io

Page 17 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models…). A non-linked data distribution is a data set distribution that contains a mapping to RDF concepts. Thiscan for instance be achieved using the TripleMap concept of R2RML.
3.3.2
Holistic Data Access and Management
As an initial approach for holistic data access and management we decided for a combined approach
between the FIWARE/U2 architecture and a triple store based approach. This provides us with a method inwhich we can easily link data from heterogeneous sources and exploit ontologies and reasoners in orderto achieve true semantic interoperability. The general idea here is that we can ask federated SPARQL
queries to both platforms. In the initial version of our implementation we decided to replicate FIWARE data
in a triple store by provdings an NGSI to RDF Mapping Service. For the Japanese u2 based instance wedecided to re-write SPARQL queries to the underlying stores used by the u2 architecture.
3.3.2.1 FIWARE-based Implementation
We have implemented an NGSI to RDF Mapping Service that translates NGSI 10 update requests to RDF
triples that are inserted into the Semantic Data Repository of CPaaS.io. The initial implementationarchitectures is depicted in Figure 3 below. The mapping service subscribes to NGSI entities and attributes
that should be transformed to RDF triples at the IoT Broker (1). This way we can control what data containedin a FIWARE-based CPaaS.io instance should be exposed via the semantic integration layer. If an NGSI
application updates context information in the IoT Broker (2) that matches the subscription a notificationis sent to the address specified in the subscription in step 1. Figure 3 shows a concrete deployment variant
that we are using in which the notification is not sent directly to the mapping service, but to a proxy (3)that secures the communication to the mapping service by using basic authentication and HTTPS (4). Thisis required as we allow the CPaaS.io infrastructure to consist of a set of cloud services that can be hosted
on different servers. The mapping service translates the received data into triples and inserts them into the
Semantic Data Repository that exposes a SPARQL endpoint (5) that allows RDF applications to access thedata (6).

Figure 5: NGSI to RDF Mapping Service

Figure 4 shows a sample subscription for the activity of a user as required for the My Events application.
The subscription is active for one month (P1M) and specifies that a notification should be sent whenever
there is a change in the activity attribute (ONCHANGE). Notifications are throttled to be sent at a rate ofat least 10s.

H2020 EUJ-02-2016

CPaaS.io

Page 18 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

Figure 6: Sample subscription on the activity of a user issued by the mapping service

Figure 5 shows how a context update request is transformed and added to the semantic data repository.
A new activity of Jon is added with an associated timestamp as metadata. The transformation service
distinguishes between attribute values with and without metadata. If attribute values of a primitive typesuch as string or integer are specified, the mapping service will by default create typed literals. In the given
example however, there is metadata specified. In this case we create a resource that is typed as
AGTCPaaSCore:StringType. This class can be understood as a kind of boxed primitive type that allows
adding additional meta information. In this case the creation time.
agtres:JonSmith agtres:hasActivity
agtres:hasActivity97118eb0-e64d-11e7-a26c-1db38a526bb7;
a foaf:Person.
agtres:hasActivity9711b5c2-e64d-11e7-a26c-1db38a526bb7
agtres:creation_time"2017-09-13 T11:38:55:120+0100"^^xsd:string,a AGTCPaaSCore:StringType;AGTCPaaSCore:hasLiteralValue"DANCING"^^<http://www.w3.org/2001/XMLSchema#xsd:string.

Figure 7: An update request for Jon's activty (left) and the generated triples in the Semantic Data Repository (right)

3.3.2.2 U2 Implementation
Figure 6 shows query handling components of u2 architecture. U2 architecture federates various databases
such as RDF database (graph database), relational database, KVS database, GIS database, and so on.

H2020 EUJ-02-2016

CPaaS.io

Page 19 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

Figure 8: Query-handling Components of u2 Architecture

Figure 7 shows a method of query handling in u2 architecture. An application accesses to u2 endpoint tosend query. In this time, a front-end module accepts this query and send the query to query converter
module. A query converter module converts query to target database, accesses to target database, andget response. After that, this query converter converts the response to JSON-LD format and sends it tofront-end module. A front-end module returns the formatted response to the application.

Figure 9: Methods of query-handling in u2 Architecture

3.3.2.3 Discussion
In the current implementation we require that all entity ids, types names and attribute names and types(except primitive types) are specified as IRIs. However, the mapping service can easily be extended to
H2020 EUJ-02-2016

CPaaS.io

Page 20 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

transform any data by mapping to a default namespace. However this would mean that we use semantic
information have to deal with more ambiguous data that is not effectively linked.
As discussed above the current approach has limitations with respect to scalability. We will investigate theboundaries of scalability and consider Ontology based data access methods (OBDA) as already offered bymany products, e.g. the concept of virtual graphs in Stardog.
As we are exploiting the subscription capability of the IoTBroker we are satisfying the requirement ofenabling analytics of “live” data (FREQ.80). However, there are certainly limitations with respect to thelatencies as we are persisting data to a triple store.
Finally we accept the costs for duplicated storage and issues related to consistency. These costs can be
avoided by directly ingesting data to the triple store.
3.3.3
Guidelines on using Ontologies
The most important guideline we provide is that (1) we encourage to link RDF data against ontologies, and(2) that established vocabularies and ontologies are used. Most prominently we recommend the usage of
the SSN 20 ontology for describe sensors, their deployments and measurements.

3.4

Summary

In summary we have explored the options for linking data from various sources and implemented a firstversion of the semantic integration layer that allows data provided via FIWARE to be offered via SPARQL
endpoint. This enable the semantic integration between the European FIWARE-based and Japanese u2based CPaaS.io instance. Moreover, it provides a common semantic data integration layer for any other
non-FIWARE based IoT platform such as AGT’s own IoT-based AI platform.
More specifically we have an initial solution for the following work package objectives:••

We provided a mechanism for linking heterogeneous data sets from variety of sources.
Enable the provisioning of the data as linked data through SPARQL endpoints.
Table 3: Mapping of requirements to WP6 contributions

UNI ID
Description
NFREQ.27 Data MUST be interoperable across thedifferent parties and domains (data/service
providers)
FREQ.3

FREQ.9The IoT system MUST enable the dynamic
discovery of IoT Resources, Virtual Entities andall services (VE Service and IoT Service) basedon respective specifications.
Object (Virtual Entities), VE- & IoT-services andresources shall be semantically described andregistered

20 https://www.w3.org/TR/vocab-ssn/

H2020 EUJ-02-2016

CPaaS.io

WP6 Contribution
Offering data as linked data that isbased on ontologies and using crossregional standards such as RDF andSPARQL
Initial high level concepts for linked,
non-linked, streaming and batch data
sources defined.

Page 21 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

FREQ.14

The platform shall be able to aggregate datafrom various sources using LOD principles

FREQ.100

It MUST be able to describe data sets (contentwise and access-wise) in order to allow for
discovery and lookup
The p/f MUST provide mechanisms forcollecting, publishing and storing semantic
data (historical data)
CPaaS.io platform MUST be able to perform
analytics, and event detection/prediction using“live” and historical data

FREQ.110
FREQ.80

NFREQ.36 The architecture MUST favour the reuse ofwide-spread standards for Data and API
FREQ.12 The platform shall provide a SPARQL end-point
for accessing semantic data (either annotateddata or semantic descriptions)
FREQ.13 Any data at the platform level shall befollowing a accepted set of standard schemas(ontologies) for annotation

4

Deployment Knowledge Base

4.1

Problem Description

LOD can be directly ingested via the
Semantic Data Repository. The initialconcept for Semantic Data Description
allows the description of external data
sets that can be ingested on demand.
Addressed by re-using DCAT
vocabulary.
Addressed by ICAT ontology that canalso be used to describe data sets
generated by CPaaS.io
Historic data access is facilitated bypersistency of SDR. Live data can either
be pushed to the SDR directly or via the
IoT and Context Broker
Use of RDF, SPARQL, OWL, SSN and
DCAT
The SDR offers a SPARQL endpoint forannotated data.
Use of SSN and DCAT

In real IoT deployments it is not just sufficient to deploy a sensor and send the data to a platform thatmanages the data. It is important to know which sensors are deployed where and more importantly, what
physical entity they observe so that the data can correctly be interpreted. The concrete problem lies in how
such deployment metadata can be created and linked to the actual sensor data. While the SSN ontology
provides some framework for describing sensors and their deployments it is insufficient to describe thedetailed deployment aspects that are required to link deployment metadata to sensor data.
We have therefore analysed various type of IoT deployments such as smart city deployment, event
deployments and smart home deployments. Each of these deployments vary with respect to therequirements on what aspects need to be described:••

Smart City deployments are characterized by having a spatial extent and sensors are typically
deployed for an undetermined duration (cf. Waterproof Amsterdam application)
In Event deployment sensors are often deployed for a fixed duration (during a Color Run) and itis crucial to know which person is associated with which wearable sensor. In particular it is important
to notice that wearable are effectively mobile deployments that do not have a static location.

H2020 EUJ-02-2016

CPaaS.io

Page 22 of 35

Deliverable D6.3•

4.2

Holistic Data Management: Initial Concepts and Models

Smart Home deployments focus on deployments in private buildings. If they are related to device
level energy readings as in the case of the GrowSmarter 21 project, it is particularly important toknow which devices are associated with which smart plugs.

Approach and Implementation

We used an internal ontology of AGT that extends the SSN ontology with deployment related concepts.
The initial ontology was heavily focused on supporting event deployment. During the first year of the
CPaaS.io project we iterated over the ontology design in several development cycles (Sprints) in order tobetter suite the requirements of the project. The development was heavily driven by the Deployment Tool,a reactive Web Application that we used to validate the ontology in real deployment scenarios. The
deployment tool manages all the data fully as linked data so that we have rich semantic information about
the sensor deployments. Versions of the tool have been successfully used in commercial IoT deployments
as well as in the CPaaS.io project for semantically enriching wearable data as used in the My Events
application (see D2.2 First Prototype Release).
The result of our development cycles lead to a new ontology version in which we did some major
refactoring in order to better support Smart City and Smart Home deployments. The deployment tool
supports the following functionalities.
•
•
•
•
•
•

Sensor, Device and Deployment package management
Type Management (devices, events, persons, groups)
Event and Venue Creation
Map-based static deployments
Mobile Deployments
Ontology-based UI creation and validation

Figure 8 shows the device inventory view of the tool in which deployable devices are managed. On asecond map device sets or deployment packages such as a mobile phone and associated wrist band canbe managed. The overview shows the number of instances. Instances can be selected and viewed and theirdetails edited or completely deleted. The details view below shows the details of The Things Uno device asused by the Waterproof Amsterdam application along with deployment information for the review eventat the University of Tokyo. Note that the view combines properties both of the device type (weight,
network) and actual instance (inventory id).

21 http://www.grow-smarter.eu/home/

H2020 EUJ-02-2016

CPaaS.io

Page 23 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

Figure 10: Device Inventory View of the Deployment Tool

Figure 9 shows the device type management view of the deployment tool that provides limited
functionalities in modifying the device types. New Types can be created, deleted and modified. New fields
or constraints of inherited fields can however not be managed via the tool. This requires direct modificationof the ontology. We also check the actual usage of the types and restrict the capability to delete or modify
types when they are used in a deployment. Corresponding warnings are displayed.

H2020 EUJ-02-2016

CPaaS.io

Page 24 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

Figure 11: Device Type Management view of the deployment tool

Figure 10 shows the event management view showing details about the CPaaS.io review event. Each event
has a name, duration, location, mobile deployment and several venue deployments. Mobile deployments
can be accessed via the top most icon on the top right. The CPaaS.io review event contains one venue
deployment at the Daiwa Ubiquitous Computing Research Building (“Daiwa Building”). Static deployments
are managed via floor plans on which the actual deployments can be interactively managed. Figure 11
shows a deployment the Things Uno device with a corresponding observation zone. Observation zones
can be interactively drawn and assigned with either unstructured meta information (rdfs:comment) or
structured information as defined in the ontology.
A device or deployment package can also interactively be deployed on the map and associated with anobservation zone. Each deployment has a duration and include meta information such as height, mounting
option, network information, orientation and additional comments. In order to support the iterativeplanning process of a deployment, the only mandatory option when creating a deployment is the
specification of the device type that should be deployed. At later point the concrete instance can be chosen.
Figure 12 shows the mobile deployment view of the Color Run event using pseudonyms for the user names.
Similar as to static deployments, each mobile deployment has a duration and associates a deployabledevice with a user. Note that the end time is optional allowing indefinite assignments that can be closed
at a later point in time. Various validations are performed in order to avoid impossible associations suchas associating a device to two users at an overlapping time.
For ease of use batch additions can be made defining a common duration for all members of a group. The
duration can also be directly linked to the event duration. For instance the color runners are part of a Color
Run team and the duration can be linked to the duration of the Color Run event.
All the data managed by the deployment tool is accessible via the SPARQL endpoint of the Deployment
Knowledge Base.

H2020 EUJ-02-2016

CPaaS.io

Page 25 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

Figure 12: Event Management View

Figure 13: Static Deployment View

H2020 EUJ-02-2016

CPaaS.io

Page 26 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

Figure 14: Mobile Deployment View

5

Privacy Support Mechanisms

5.1

Problem Description

With each piece of information newly available to a smart city the city increases its potential to improveand offer its citizen a wealth of information for their own purposes. With the growing number of smart and
IoT devices in private households, being able to include private information into the stream of data and at
the same time not to expose CPaaS.io users and their information is crucial for a smart city. The newEuropean General Data Protection Regulation (GDPR) dictates that data belongs to the data subject andmust only be used in accordance with the data subject, even if the data has been created, gathered orinferred by a third-party. This fundamentally changes the perspective on data for any organisation, person,or service processing personal data. It also changes today’s and tomorrow’s perspectives on the private
data pools of smart buildings, smart neighbourhoods, and smart cities. Even more so as under the new
GDPR any data with a relation to an individual is considered personal data, e.g. data concerning electricity
usage and production, water, waste water, traffic movement, consumer habits, communication habits,
utilization of buildings, etc.
Not only private citizens but also many data consumers are uncertain about how to react to the new
regulation in order to ensure the legal compliance of their data processing. Uncertainty to this degree
resulting in fear of a misstep can be paralysing for any project relying on private and/or personal data.
Privacy issues arise as soon as personal data is collected by CpaaS.io Since this data has to be considered
protected, it is the platform’s duty to treat the data accordingly. In addition, the data subject might decideto open its data (or a part of it) to a third-party. Both these facts combined lead to the following questions
to be addressed by the CpaaS.io Privacy Support Mechanism:•

How can a data subject know whether there is personal data related to them on the platform andwhat it is?

•

How can a data subject determine who has access to its data?

H2020 EUJ-02-2016

CPaaS.io

Page 27 of 35

Deliverable D6.3•

5.2

How can a data subject monitor the access to its data?

Relation to Requirements

UNI IDNFREQ.9

Description
Expose data and services toauthorized users

NFREQ.18

Allow users to use free services
anonymously

NFREQ.19

Allow people to use free services
anonymously

NFREQ.20

Allow users to control which data
they are willing to provide and how
their data should be used.
Provide communication
confidentiality

NFREQ.23

NFREQ.24

Be extensible for future technologies.

FREQ.23

Data owners shall be able to set
access-control rights/ policies (set upby data subject) to their data stored
on resources.
CPaaS.io platform shall be able toperform analytics, and event
detection/prediction using “live” andhistorical data

FREQ.80

5.3

Holistic Data Management: Initial Concepts and Models

Contribution
The dashboard exposes the data stored for the data subject
in a visualized way. Unless exclusively open data is exposed,
the user is subject to authentication beforehand.
If only public, open data is used for the visualizations, nouser authentication is required. Consequently, the service
can in these cases be offered without any authentication or
identification.
If only public, open data is used for the visualizations, nouser authentication is required. Consequently, the service
can in these cases be offered without any authentication.
Each data subject can use the dashboard to determine whichdata is shared with whom. Additionally, the data subject can
specify the purpose and indented use of the data.
All communications between the dashboard and the context
broker, as well as all communications between thedashboard and the data subject are subject to transportencryption.
Due to the modular setup of the dashboard and the use ofopen-source technologies, an adaption or evolution istherefore easily possible, can publicly be reviewed anddistributed.
When sharing their data, users can define the conditions,which must be met for the data to become accessible.
The dashboard can be used to perform analytics on older
data or even download the data for detailed analysis, unlessretention of this data is prohibited or constrained either by
law or by the data subject’s terms of usage. Live data canalso be used to track an ongoing development.

Design Options

There are two basic ways to address the challenge of storing and using personal data as well as managing
third-party access to it: the centralized or the decentralized way.
5.3.1
Decentralized Approach
The decentralized setup has the advantage that the data subject not only owns but also hosts its data. Not
having to trust anyone, the data subject can store all the data on a home server, upload it in an encrypted
form to the cloud or simply store it on a storage device. As long as the home server is not exposed to theinternet, the encryption and key-length used for the cloud upload is up-to-date and the storage device is
physically not lost or stolen, the probability of unauthorized data access is relatively small.
The major risk in well-maintained decentralized approaches is not data theft and abuse but rather data
loss for several reasons:H2020 EUJ-02-2016

CPaaS.io

Page 28 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models•

When using a home server, data subjects would have to set up and administrate this server either
themselves using open-source tools to guarantee transparency and some kind of fall-backsolution in case of hardware failure. For most data subjects this expectation is excessive. Using athird-party service to administrate and secure the home server would of course be an option.
However, due to privacy reasons, the data on the home server would have to be encrypted toprotect it from third-party access. Under these circumstances, the home server approach would
slip into a cloud approach, with the speciality that the cloud consists of one device and isphysically located at the data subject’s premises. Hence, either the data subject can manage andsecure its home server itself if it has the necessary know-how (and most don’t), or it canoutsource it to a service provider (and it ceases to be a home server approach). Taking on the
responsibility itself, the data subject potentially risks data loss over time, e.g. due to unmanaged
hardware failures, key or password loss, improper software configuration, etc.

•

When using a cloud approach, data loss is only indirectly a problem. Due to storage area
networks and server farms across the world, most cloud providers are able to reliably prevent
data loss due to hardware breakdown or software failure. The problem remaining is the questionof access. It is conceivable that data subjects could lose their credentials to access their cloudstorage, but most cloud providers offer a process to recover them in order to restore data access.
However, since the data had to be locally encrypted by the data subject before uploading it to
the cloud, loss of the encryption key or the password protecting the encryption key equals to aloss of all the data encrypted with the lost encryption key. As the cloud provider never shouldhave possession of the encryption key for data protection reasons, it cannot offer any recovery
process this time. Although there are tools and services offering zero-knowledge support when it
comes to key management, the responsibility on the data subject’s shoulders is considerable.

•

Storing the data on a local storage device (e.g. USB sticks, hard disks, DVDs, even printed out QR
codes) bears an inherent risk of both, data loss and data theft. Most data carriers’ reliability
rapidly declines after a few years or decades. To prevent data loss, all the data would have to becopied from old data carriers to new ones regularly, including some kind of backup, not a small
responsibility to bear for each and every data subject. Additionally, the risk to lose data as aconsequence of a data carrier damage or loss has to be considered.

While giving the data subject full control of its data, the decentralized approach also encumbers the data
subject with exclusive responsibility for the safety and accessibility of its data. Another drawback is therather elaborate steps necessary to ensure that the data subject’s data is safe and inaccessible forunauthorized requests, while still remaining discoverable. In summary, it can be said that decentralized
solutions have the potential to offer the data subject a higher flexibility, freedom of choice, and security,
while at the same time demanding constant attention, training, and sole responsibility.
5.3.2
Centralized Approach
While the centralized approach takes away the data subject’s direct control, it also relieves it from its
responsibilities to keep the data safe, inaccessible for unauthorized requests and still discoverable as wellas instantly shareable around the clock. The major risk in centralized approaches is not data loss but
rather mandatory trust and data theft/misuse:
1. While decentralized approaches have options to effectively provide information concerning theirdata without exposing it to an external party, when confiding its data to the CPaaS platform, thedata subject has no other option but to trust that the data will not be misused by the operators of
the CPaaS platform in any form. At this point, technical safeguards can only ensure that the data
H2020 EUJ-02-2016

CPaaS.io

Page 29 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

does not leave the CPaaS platform’s access oversight. The confidence and trust into the cityadministration itself is something that has to be established outside the smart city project.
2. Should the CPaaS platform’s security measures ever be breached, intruders could steal and later
misuse a larger amount of data than a decentralized solution would allow them to.
5.3.3
Direct or zero-knowledge access
Data can either be shared in its entirety or access to the information contained in the data can be
granted without exposing the data itself. Both approaches have their own advantages and drawbacks inperspective to the CPaaS approach.
The advantages of e.g. only answering the question, whether the data subject is older than 18 without
revealing the actual age or the possibility to prove that my data contains something without exposing it,
seems obvious. From a utility point of view, there is no reason why such methods shouldn’t be used.
However, since methods like zero-knowledge proofs and above all homomorphic encryption andcomputation are at the moment not a wide-spread standard and intensively resource-intensive at thescale of an entire city, this additional utility comes at a concrete cost: lower compatibility and scalability.
As the CPaaS project explicitly aims at scalable solutions compatible with already existing standard, direct
data access is the approach of choice.

5.4

Approach and Implementation

The dashboard’s menu structure has to reflect the three main topics from a data subject’s perspective:
Data Maintenance, allowing the data subject to examine all own data sets stored in CpaaS.io, check theirmeta-data, and if necessary request data correction or data deletion.
Access Management, allowing the data subject to examine all currently active data access policies, add
new policies, or remove unwanted ones.
Data Reporting, allowing the data subject to visualize own data sets and data sets the data subject had
been granted access for.

The dashboard is only accessible after successful user authorization. The following chapters address anddescribe the detailed functionalities of the three modules of the dashboard.
H2020 EUJ-02-2016

CPaaS.io

Page 30 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

5.4.1
Data MaintenanceThe data maintenance module addresses the following tasks or problems:•

Provide insight into own and shared, foreign data (incl. meta-data)•

Provide insight of active data feeds (incl. source meta-data)••

Allow initiation of a data correction or a data completion request
Allow initiation of a data deletion request

H2020 EUJ-02-2016

CPaaS.io

Page 31 of 35

Deliverable D6.3•

Holistic Data Management: Initial Concepts and Models

Allow for downloads of personal data in a structured, standardized, digital format

From a data storage point of view the new GDPR Provenance ontology can be used to better describe
the to “express a data-flow model that can trace how consent and data are used by extending theappropriate vocabulary with GDPR-related terms” (Adapt Centre, 2017)
5.4.2
Access Management
The access management screen primarily serves the purpose to request, grant, adjust, or revoke access to
own data sets. As per default, no personal data is shared with anyone but the data subject. Should a data
subject decide to grant access to some of his/her private data, the following limitations become relevant:
1. The data subject has to choose a data source or data series to be shared. Although read accesscan be reduced to individual data entries, only entire data sources or data series can be shared.
2. Other CPaaS.io users can be granted read access to the entire data source or data series orindividual data sets of this data source. For access to individual data sets, read access to theresource has to be granted, in order to allow data to flow at all. Then, access to certain data sets
has to be restricted according to a user’s wish. Access can be defined using any CP-ABE policies,
stating the attributes that have to be met before access is granted, e.g. [FirstName: Bruce,
LastName: Wayne] or [Organisation: MunicipalAdministration, Department: DisasterRelief].
3. All access rights have to be re-confirmed after a default period defined per CPaaS.io instanceaccording to the local context.
4. After a resource and its data sets have been shared with other CPaaS.io users, the newly sharedresource shows up as available in the list of resources “shared with me”. At this point, the shareddata can be access and evaluated in the data reporting modules of the users it has been sharedwith.
5. All access rights can be revoked at any time, immediately preventing the affected users fromaccessing the data previously shared with them, but not removing already existing reports basedon the now restricted data.
Apart from allowing users to share own data with other CPaaS.io users, the access management module
also presents its users with a current overview of what data has been shared with them. Shared data isdistinguished from open data and only shared, personal data is listed here. The shared data resourcesand data sets can also be found in the reporting module.
5.4.3
Data Reporting
The Data Reporting module mainly consists of a visualization tool receiving data from the context broker
and visualizing them in a semi-configurable way. The dashboard administrator sets up different reports,
charts and maps for the CpaaS users to take advantage of. While the users cannot change these premade reports by e.g. changing data fields or data sources, they can vary the time range to analyse andcan assemble own selections of favourite reports.
Data reporting can be perceived as rather a result than an element of holistic data management view. It
bases on all decisions made in course of the holistic data management, such as which data to subscribe
to or which data to share with other users to enrich their data reporting. While this is true, there is still a
crucial management aspect included in how the data reporting module is set up by the local
administrator. Setting up the dashboard is a task heavily depending on the local application of CPaaS.
Using the system to track visitor activities during an open air festival demands completely different
visualizations than using CPaaS to track the operational data of a community-owned wind turbine park.
It is the local administrator’s responsibility to decide which data sources are used in what kind of reportin which granularity and inter-relatability. This is especially true for public reports, where the released
H2020 EUJ-02-2016

CPaaS.io

Page 32 of 35

Deliverable D6.3

Holistic Data Management: Initial Concepts and Models

data has to be informative in relation to the event at hand, but at the same time must not allow interrelation with other data sets, since such inter-relations could reveal personal information to an attacker.
For personal reports, each decision what to share with whom still remains the data subjects decision.
Nevertheless, it is recommended to rather create a higher number of reports expressing fewer
information each, than a lower number of all-included-reports. The latter solution complicates theprecise sharing of information as the sharing process cannot be customized to the degree necessary.

6

User empowered master data management

6.1

Problem Description

According to the General Data Protection Regulation (GDPR) that will be enforced in May 2018 in theEuropean Union, each data subject does not only have the rights to know that there is data stored inrelation to it, or the right to access this data, but also the right to enforce correction or even deletion ofthis data. Consequently, the CPaaS.io platform has to offer mechanism for the data subject to trigger sucha data correction or deletion.

6.2

Design Options

When operating in a decentralized setup with the data subject hosting all its data, the correction or deletionof data is not an issue, as it can be performed by the data subject itself. In all other cases, the data subject
needs a mechanism how to trigger a data correction or deletion.
One option would be to combine these mechanisms with the data display for the data subject to have theopportunity to click “correct” or “remove” whenever wrong data shows up. This would offer the advantagethat incorrect data can be corrected or deleted directly.
Alternatively, the data correction can be separated from the data reporting and combined with the
management of data access rights. This would not allow for direct intervention in the case of wrong data
showing up, but it would allow for a functional separation between the data display module and the data
access management module, the latter also including the mechanisms for data correction and deletion.
For best compatibility, CPaaS.io implements a data correction module separated from data reporting but
nevertheless allows for direct correction or deletion from the data reporting module, if supported by thedata reporting tool. Many graphical dashboards are able to add links to the graphics produced. This feature
can be used to generate links to the correction module, but such links are not functionally necessary.

6.3

Approach and Implementation

The implementation of holistic data management is part of the implementation of the dashboard which inturn is hosting all front-end parts necessary for the data-subject to perform holistic data management.
From a technical point of view there are two fundamentally different dashboards: one dashboard
performing the visualization of stored or shared data (not used for holistic data management) and anotherone handling the assignment and reporting of access permissions called holistic data management. The
latter dashboard is built using a NGSI-ready JavaScript library such as Wirecloud or its context-free clone
NGSIJS.
H2020 EUJ-02-2016

CPaaS.io

Page 33 of 35

Deliverable D6.3

7

Holistic Data Management: Initial Concepts and Models

Summary

In this deliverable we described a first version of the initial concepts and models for holistic data
management. Adressing both the challenge of (1) coping with the semantic integration of heterogeneous
Smart City data sources across diverse IoT platforms and (2) managing personal data and providing citizens
full control over their data. We described five problems in this area, analysed them with respect to thedesign options and described our initial approach and implementation for each of them. The problems
were as follows:
1. Data Quality: How to assess the data quality ?
2. Semantic Interoperability: How to link data from various soruces across heterogeneous platform
instances ?
3. Deployment Metadata: How to keep track about which sensors are deployed where and which
properties of a physical entity they observe ?
4. Privacy Support Mechanisms: How can a data subject know about and control her data managed
by the platform?
5. User Empowerment: How does the CPaaS.io platform empower users to enforce correction ordeletion of their data?
For data quality we will integrate SHACL based data validation extensions for specific use cases andpipelines. A vocabulary for expressing data quality will be developed within the prototypal data quality
assessment implementations.
We provided an initial approach for solving the semantic interoperability between the European FIWAREbased platform instance and the Japanese u2-based instance by exchanging data via federated SPARQL
endpoints. For that purpose we have implemented a NGSI to RDF mapping service providing a SPARQL
endpoint for the FIWARE based platform instance. On the u2 side we chose to rewrite SPARQL queries tothe underlying u2 stores.
In order to solve the deployment metadata problem we have developed and tested extensions of the SSN
ontology and implemented a web-based management tool for managing deployment data that is fully
stored as RDF data in the deployment knowledge base. The deployment knowledge based has beensuccessfully used for the My Events application.
In order to support data subjects optimally we have described a dashboard application that supports data
maintenance, access management and data reporting. Data maintenance allows the data subject toexamine all own data request data correction or data deletions. Access management allows the data
subject to examine all currently active data access policies, add new policies, or remove unwanted ones.
And finally, data reporting allows the data subject to visualize own data sets and data sets the data subject
had been granted access for.
As a next step, the intial concepts, approaches and implementations described in this deliverable will befurther refined driven and tested by the applications.

H2020 EUJ-02-2016

CPaaS.io

Page 34 of 35

Deliverable D6.3

8

Holistic Data Management: Initial Concepts and Models

References
Adapt Centre (2017) The GDPR Provenance ontology. Release 2017-10-20. Available at:https://openscience.adaptcentre.ie/ontologies/GDPRov/docs/index-en.html.
Barnaghi, P., Bermudez-Edo, M. and Tönjes, R. (2015) ‘Challenges for Quality of Data in Smart Cities’,
Journal of Data and Information Quality, 6(2–3), pp. 1–4. doi: 10.1145/2747881.
Bizer, C. et al. (2009) ‘DBpedia - A crystallization point for the Web of Data’, Journal of Web Semantics,
7(3), pp. 154–165.
Debattista, J., Lange, C. and Auer, S. (2015) ‘Luzzu - A framework for linked data quality assessment’,in ISWC 2015 Posters and Demonstrations Track. Available at:
http://www.scopus.com/inward/record.url?eid=2-s2.0-84955600970&partnerID=tZOtx3y1.
Fraefel, M. et al. (2016) ‘Holistic Data Management : State of the Art’, pp. 1–61. Available at:https://cpaas.bfh.ch/wp-content/uploads/2017/08/CPaaS.io_D6.1_Holistic-Data-Management_Stateof-the-Art.pdf.
Heuel, S. (2015) Internet of Things: Live Data Quality Assessment for a Sensor Network – geo.ebp.ch.
Available at: http://geo.ebp.ch/2015/03/21/sense-your-city/ (Accessed: 29 November 2017).
Karkouch, A. et al. (2016) ‘Data quality in internet of things: A state-of-the-art survey’, Journal of
Network and Computer Applications. Elsevier, 73, pp. 57–81. doi: 10.1016/j.jnca.2016.08.002.
Kolozali, S. et al. (2013) ‘Real-Time IoT Stream Processing and Large-scale Data Analytics for Smart
City Applications’, Eu Fp7, 18(609035), pp. 2–3. Available at: http://www.ictcitypulse.eu/page/sites/default/files/citypulse_d3.1_semantic_data_stream_annotation_for_automated_processing_final.pdf.
Timms, G. P. et al. (2011) ‘Automated data quality assessment of marine sensors.’, Sensors (Basel,
Switzerland), 11(10), pp. 9589–602. doi: 10.3390/s111009589.
Zaveri, A. et al. (2014) ‘Quality Assessment for Linked Open Data: A Survey’, Semantic Web Journal (byIOS Press), 1, pp. 1–31. doi: 10.3233/SW-150175.

H2020 EUJ-02-2016

CPaaS.io

Page 35 of 35