#  Introduction

##  Purpose and Scope

A Data Management Plan (DMP) is a formal project document which outlines the
handling of the data sources at the different project stages. The H2020
guidelines [22] provide an outline that must be addressed. The DMP covers how
data will be handled within a project frame, during the research and
development phase but also details the intentions for the archiving and
availability of the data once the project has been completed [5,8]. As the
project evolves, the DMP needs to be updated to reflect the changes in the
data situations and the understanding of data source becomes more concrete.

YDS as project aims to create a data eco system bringing together state of the
art data processing technology with recent content about governmental
budgetary and economical transparency in a platform that facilitates European
citizens and, in particular, journalists, creating stories based on factual
data.

The technological foundations of the data management platform being
established within YDS are such that it is intended to be multi-purpose and
domain-agnostic. Within YDS, this generic data management platform is piloted
using three closely related data domains: the financial transparency in the
Greek and Irish governments and governmental development aid. This core
activity of collecting and aggregating data from many different (external to
the YDS project) data sources makes that metadata management of the used and
produced datasets is key. By applying the DCAT-AP [16] standard for dataset
descriptions and making these publicly available, the YDS DMP covers 4 out of
4 of the key aspects of FAIR data management, as specified in [22] as integral
parts of the platform:

  * Making data findable, including provisions for metadata; 

  

  * Making data openly accessible; 

  

  * Making data interoperable; 

  

  * Increasing data re-use (through clarifying licences). 

  

It is important to note here that even though the YDS DMP strives to align
with the Open Research Data Pilot (ORD) guidance notes provided by the
European Commission  1  , the ORD pilot “applies primarily to the data needed
to validate the results presented in scientific publications”. As the
YourDataStories project neither outputs such data, nor is the original
publisher of data (but establishes an Open Data Repository), the consortium
must align with the associated licensing conditions assigned by the publisher.
This means that one DMP would not be enough to cover all YDS data sources.
Hence, this document establishes a shared framework for all data harvested and
published by the YDS consortium, while additional, data source specific
information is provided in the individual DMP instances, in the Annex of this
deliverable.

##  Approach for Work Package and Relation to other Work Packages and
Deliverables

This deliverable provides the final link in the feedback loop between the DMP
and the Data Source

Assessment Methodology deliverables (D3.1 and D3.2). The second and final
version of the Data

Source Assessment methodology, D3.2, defines a continuous process and related
activities which ensure that relevant data (for open access and to be made
available publicly through existing open access repositories and services) is
identified and verified during the course of the project and beyond. It has,
therefore, proven crucial for completing the individual DMP instances (i.e.
per data source), which are provided in the Annex of this report.

Moreover, the overall project approach to data processing at the time of
writing is provided in D3.7 Data Harvesters v2.0 and both practical and
technical consideration with respect to data storage and sharing are given in
D3.10 Open Data repository v2.0. Both deliverables are to be followed by their
final versions in month 32 of the project.

##  Updates with respect to D2.7

In summary, this deliverable updates the previous version of the Data
Management Plan with regard to the following aspects:

  * Long term data management and storage; 

  

  * Risk management with respect to stigmatization of companies and individuals; 

  

  * User-generated content 

  

  * Destruction of data; 

  

  * Social media data management; 

  

  * Individual Data Management Plans, per data source; 

  

  * Individual (updated) DCAT-AP descriptions, per data source. 

  

##  Methodology and Structure of the Deliverable

The YDS DMP life-cycle which will elaborate the general conditions and data
management methodology is outlined in Section 2. As the YDS pilots frequently
handle manually created content (tables, reports, analyses …), as well as low
quality data, the tooling often requires manual intervention and, hence, the
complete data integration process from source discovery to published
aggregated data cannot be completely automated. Therefore, an important aspect
of the YDS DMP is the general methodology. This, final, version of the YDS DMP
also takes into account the experiences of the pilot cases. The remainder of
this report is structured as follows:

  * _Data Management Plan Checklist_ \- Section 3 provides a description of the basic information required about the datasets that are going to be used in the YDS project. 

  

  * _Metadata Management_ \- Each data source and each resulting dataset of the YDS aggregation process is described with meta-data. This meta-data can be used on the one hand for automating the YDS data ingestion process, but on the other hand also for external users to better understand the published data. This is further described in Section 4. 

  

  * _Access, sharing and re-use policies_ \- An important challenge in the YDS platform is the ambition to combine data from datasets having different usage and access policies. Interlinking data having payment requirements with data that is publicly and freely available impacts the technological and methodological approaches in order to implement the desired access policy. Section 0 outlines this further. 

  

As the YDS pilots are now well defined, the questions relating to the Data
management and storage (long term) are now addressed in much more detail.
Section 6 provides answers to the questions which are considered common for
all pilots, whereas the pilot and data source specific questions are addressed
in the individual DCAT-AP descriptions and DMPs, in the Annex of this
deliverable.

#  The YDS data lifecycle

The YDS platform is a Linked Data platform; therefore, the data ingested and
managed by the YDS platform follows the Linked Data life cycle [4]. The Linked
Data life cycle describes the technical processing steps which are possible to
create and manage a quality web of data. In order to smoothen the process best
practices are described to guide data contributors in their usage of the YDS
platform. This is further discussed in section 2.2 “The generic YDS data value
chain”, while the common best practices [12, 13] are quoted in section 2.3
Best Practices.

Prior to the Linked Data approach to the use of data, data management was
perceived as being an act done by a single person or unit. Responsibilities
(involving completeness, consistency, coverage, etc. of the data) were bound
to the organizations duties. Today, with the used of the Internet and the
distribution of data sources, this has changed: data management is seen as
being a living service within a larger ecosystem with many stakeholders across
internal and external organization borders. For instance, Accenture Technology
Vision 2014 indicated this as the third most important trend in 2014 [9].

##  Stakeholders

For YDS, the key stakeholders have been identified which influence the data
management. Their main interaction routes are depicted in Figure 1: DMP Role
Interactions.

**Figure 1: DMP Role Interactions**

_**Data end-user(s)** : _

The data end-users make use of the aggregated datasets to create their own
story. In Deliverable D2.1 the main data end-user types for the YDS platform
are identified: media & data journalists, auditors, web developers, and
suppliers of business opportunities in public procurement, the civil society
and public institutions. The data end-users are the main drivers of the YDS
platform content: their need for data is the key driver for the content of the
YDS platform.

**_Data source publisher/owner(s):_ **

Represent the organization(s) which provide the data being integrated into the
YDS platform. For many data sources, especially those that are published as
Open Data by the public bodies, the interaction between YDS and the data
source publisher/owner is limited to technical access to the data (a download
of a file, a registration to obtain an API key). As the YDS platform is now
much more mature, to ensure a quality service level to the data end-users, the
consortium has set up a more intense collaboration with some of the key data
sources, such as the International Aid Transparency Initiative..

_**Content business owner** : _

Is the person responsible for the content business objectives. The content
business owner makes sure that the necessary data sources are found in a
usable form and that the desired aggregations are being defined so as to
realize the aggregated enriched content for the supported YDS stories. For
each content domain / project pilot a business owner has been identified based
on the experience, familiarity and proximity to the content domain and related
data publishers. The content business owner is also the responsible party for
any questions related to a data source covered by a given pilot. For cross-
domain data, TF, as the work package leader, is the contact point. The
identified CBOs are listed below:

  * Pilot 1: NCSR-D 

  

  * Pilot 2: TF 

  

  * Pilot 3: NUIG, NCSR-D 

  

_**Data wrangler [10,11]** : _

This person acts as a facilitator in that they interact with all stakeholders
but at the level of the integration of the source data into the platform. The
data wrangler ‘massages’ the data using the YDS platform to realize the
desired content. They must understand both the business terminology used in
the source data model(s) and the YDS target mode, understand the end user
objectives and ensureg that the mapping between the models is semantically
correct. The data wrangler is assisted by the YDS system administrator and YDS
platform developers to tackle the technical challenges, but their central
concern is the mapping of the data.

_**System administrator and platform developer** : _

Are responsible for the building and support of the YDS platform in a domain
agnostic way.

##  The generic YDS data value chain

The complex process of a data value chain can be described using the following
stages:

**Figure 2: Data value chain stages**

  * **Discover** : In today’s digitized world, there are many sources of data that help solve business problems that are both internal and external to organizations. Data sources need to be located and evaluated for cost, coverage, and quality. For YDS, the evaluation of the data sources is part of the data source assessment methodology (See Deliverable D3.2). The description and management of the resulting dataset meta-data is one of the main best practices used in the Linked Data community. 

  

  * **Ingest machine processable data** : The ingest pipeline is fundamental to enabling the reliable operation of entire data platforms. There are diverse file formats and network connections to consider, as well as considerations around frequency and volume. In order to facilitate the value creation stage (Integrate, analyze & enrich) the data has to be provided in, or turned into a machine processable format. In the YDS case, the preferred format is RDF [1]. 

  

  * **Persist** : Cost-effective distributed storage offers many options for persisting data. The choice of format or database technology is often influenced by the nature of other stages in the value chain, especially analysis. 

  

  * **Integrate, analyze & enrich ** : Much of the value in data can be found from combining a variety of data sources to find new insights. Integration is a nontrivial step which requires domain knowledge and technical knowhow. Exactly by using a Linked Data approach with a shared ontology the integration process is facilitated in YDS. Where-as the other stages have a high potential of automation to a level where humans are not anymore involved, this stage is driven by human interest in the data. New insights and better data interconnectivity are created and managed by a growing number of data analytical tools and platforms. 

  

  * **Expose** : The results of analytics and data that are exposed to the organization in a way that makes them useful for value creation represents the final step in deriving value from data. The structure of the stages is based on the vision of the IBM Big Data & Analytics group on the data value chain [21]. 

  

When contributing a new data source to the YDS platform the stages are roughly
followed from left to right. In practice the activities are, however, more
distributed in order to keep the platform and the data it provides in the
desired state. Indeed, data that is not actively nursed becomes quickly
outdated. More and more imperfections will show up, to the point that data
end-users consider the data not valuable anymore. Taking care of the YDS
platform content is, hence, a constant activity. From a technical perspective,
this work is supported by the tooling available during the Integrate, Analyze
and Enrich phase. It is similar work as creating newly added value, but then
with the objective to improve the overall data quality (coherency,
completeness, etc.).

A further point to consider is that data based applications also have a
tendency to generate new requirements based on insights which are gained when
studying the data (this forms a loop whichcontinues as understanding of the
data increases  2  and this is shown in Figure 3: Linked Data ETL Process).
This depends heavily on what the data is intended to allow or what it is
intended to be used for (search for understanding, support of a particular
story, tracking of an ongoing situation, etc.).

In the following sections, the above data value chain stages are made more
concrete.

###  Discover

The **content business owners** are the main actors in this stage. Using the
data source assessment methodology, relevant data sources for their content
domain are being selected to be integrated.

An important outcome of the data source assessment is the creation of the
meta-data description of the selected datasets. In section 4, the meta-data
vocabulary that being used is described (DCAT-AP). The expectation raised in
creating the meta-data is that the data sources are well described (what is
the data, which are the usage conditions, what are the access rights, etc.),
but experience has shown that collecting this information represents a non-
trivial effort because it is often not directly available.

###  Ingest machine processable data

The selected datasets are prepared by a _data wrangler_ so that they can be
ingested in the YDS platform. The data wrangler hooks up the right data input
stream, for instance a static file, a data feed or an API, into the YDS
platform. During this work the data is prepared for machine processing.
Especially for static files, such as CSV’s, often additional contextual
information is required to be added in order to make the semantics explicit.
Without this preparation the conversion to RDF results in a technical
reflection of the input, yielding more complex transformation rules in the
Integrate, analyze and enrich stage.

###  Persist

Persistence of the data is de-facto an activity that happens throughout the
whole data management process. However, when contributing a new data source to
the platform, the first moment data persistence is explicitly handled is when
the first steps have been taken to ingesting data into the YDS platform.

Since the YDS platform is about integrating, analyzing and enriching data from
different sources _external_ to the YDS partners, persistence of the source
information is not only an internal activity. It requires interaction between
the content business owner and the data source publisher/owner to guarantee
that during the life time of the applications built on top of the data the
source data stays available. Only carefully following up and the continuous
interaction with the data source publishers/owners will create a trustable
situation. Technically, this is reflected in the management of the source
(meta)data activity.

Despite sufficient attention and follow up, it will occur that data sources
become obsolete, are temporary not available (e.g. due maintenance) or
completely disappear (e.g. the organization dissolves). Many of these cases
are addressable to a certain extent by implementing data persistence
strategies such as:

  * _Keeping local copies_ : explicit activity of copying data from one location to another. The most frequent case is copying the data from the governmental data portal to the YDS platform. 

  

  * _Caching_ : a technical strategy which main intention is to enhance data locality so that the processing is smoother. It may also act as a cushion to reduce the effects of temporary data unavailability. 

  

From the perspective of the YDS data user, _archiving & highly available data
storage _ strategies are required to address the availability of the outcome
of the YDS platform. This usually goes hand in hand with a related, yet
orthogonal activity, namely the application of a dataset versioning strategy.
Introducing dataset versioning provides clear boundaries were along data
archiving has to be applied.

###  Integrate, analyze and enrich

In this stage, the actual value creation is done. The integration of data
sources, their analysis and the analysis of the aggregated data and the
overall content enrichment is realized by a wide variety of activities. In
[4], the Linked Data life cycle is described: a comprehensive overview of all
possible activities applicable to Linked Data. The Linked Data life cycle is
shown in Figure 3: Linked Data ETL Process. (Note: Some activities of the
Linked Data life cycle are also part of other phases like ingestion,
persistence and expose.)

**Figure 3: Linked Data ETL Process**

Start reading from the left bottom stage called “Extraction” and proceed
clock-wise.

As most data is not natively available as RDF, extraction tooling provides the
necessary means to turn other formats into RDF. The resulting RDF is then
stored in an RDF storage system, available to be queried using SPARQL. Native
RDF authoring tools and Semantic Wiki’s allow then the data to be manually
updated to adjust to the desired situation. The interlinking and data fusion
tools are unique tools in the world of data management: Linked Data (or a data
format with similar capabilities as RDF) are the enablers of this process in
which data elements are interlinked with each other without losing their own
identity. It is the interlinking and the ability of using entities from other
public Linked Data sources that creates the web of data. The web of data is a
distributed knowledge graph across organizations which is in contrast to the
setup of a large data warehouse. The following 3 stages are about further
improving the data: when data is interlinked with other external sources new
knowledge can be derived and thus new enrichments may appear. Data is, of
course, not a solid entity but it evolves over time: therefore, quality
control and evolution is monitored. To conclude the tour, the data is
published. RDF is primarily a **data publication format** . This is indicated
by the vast amount of tooling that provides the search, browsing and
exploration of Linked Data.

###  Expose

The last stage is about the interaction with the YDS data users. The YDS
platform is a Linked Data platform, and hence the outcome of the data
integration, analyzes and enrichments is made available according to the
common practices for Linked Open Data:

  * A meta-data description about the exposed datasets 

  

  * A SPARQL endpoint containing the meta-data and the resulting datasets 

  

  * A public Linked Data interface for those entities which are dereferenceable. 

  

Additionally, the YDS platform supports dedicated public API interfaces to
support application development (such as visualizations), discussed in D3.10
Open Data Repository V2.0.

##  Best Practices

The YDS platform is, first and foremost, a Linked Data platform, and in this
section, the relevant best practices for publishing Linked Data are described
[12, 13]. The 10 steps described in [13] are an alternative formulation of
these stages in the context of publishing a standalone dataset. Nevertheless,
these steps formulate major actions in the creation of Linked Data content for
the YDS platform concisely (and that is why they are quoted here):

  1. _STEP #1 PREPARE STAKEHOLDERS:_

  

  

_Prepare stakeholders by explaining the process of creating and maintaining
Linked Open Data._

  2. _STEP #2 SELECT A DATASET:_

  

  

_Select a dataset that provides benefit to others for reuse._

  3. _STEP #3 MODEL THE DATA:_

  

  

_Modeling Linked Data involves representing data objects and how they are
related in an application-independent way._

  4. _STEP #4 SPECIFY AN APPROPRIATE LICENSE:_

  

  

_Specify an appropriate open data license. Data reuse is more likely to occur
when there is a clear statement about the origin, ownership and terms related
to the use of the published data._

  5. _STEP #5 GOOD URIs FOR LINKED DATA:_

  

  

_The core of Linked Data is a well-considered URI naming strategy and
implementation plan, based on HTTP URIs. Consideration for naming objects,
multilingual support, data change over time and persistence strategy are the
building blocks for useful Linked Data._

  6. _STEP #6 USE STANDARD VOCABULARIES:_

  

  

_Describe objects with previously defined vocabularies whenever possible.
Extend standard vocabularies where necessary, and create vocabularies (only
when required) that follow best practices whenever possible._

  7. _STEP #7 CONVERT DATA:_

  

  

_Convert data to a Linked Data representation. This is typically done by
script or other automated processes._

  8. _STEP #8 PROVIDE MACHINE ACCESS TO DATA:_

  

  

_Provide various ways for search engines and other automated processes to
access data using standard Web mechanisms._

  9. _STEP #9 ANNOUNCE NEW DATA SETS:_

  

  

_Remember to announce new data sets on an authoritative domain. Importantly,
remember that as a Linked Open Data publisher, an implicit social contract is
in effect._

  10. _STEP #10 RECOGNIZE THE SOCIAL CONTRACT:_

  

  

_Recognize your responsibility in maintaining data once it is published.
Ensure that the dataset(s) remain available where your organization says it
will be and is maintained over time._

#  Data Management Plan Checklist

Each YDS pilot handles content within the Linked Open Economy domain. The
following information is recorded by the _**content business owner** _ of each
pilot. These questions provide the starting point for using the data sources –
the aim being to find any data usage issues earlier, rather than later  3  .
This basic data information, information about the data or meta-data, require
managing and will be further discussed in section 4. The questions also serve
as a checklist, similar to that provided by the UK’s Digital Curation
Center[5] or the template provided by the Guidelines on FAIR Data Management
in Horizon 2020 [24], and the answers serve as direct input for the individual
DMPs, which are also provided in a machine-readable form as DCAT-AP
descriptions (section 4).

**Table 1: Data Management Plan Checklist**  
  
<table>  
<tr>  
<th>

DMP aspect

</th>  
<th>

Questions

</th> </tr>  
<tr>  
<td>

**Administrative Data**

</td>  
<td>

  * How will the dataset be identified? o A Linked Data resource URI  What is the title of the dataset? 

  

  * What is the dataset about? 

  

  * What is the origin of the data in the dataset? 

  

  * Who is the data publisher? 

  

  * Who is the contact point? 

  

  * When was the data last modified? 

  

</td> </tr>  
<tr>  
<td>

**Data Source**

</td>  
<td>

  * Where will the data be acquired? 

  

  * What documentation is available for the data source models, attributes etc.? 

  

  * For how long will the data be available? 

  

  * What is the relationship between the data collected and existing data? 

  

</td> </tr>  
<tr>  
<td>

**Data formats**

</td>  
<td>

  * Describe the file formats that will be used, justify those formats, 

  

  * Describe the naming conventions used to identify the files (persistent, date based, etc.) 

  

</td> </tr>  
<tr>  
<td>

**Data Harvesting and Collection**

</td>  
<td>

  * How will the data be acquired? 

  

  * How often will the data be acquired? 

  

  * What are the tools and/or software that will be used? 

  

  * How will the data collected be combined with existing data? 

  

  * How will the data collection procedures/harvesting be documented? 

  

</td> </tr>  
<tr>  
<td>

**Post Collection Data Processing**

</td>  
<td>

  * How is the data to be processed? 

  

  * Basic information about software used, 

  

  * Are there any significant algorithms or data transformations used (or to be used)? 

  

</td> </tr>  
<tr>  
<td>

**Data Quality Assurance**

</td>  
<td>



</td>  
<td>

Identify the quality assurance & quality control measures that will be taken
during sample collection, analysis, and processing  4  ,

</td> </tr>  
<tr>  
<td>

</td>  
<td>



</td>  
<td>

What will be the data validation requirements? Are there any already in place?

</td> </tr>  
<tr>  
<td>

</td>  
<td>



</td>  
<td>

Are there any community standards you can re-use?

</td> </tr>  
<tr>  
<td>

**Short-term Data Management**

</td>  
<td>



</td>  
<td>

How will the data be managed in the short-term? Consider the following:

</td> </tr>  
<tr>  
<td>

</td>  
<td>



</td>  
<td>

Version control  for files,

</td> </tr>  
<tr>  
<td>

</td>  
<td>



</td>  
<td>

Backing up data,

</td> </tr>  
<tr>  
<td>

</td>  
<td>



</td>  
<td>

Security & protection of data and data products,

</td> </tr>  
<tr>  
<td>

</td>  
<td>



</td>  
<td>

Who will be responsible for management (Data ownership)?

</td> </tr>  
<tr>  
<td>

**Long-term Data Management**

</td>  
<td>



</td>  
<td>

See Section 6 for more details

</td> </tr>  
<tr>  
<td>

**Data Sharing**

</td>  
<td>



</td>  
<td>

How will the data be shared with the public?

</td> </tr>  
<tr>  
<td>

</td>  
<td>



</td>  
<td>

Are there any restrictions with respect to the dataset or parts of it to be
shared?

</td> </tr>  
<tr>  
<td>

**Ethics and Legal Compliance**

</td>  
<td>



</td>  
<td>

How will any ethical issues, should they arise, be managed?

  * Have you gained consent for data preservation and sharing? o How will you protect the identity of participants if required? 

  

  * How will sensitive data be handled to ensure it is stored and transferred securely? 

  

</td> </tr>  
<tr>  
<td>

</td>  
<td>



</td>  
<td>

What are the licenses required to access and used the data?

</td> </tr>  
<tr>  
<td>

</td>  
<td>



</td>  
<td>

How will any copyright and Intellectual Property Rights (IPR) issues, should
they arise, be managed?

</td> </tr> </table>

**Note:** The full checklist and the related answers, per dataset, are
provided in the Annex. The answers to some of the above questions, such as
Ethics and Legal Compliance (to be discussed in section 7), will be provided
in the sections below, and will serve as default input for the individual DMP
instances.

#  Meta-data management

The data collected and aggregated in the YDS platform is distributed to the
public or used in another aggregation process. A coherent set of data is
called a dataset. Distributing the dataset requires describing the dataset
using meta-data properties.

Within Europe, an application profile of the W3C standard DCAT [15] called
DCAT-AP [16] is being used to manage data catalogues **.** This standard,
which is also a **European Commission recommendation** , enables dataset
descriptions in Europe to be exchanged in a coherent and harmonized context.
Since D2.7 Data Management Plan V1.0, i.e. M6 of the project, DCAT-AP has
undergone a revision to better fit the European needs. At the time of writing,
the current version of DCAT-AP is 1.1.

In addition to this motivation, YDS has extensive in-house knowledge and
experience: the YDS partners, NUIG and TenForce are organizations that played
key roles in the establishing and success of the standards. NUIG actively
supported the creation of DCAT as being the co-editor of the standardization
process and it has continued sharing its expertise in the development of the
DCAT application profile. TenForce, lead and was/is participating in several
projects that contributed to the technological application of the standard
DCAT and the creation of DCAT-AP: LOD2, the European Open Data Portal, Open
Data Support (in which TenForce established the first implementation of DCAT-
AP). Recently TenForce supported the revision of the DCAT-AP process and it is
responsible for the first study on creating a variant for statistical data
STAT DCAT-AP.

Building upon DCAT-AP makes the YDS platform compliant with the European
(Open) Data Portal ecosystem. Data being made available through the YDS
platform can be picked up and distributed to the whole of Europe. On the other
hand, the European (Open) Data Portal ecosystem can provide access to data
that has not yet being identified as relevant. For instance, the Open Data
Support project which finished in December 2015, was handed over to the
European Data Portal [17] which offers access to more than 640,000 dataset
descriptions from all over Europe.

The core entities are Dataset and Distribution. The Dataset describes the data
and its usage conditions. Each Dataset has one or more Distributions, the
actual physical forms of the Dataset. A collection of Datasets is managed by a
Data Catalogue. The details are shown in Figure 4: DCAT-AP Overview.

As the DCAT-AP vocabulary is a Linked Data vocabulary, it fits naturally the
technological choices of the YDS platform. The vocabulary covers the majority
of the YDS data cataloguing needs. Any gaps or more specific needs, are
covered by the individual DMP instances.

The adoption of DCAT-AP creates also the availability of tooling. There is
EDCAT [18], an API layer to manage data catalogues, which has evolved into the
JSON API compliant interface of the YDS Open Data Repository (first MuDCAT,
then mu-cl-resources), a web interface [19], a validator [23], and the ODIP
platform [17] that harvests open data portals (based on an earlier version of
UnifiedViews [20], the central component of the YDS platform).

**Figure 4: DCAT-AP Overview**

#  Access, sharing and re-use policies

For a data platform, such as YDS, the access, usage and dissemination
conditions of the used source data determine the possible access, usage and
dissemination conditions of the newly created aggregated data. Despite the
sizeable amount of public open data that is available and imported, it is
likely to occur that there will be source data which is subject to
restrictions. When combining open data with restricted data, it cannot be
taken for granted that the resulting new data is open (or restricted). In such
mixed licensing situations, decisions will need to be made by the content
business owner and the data source owners concerning the accessibility of the
merged data. For example, it may be decided that some aggregated data is only
accessible for a selected audience (subscription based, registration based,
payment required or not etc.).

This context poses not only a business challenge, but also a technological
challenge. Some common practices when moving data from one source to another
may not be acceptable anymore. For example: if one data source A describes the
overall spending of a government by project and another data source B
describes the governmental projects and their contractors. The aggregated data
A+B provides thus insight in how the budget was spend by the contractors.
Merging the data into one aggregation usually makes it impossible to determine
from where the individual data elements came from. This is not problematic
when the aggregated data is subject to the same or more restrictive access,
usage and dissemination conditions as the source data themselves.

More complex and problematic is the situation where the aggregations are being
distributed throughout channels to audiences that do not satisfy the
conditions stipulated by one of the sources. To prevent incorrect usage,
managing the access, usage and dissemination conditions of the newly created
aggregations is important. That information forms the cornerstone of the
correct implementation of the required access, usage and dissemination
policies.

As shown above this aspect of the data management is a non-trivial work. Today
it is part of ongoing discussions. See the outcomes of the LAPSI project [14].
Therefore, YDS applies the following strategy:

  * The content business owner ensures that for each data source the access, sharing and reuse policy information is known. 

  

  * The content business owner decides whether the outcome of the integration & aggregation process is open (in all meanings = public, reusable, free of charge) or non-public (some restrictions apply). 

  

  * The data wranglers and system developers set up a data aggregation flow and data publication exposure according to the specification by the content business owner. 

  

  * The dataset meta-data of the created outcome is always public. This ensures transparency of the knowledge that is gathered within the YDS platform. The openness of the meta-data repository yields transparency. 

  

As the above situations already indicate, the situations that might occur may
be very complex. Our experiences gained in the first 24 months of the project
have shown that even open data sources can have conflicting licenses.
Therefore, our setup harvests and redistributes only open data that is free
for reuse, and we leave the licensing information intact (along with
appropriate provenance information) explicitly linked to each dataset. Since
each dataset is accompanied by a DCAT-AP entry, the base usage conditions get
registered. In doing this, the DCAT-AP record for a dataset becomes the key
reference point of the dataset decision making.

##  Data sharing

All collected data is shared via the YDS Open Data Repository (D3.10) as
**findable, accessible, interoperable and reusable (FAIR)** . The Open Data
Repository provides machine-readable means for accessing all YDS data through
multiple channels, along with the accompanying DCAT-AP descriptions. The DCAT-
AP descriptions allow for easy discovery and automatic harvesting by third
parties supporting the European application profile, such as the European Data
Portal  5  . Further technical and practical considerations and the
implementation of the data endpoints that are made accessible so as to
disseminate/share the YDS data with the public are described in the D3.10 Open
Data Repository v2.0 deliverable (to be followed by its final update in M32).

#  Long term data management and storage

The questions to be addressed concerning long-term storage are not new:
environmental datasets, medical testing datasets, component test results
relating to safety all have to be stored for a long time (the definition of
long-term being defined as part of a legal requirement, others will simply be
seen as being expected, e.g. datasets relating to academic published results).
These issues are complicated for when the data is made available over the
internet, in that the data could be merged with other data coming from other
sources, so the definition of meaningful long-term becomes problematic. So,
each content business owner needs to consider:

  * What is the volume of the data to be maintained? 

  

  * What is considered long-term (2-3 years, 10 years, etc.)? 

  

  * Identification of archive for long-term preservation of YDS data. 

  

  * Which datasets will need to be preserved in the archive? 

  

  * What about relevant dependent datasets? Snapshots of external datasets? 

  

  * Preserved datasets will need to be updated and this means a data preservation policy and process will need to be defined (and operational). 

  

A central consideration for any long-term DMP is the cost of preserving that
data and what will happen after the completion of the project? Preservation
costs may be considerable depending on the exploitation of the project after
its finalization. Examples include:

  * Personnel time for data preparation, management, documentation, and preservation, 

  

  * Hardware and/or software needed for data management, backing up, security, documentation, and preservation, 

  

  * Costs associated with submitting the data to an archive, 

  

  * Costs of maintaining the physical backup copies (disks age and need to be replaced). 

  

##  Practical considerations

Below, we outline and address the practical considerations with respect to the
above questions and long-term data management and storage.

###  Defining “long term”

From the perspective of a YDS content business owner, 4 years beyond the
lifespan of the project can be considered “long time”. From the perspective of
a server administrator, this is rather acceptable, and hence, the associated
costs of storage boil down to consumed energy and repair due to possible,
though unlikely, disk failure.

###  Data to be preserved

All datasets made publicly available through the YDS Open Data Repository are
to be stored long-term (i.e. all datasets with DCAT-AP descriptions).
Additionally, supporting datasets, such as relevant SKOS taxonomies, will also
be preserved during this period. User feedback received during the course of
the project, e.g. user evaluation forms, will be analyzed and reported in the
respective deliverables. Therefore, there is no need to preserve the original
forms after the end of the project. Social media content fetched by YDS
components will be preserved long-term in the form of links to the original
piece of content, e.g. links to tweets. This way no original social media
content will be saved during the period of the project and subsequently after
the end of it.

##  Technical considerations

Within WP5, a decision was taken to organize the YDS server, in order to
support development activities, i.e. accommodate a development Virtual Machine
(VM). The current YDS server setup is as follows:

**Table 2: YDS server setup**  
  
<table>  
<tr>  
<th>

</th>  
<th>

Development

</th>  
<th>

Production

</th> </tr>  
<tr>  
<td>

**CPU cores**

</td>  
<td>

7

</td>  
<td>

24

</td> </tr>  
<tr>  
<td>

**Memory**

</td>  
<td>

8 GB

</td>  
<td>

37 GB

</td> </tr>  
<tr>  
<td>

**Disk space**

</td>  
<td>

345 GB

</td>  
<td>

2.5 TB

</td> </tr> </table>

The development VM was initially a clone of the production one, meaning both
machines host Unified Views and OpenLink Virtuoso 7 instances, but the actual
amount of data at any given point in time might vary. During the initial
harvesting and transformation procedures, all data is stored on the
development server. As the data matures, it is transferred to the production
server.

###  Data volume

The triple store on either of the servers is not expected to consume more than
50 GB of disk space (backups included). Considering the graph nature of the
database, as the amount of data grows, so does its complexity, which is why
all additions to the Open Data Repository are considered with care and server
performance in mind.

###  Backups

Even though the data on the development server is never made public, both
servers have failsafe mechanisms in place for both data and the associated
harvesting processes. The data on both servers is backed up once a week on
local disk, i.e. every Tuesday, at 4 AM (i.e. outside peak hours). Moreover,
the harvesters are backed up on GitHub, in a dedicated repository  6  ,
ensuring fast recovery even in case of loss of data.

#  Risk management

In addition to all of the above discussed issues, a robust approach to data
storage and management needs to implement a range of practices to ensure data
is stored securely, particularly if it has been collected from human
participants. This means foreseeing the “worst-case scenario”, considering
potential problems that could occur and how to avoid these, or at least
minimize the likelihood that they will happen.

##  Personal data protection

Even though the project will avoid collecting such data unless deemed
necessary, encountering it is inevitable, and necessary measures must be
foreseen to avoid unauthorized leaks of personal information. Failing to
address this properly could consequently translate to breaching Data
Protection legislations and potentially result in reputation damage, financial
repercussions, and legal action. We foresee three potential sources of
personal data in YDS.

###  Platform users

The YourDataStories platform will provide the users with the possibility to
create their own accounts and data spaces. This means that even a minimum set
of essential user information might contain sensitive data (e.g. an e-mail
address).

###  Social media

Any user data on the social web is by default deemed personal. For the
YourDataStories project to deliver on the social-to-semantic and semantic-to-
social promise, without endangering user privacy, any information obtained
from the social media must be handled with care.

###  Evaluations with users

Even though it is undesirable, for some of the activities to be carried out by
the YDS project, such as platform evaluation via focus groups, it may be
necessary to collect basic personal data (e.g. full name, contact details,
background).

**Table 3: Personal data risk mitigation strategies**  
  
<table>  
<tr>  
<th>

Risk source

</th>  
<th>

Mitigation strategy

</th> </tr>  
<tr>  
<td>

**Platform users**

</td>  
<td>

To ensure none of the sensitive data is released to third parties, the
platform will leverage access control policies on an isolated, secure server,
providing only authorized users (data owners) and the YDS administrator with
access to such data. Furthermore, the user access credentials (passwords) will
be encrypted.

</td> </tr>  
<tr>  
<td>

**Social media**

</td>  
<td>

The YDS platform will never integrate any sensitive information collected from
the social networks in its data sets/streams permanently. Instead, the YDS
Data Layer will store and publish only anonymized information, or seek to
remove identifiable information at the earliest opportunity.

</td> </tr>  
<tr>  
<td>

**Evaluations with users**

</td>  
<td>

Such data will be protected in compliance with the EC's Directive 95/46/EC1
(General Data Protection Regulation)  7  aiming at protecting personal data.

</td> </tr>  
<tr>  
<td>

</td>  
<td>

National legislations applicable to the project will also be strictly
followed, such as laws 2472/1997 Protection of Individuals with regard to the
Processing of Personal Data  8  , and 3471/2006 Protection of personal data
and privacy in the electronic telecommunications sector (and amendment of law
2472/1997)  9  in Greece.

Any data collection by the project partners will be done only after providing
the data subjects with all relevant information, and after obtaining signed
informed consent forms. All paper consent forms that contain personal
information will be stored in a secure, locked cabinet within the responsible
partner’s premises.

</td> </tr> </table>

##  Undesirable disclosure of information due to linking

Being a Linked Open Data project YDS encodes all publishable information in
the form of an RDF graph. Although such an approach gives a clear edge to the
platform over its potential competitors in the market, its very nature bears a
certain degree of risk when it comes to unwanted disclosure of information due
to linking. This applies to both personal information and other private
information, either due to its nature or licensing limitations.

##  Linking by reference

An important advantage of LOD as a data integration technology, even in
enterprise use cases, is that it does not require physical integration.
Instead, it employs the _linking by reference_ principle, where it relies on
the resource identifiers (URIs) to _point_ to the data entry that is to be
integrated. This means that a public dataset can point to a resource in a
private one without disclosing any accompanying information.

Nevertheless, in YDS, special attention is paid to what data is triplified in
the first place. The data harvesters will collect, transform and store only
information which is already publically available, with the exception of
social media data which, as discussed above, will be anonymized so as to make
reidentification of individuals impossible.

**Note:** If there are concerns that certain data cannot be fully anonymized,
it will be made available only on condition that end users apply for access
and sign a Data Transfer Agreement indicating that they will not share the
data or attempt to re-identify individuals, assuming that no licenses are
broken by the YDS consortium in making such data available in the first place.

##  Risk of accidental stigmatization

All data with respect to organizations is published as-is, meaning the risk of
accidental stigmatization is inherited from the original data published, by
default (a provenance trail back to the original publisher is always provided
in the accompanying DCAT-AP description). However, during automatic,
supervised, and even manual data reconciliation, when interlinking with other
datasets, such as the OpenCorporates  10  database, there is a risk of false
positives. For this reason, all links between matching entities are via the
skos:closeMatch property, which does not guarantee semantic equivalence.
Moreover, the front end provides a disclaimer clarifying the nature of a link
whenever such information is present.

##  User-generated stories

It should not be forgotten that the YDS platform allows users to create their
own stories about the topic of their own choosing. Moreover, such stories may
contain data from external sources. These stories are stored on the YDS
platform and can be made public via YDS. However, it is worth noting that that
the contents of such stories, as well as the data not-originating from YDS,
are the sole responsibility of the user in the same way as any social media
platforms cannot take ultimate responsibility for what a member writes, save
to remove it when a valid objection is made.

##  Destruction of data

As sensitive, non-encrypted, digital data is never stored on disk, and
physically collected personal data (if any, e.g. during evaluations with
users) is stored securely, as explained in Table 3, data destruction is not
foreseen until the expiry of the long term storage period, as defined in
Section 6. Upon expiry, all data collected by YDS will be destroyed.

#  Conclusions

Applying & setting up a data management platform requires not only the
selection of the right technological components but also the application of a
number of best practice data management guidelines [12, 13] as given in
Section 2.3. Those best practices guide the users to the creation of
sustainable data sources in the web of data. Two of these best practices have
led to a concentration on two focal areas that required special attention:

 Dataset meta-data management both for the sources and the published
datasets, and  Data access considerations, sharing possibilities and re-use
policies and licenses.

In all this, the DCAT-AP dataset descriptions are a key requirement. Having
the dataset descriptions in a machine readable format creates potential on
effective traceability, status monitoring and sharing with the YDS target
audiences. Each DCAT-AP entry acts as a **machine-readable DMP instance** for
the dataset it describes, whereas human-readable DMP’s are given in the form
of DMP checklists (in the Annex of this report).

The core principles of the YDS project DMP have been presented from data
source discovery up to publishing of the aggregated content. The best
practices for publishing Linked Data – which is followed by YDS – describe a
data management plan for publication and use of high quality data published by
governments around the world using Linked Data. Via these best practices the
experiences of the Linked Open Data community are taken into account in the
project.

The technological foundations of the YDS platform separate very cleanly data
semantics, data representation and software development. Linked Data has made
the platform more flexible to implement the technological and data provenance
support which was required by the pilots as basic support. This ability is
unique in the data management technology space. Even though the deliverable
does touch upon the topic of tooling it must be noted that the actual software
is irrelevant for the discussion in this report.

This deliverable also extends the original DMP with respect to a number of
additional aspects. Now that the data, the model, and the platform are much
more mature, the DMP looks at the long term data management and storage
questions in more detail, and addresses them so as to provide a common
framework for all data collected and published by the project. In response to
new experiences and risks which arose in the second year of the project, we
also address additional risk management concerns, such as the management of
data collected from Social media, the destruction of data, as well as the risk
of accidental stigmatization of organizations and individuals (as a
consequence of automatic, supervised or manual interlinking).