#  Data Summary

An important objective of the NewsEye project is to develop methods and tools
for the effective exploration and exploitation of European newspaper data (see
_https://www.newseye.eu/_ ). This will be done using new technologies and “big
data” approaches, combining “close” and “distant reading” methods from the
digital humanities. This aims to improve the way of studying the European
cultural heritage not only for researchers and experts but also for the
general public.

To this purpose, NewsEye will collect data and metadata related to newspapers
from 3 European libraries (the National library of Finland, the National
library of France and the National library of Austria). The collected data
that will form the test cases represents about 1.5 million pages of newspapers
in 4 different language (Finnish, French, German and Swedish), covering the
period 1850-1950. This dataset consists of 19 different newspaper titles.

##  NewsEye workflow and input data

The NewsEye project follows a typical digitization workflow: Starting with
simple image files and some general metadata these images are processed with
layout analysis (LA), automated text recognition (ATR)  1  , article
separation (AS) and named entity recognition (NER). Furthermore, named
entities are linked with external sources (named entity linking) and enriched
with properties such as ‘stance’. Finally these data are used to set up
powerful ways to access the collection not only by searching, but in an
interactive and adaptive way which will take into account the full wealth of
the data. The simple image, together with some metadata is therefore the
starting point of the whole workflow. In our case also text data from former
ATR campaigns carried out by the participating libraries will be available as
a starting point and as a reference for comparing results achieved in the
project with the state of the art. This section describes in more detail which
data were available for the NewsEye project in the participating libraries and
how we tackled the collection process itself.

##  Data types and formats

The main objective of the NewsEye project is to make data openly accessible
via an open platform, which will be conforming to the linked data requirements
through the use of international standards (RDF, JSON-LD, IIIF, XML). This
data will correspond to the ground truth data and the results of the project.

As a matter of fact the digital library community was dominated during the
last 15-20 years by the XML based standards set up by the Library of Congress.
The most important are:

  * MODS: Metadata Object Description Standard 

  

  * METS: Metadata Encoding and Transmission Standard 

  

  * ALTO: Analyzed Layout and Text Object 

  

These standards are used worldwide, including the participating libraries in
NewsEye which preserve and manage their data by using these standards.
However, in the last years a new development was initiated by some well-known
libraries who gathered under the hood of the ‘International Image
Interoperability Framework’ (IIIF). The main difference to the conventional
XML schemas is the shift in the perspective: Instead of starting with the
concept of ‘meta data’ - as it is natural for analogue libraries – the image
itself, or in the notion of IIIF, the ‘canvas’ is the main focus.

At the NewsEye kick-off meeting in La Rochelle it became very clear among the
consortium members that the NewsEye project - especially in its role as being
a vanguard of future digital library applications - should go towards this
direction. This does of course not mean that METS/ALTO are outdated or no
longer useable, but that the main distribution format with which we would like
to describe data and meta data within the project will be IIIF and the web
annotation framework from W3C. In this way each of the tools, such as layout
analysis, text recognition and named entity recognition will provide an
additional annotation layer to the source document (body/target).

The NewsEye platform will then follow these recommendations in order to be
fully interoperable with existing tools and to ease the sharing of
information. The data and metadata produced by the NewsEye platform and which
will correspond to the results of the projects will be stored in an Apache
Solr platform built with Apache Lucene. Apache Lucene provides a Java-based
indexing and search technology, while the Solr tool provides a high-
performance search server with XML/HTTP and JSON/Python/Ruby APIs and many
other features.

Based on this architecture, the platform provides various API endpoints to
extract data in all the formats used by the partners (XML, JSON-LD, IIIF, XML)
based on the type of data and the international standards. Moreover, some
collaborations with the Impresso project are conducted in order to propose a
European standard to share contents of historical newspapers.

##  Data storage

In principle, the NewsEye platform will only host metadata, computed from the
original data, and make results available and harvestable by the national
libraries. However, in case the owner of documents or data provides no
adequate API, the documents will be stored in the platform repository,
together with the metadata.

To this end, a centralized metadata repository (named the NewsEye
Demonstrator) will be built following current standards in metadata vocabulary
(data types and formats presented below). The internal data representation
within the NewsEye Demonstrator is based on Samvera/FEDORA.

After the first year of the project, datasets are provided by the libraries,
enriched by Transkribus and several other tools, analyzed with dynamic text
analysis methods and made available to users again via the NewsEye
Demonstrator and the Personal Research Assistant as a distinct part of the
final demonstrator.

Data will be available within the lifecycle of the project via public access
provided by the contents owner through APIs. One particular point of the
project is that it will mainly deal with public data, hosted by national
libraries or in public repositories.

The NewsEye project is partly connected to the READ Project  2  (more
specifically with the Transkribus tool and team) and some results may be
shared between them. However, the major part of data will correspond to
digitized newspapers hosted by the national libraries involved in the NewsEye
project.

The volume of data that need to be stored on the demonstrator server varies
between the partners libraries. When a library publishes images through an
IIIF server, the only data that need to be stored and indexed are the text and
the associated enrichments produced by the project (named entities, article
separation, _etc_ ). The volume of the dataset used in the NewsEye project is
about 1TB. This number considers both the raw images, the raw ATR files and
the indexed data and metadata.

#  FAIR data

In the NewsEye project, partners have agreed to aim at making their research
data findable, accessible, interoperable and reusable (FAIR). This is why the
project participates in the open research data pilot (ORDP) set by the
European commission under the H2020 programme.

##  Data and results accessible to NewsEye consortium partners

Making results and data accessible is one of the main objectives of the
NewsEye project. Bearing that in mind, many tools and guidelines have been
planned in accordance with the description of the action. First, a global
digital library application has been created in order to store metadata and
create standardized content between partners. This repository will directly be
integrated in the project demonstrator: the NewsEye digital library
demonstrator. It provides access to data via a web interface and a platform to
utilise and visualize the results of all NewsEye tools. It uses an open
architecture, extensible via APIs for applications and plugins _._ On top of
that, it works with any kind of storage facility, using international exchange
standards in order to possibly be harvested by the national libraries and
easing the access to such data/metadata for each partner in the project.

For existing data, not generated within NewsEye, such as the image data from
the national libraries, the data owner/data provider will specify how data
will be shared and made available. These aspects will be continuously
elaborated upon in deliverables of WP1. Besides, this work package will
describe the global data model and the way to access data _._

##  Making data findable, including provisions for metadata

The data produced during the NewsEye project will be metadata, which will be
made available via the centralised metadata repository, which will be built as
part of the NewsEye web portal platform. Each newspaper issue imported in the
NewsEye platform is identified by a unique and persistent identifier. The
metadata generated during the project will cite not only this “NewsEye ID”,
but also the original ID coming from the libraries. Solutions for
sustainability will be examined in details as the core subject of task T7.4,
running from M13 to the end of the project. Moreover, a copy of all this data
will be accessible through the Zenodo platform at
_https://zenodo.org/communities/newseye/_ .

Metadata will be collected from the results of WP1 through WP6. Then, it will
follow the model and the quality requirements defined in WP1 and WP8. Later,
the metadata will be included in the NewsEye platform regarding the specific
features of WP7. All these metadata will be provided under an interoperable
format, following international standards (such as JSON-LD, Linked Data, OAI-
PHM, IIIF and XML). As stated on the previous section, metadata will be
produced under various formats depending on the need of each partner. In
practice, all data and metadata are stored in the Apache Solr server from the
NewsEye Platform. Apache Solr is highly reliable, scalable and fault tolerant,
providing distributed indexing, replication and load-balanced querying,
automated failover and recovery, centralized configuration and more.

Based on these international standards, the NewsEye platform will be fully
interoperable with all the partners of the project. This will make data
findable and easy to reach at an international level, allowing any institution
to harvest results from our project.

Finally, the source code of software and tools developed in the project will
be accessible through the different git repositories of the partners, to be
listed on the project Website.

##  Making data openly accessible

We want to make data, produced in the NewsEye project life cycle, easy to
reach. A focus will be made on research data produced in the project, for
instance its datasets (including especially training data / ground truth for
ATR raining data, NER, event detection, etc.). As all research papers will be
built upon these data, and in order to share these data with research groups
outside the consortium, we intend to rely on the Zenodo platform for making
them available, both during and after the project. In accordance with the
consortium agreement, the choice of license to apply on each dataset will be
discussed by all partners linked to the dataset. Following the recommendations
of the first monitoring meeting we have already published 2 datasets on Zenodo
for the training of text recognition engines  3  , and trained word embeddings
of changing vocabulary in English, Dutch, Swedish and Finnish over 1750-1950
4  . Other datasets are under preparation.

Raw data are already openly accessible as they are available through the
online platforms of the national libraries. The associated metadata will be
available in a centralized portal embedded on the NewsEye website (
_https://newseye.eu/_ ). Data will be linked with associated metadata. The
adequate subset of data (depending on institutional policy) will be made
available via the NewsEye website and later through a long-term storage
platform which will be defined within task T7.4 on sustainability.

Software developed under NewsEye, e.g., software tools for processing data or
automatically interacting with data will be deposited on code repositories
(such as github/gitlab). Restricted-use data, software and code are recorded
in the NewsEye grant agreement, and may vary according to institutional and
national policies and legislations. In case of restricted-use, metadata is
still provided so that we can still contact the data owner. The access request
will then depend on the data owner’s consideration, and full access to the
data may be granted.

The raw data studied in this project (both raw images and ATR files) will be
accessible through the NewsEye demonstrator either through a web browser or
programmatically through an API. Users are requested to create an account in
order to access the original data along with additional metadata created
during the life cycle of the project (better ATR, article separation, named
entities, topics, _etc_ ). Part of the original data is under copyright and is
thus restricted to project members. However, produced metadata will still be
shared to the community as NewsEye participates in the open research data
pilot (ORDP) and will share what is produced within its framework.

##  Making data interoperable

The NewsEye project aims to collect and document the data in a standardized
way. We must make sure that the datasets can be understood, interpreted and
shared in isolation alongside accompanying metadata and documentation. To this
purpose, the NewsEye Digital Library Demonstrator will contain all data
produced in the project and make them available in different ways and via
different channels to several user groups. Figure 1 details the data flow in
the project and illustrates how data are collected, exchanged and preserved
within NewsEye.

Figure 1: Data flow in NewsEye

The exact implementation of this system is expected for Y2 and Y3 where we
will be able to open up our workflows and tools towards interested libraries
or research groups also from outside the project. Some collaboration are
already been established with the Impresso project in order to get uniform
URLs and API at a European level. Such cooperation can be intensified also as
part of WP7 Demonstration, Dissemination, Outreach and Exploitation.

##  Increasing data re-use (through clarifying licenses)

Most data used in the NewsEye project already belongs to the project partners.
Following this, all existing data will keep their existing license, or a
license will be provided with such data.

Data collected under NewsEye will be made available for re-use upon completion
of the experiments. Data produced and made openly available under NewsEye will
be available for third parties, provided this does not contradict specific
rules, as specified in Section 2.3. In case of restricted-use, metadata is
still freely provided **,** enabling to contact the data owner. The request
will then be up for the data owner’s consideration, and depending on her
decision full access to the data may be granted. The data will be available
for at least 5 years after the conclusion of the project. Data quality
assurance will be covered in deliverables of WP8.

All metadata produced in the framework of the NewsEye project will be made
public, using appropriate licenses. The choice of the license will be defined
after discussion between partners involved in the production and management of
such metadata. Special attention is given to the sustainability of the
produced data and metadata, which will be made available on the Zenodo
platform in order to ease its reusability.

In any case, the rules set out and agreed upon within the consortium agreement
shall prevail.

#  Allocation of resources

On the one hand, the immediate costs anticipated to make the produced datasets
FAIR are related to hosting the NewsEye Demonstrator, which will be managed by
the University of La Rochelle (ULR) within task T7.1 of WP7. On the other
hand, a long-term deposit system for the datasets (data and metadata) will be
proposed within task T7.4 (for instance within the context of European
research infrastructures and/or through repositories such as Zenodo) for at
least 5 years after the conclusion of the project. It is noted that any
unforeseen costs related to open access research data in Horizon 2020 are
eligible for reimbursement during the length of the project under the
conditions defined in the Grant Agreement, in particular Article 6 and Article
6.2.D.3.

The costs for research data management are essentially covered through by
staff work within WP1 and WP7, overall estimated at 6PM. The infrastructure
and hardware for running the NewsEye demonstrator are run and powered through
the IT services of ULR and thus not covered by direct costs of the project.
When it comes to the resources required for sustainability, full details will
be developed from M13 within task T7.4. The current plan is to make datasets
and code available through open, free and sustained repositories such as
Zenodo and github, while the platform would be taken over by a European
research infrastructure (advanced contacts have notably been made with
DARIAH), and ideally by future subsequent projects in the vivid application
domain of historical newspapers, some of which are currently being proposed by
NewsEye partners.

The University of La Rochelle is responsible for data management within the
NewsEye project and specifically for creating and updating the present data
management plan. The contact person is Mickaël Coustaty.

Each NewsEye partner must follow the policies set out in this DMP. Datasets
have to be created, managed and stored properly and in accordance with the
European and national legislation. Dataset approval, metadata registration and
data archival and sharing through repositories is the responsibility of the
partner that generates the data.

The PIs of each partner will have the responsibility of implementing the DMP
in their institution.

#  Data security

The NewsEye project will be based on public archives hosted by the national
libraries. No data generated within the project is thus considered as highly
confidential. Thus, data security regulations are not deemed critical in this
project.

Following the completion of the project, all responsibilities concerning data
recovery and secure storage will be integrated with the dataset repository.
The centralised repository related to the demonstrator will be hosted by the
University of La Rochelle, which will archive and preserve them locally, using
daily

backup routines in operation under institutional policies **.** In details,
this server is managed by the IT services of ULR and follows the classic CIA
triad (Confidentiality, Integrity, Availability):

  * Confidentiality: This means that only people from our project will access to data and sensitive information (like logs) is accessed only by authorized persons (i.e. administrators of the server) and kept away from those not authorized to access them; 

  

  * Integrity: information will remain readable and correct. This will be implemented using hashing technics to ensure that data remains the same compared to previous backups; 

  

  * Availability: information and resources will remain available to those who need it. This part is managed by the IT services of ULR which provides a 99% availability of the infrastructures from the University through processes such as redundancy (RAID), Intrusion Detection System and DDoS protection 

  

Partners are expected to adopt suitable tested backup strategies enabling full
data recovery in case of an unexpected event. The responsibility for data
security and long-term preservation lies within the institutions. The server
used for the NewsEye platform includes a backup strategy managed by the host
provider. Moreover, in order to ensure the improving quality of project
results, a backup of the Solr Index will be made before each major update of
the data / metadata.

#  Ethical aspects

The NewsEye project will mainly deal with the enrichment of public data.
However, partners need to comply with the Ethics on research integrity as
described in the article 34 of the Grant Agreement.

Regarding the involvement of human participants, it will only occur for the
purpose of the demonstrator developed in task T7.1, where activity will be
logged to improve the performance of the personal research assistant. All
users will be informed of the data collection and its consequences. The
project strictly adheres to the General Data Protection Regulation (GDPR)
2016/679 of the European Parliament and of the Council of 27 April 2016, on
the protection of natural persons with regard to the processing of personal
data and on the free movement of such data  5  , as well as to national
regulations.

The full details on the implementations of ethics in NewsEye have been
delivered within the deliverables D9.1 to D9.5 of the Ethics work package
(WP9).

#  Further work

Data management procedures are visible to NewsEye partners. In the near-
future, standardization of data management will be one important part of the
DMP through the provision of data models as developed in task T1.1 of WP1. The
other part will consist in setting guidelines, such as for the demonstrator
and the quality assurance plan as developed in WP7 (task T7.1) and WP8 (task
T8.3).