D9.2 Service deployment in computing and data e-Infrastructures Version 2
Creators: name: Chen, Yin
name: Haggström, Ingemar
name: Buck, Justin
name: Stocker, Markus
name: Carval, Thierry
name: Vitale, Domenico
name: Huber, Robert
name: Hellström, Margareta
name: Candela, Leonardo
name: Haslinger, Florian
Publisher: name: Zenodo
Year: 2018

description: This document reports on implementation results of T9.1 use cases for testing and validating<br>
ENVRIplus Theme 2 service solutions. It is an update of the previous report D9.1 that was<br>
submitted on M28. In D9.1 we explained the approach to the service integration and validation.<br>
The driver was to define appropriate use cases that address community’s real needs, integrate<br>
ENVRIplus Theme2 services as part of the service solutions, and implement those use cases<br>
following an agile approach.

This document describes 7 well-developed use cases that are selected as the final Science<br>
Demonstrators. By Science Demonstrator we mean “a showcase of a service solution illustrated<br>
through a prototype implementation, which serves as proof or evidence that the Theme2<br>
services can bring added value for supporting ENVRIplus community to deliver scientific<br>
research”.

Science Demonstrator 1 addresses a requirement of the EISCAT RI community, namely to allow<br>
individual scientists to process their experimental data using their own algorithms. The challenge<br>
is common to many ENVRIplus RIs, where data is often processed using standard models and<br>
methods. As researchers want to use different analysis models, easily modify parameters or<br>
algorithms, and collaborate with each other, they need a Virtual Research Environment (VRE).<br>
This demo showcases a model making use of the D4Science gCube platform developed by T7.1,<br>
which enables scientific researchers to re-process data by implementing and adapting algorithms<br>
and parameters from other sources.

Science Demonstrator 2 showcases a novel implementation of a computationally efficient tool<br>
for processing of Eddy Covariance (EC) data which offers to users the possibility to calculate EC<br>
fluxes through the EddyPro® software (LI-COR Biosciences, 2017; Fratini and Mauder, 2014)<br>
according to 4 processing schemes resulting from a different combination of existing methods.<br>
To reduce the computational runtime required, the 4 processing schemes were implemented<br>
and executed in parallel mode. The whole service setup including a metadata management<br>
algorithm, was implemented and tested in the D4Science gCube Virtual Research Environment<br>
provided by Task 7.1, and the final computational runtime for Near Real Time (NRT) processing<br>
(i.e. flux estimates based on raw data collected the previous day) is of about 4 minutes, similar to<br>
those required for a standard run involving only a single processing scheme.<br>
Science Demonstrator 3 addresses a common problem for ENVRIplus RIs (specifically<br>
observatories that build on environmental sensor networks) that data acquisition service, in<br>
particular, the preparation of data transfer prior to data transmission are often not yet<br>
sufficiently standardized. This hinders the operation of efficient cross-RI data processing<br>
routines, e.g., for data quality checking. The demonstrator showcases a service prototype that<br>
allows submitting and publishing raw observational (non-geophysical) environmental time series<br>
data in common standard formats (T-SOS XML and SSNO JSON). A messaging API (EGI ARGO) is<br>
used to perform Near Real Time (NRT) quality control procedures by an Apache Storm NRT QC<br>
Topology, which publishes the quality controlled and labelled data via a messaging output<br>
queue.

Science Demonstrator 4 describes the EuroArgo Data Subscription Service (DSS) that allows<br>
researchers to subscribe to customized views of Argo data, selecting specific regions and time<br>
spans, and choose the frequency of updates. Tailored updates are then provided on schedule to<br>
researchers’ private storage. The demo showcases an integration solution that combines the<br>
EuroArgo community data portal with e-Infrastructure services (EUDAT B2SAFE, EGI FedCloud,<br>
etc.), and uses the DRIP service developed by T7.2 for optimised service deployment. The pilot<br>
activity was initiated by the marine research community, however, the possibility to receive<br>
regular transmissions of data, especially in near-real time, directly from the organisation<br>
responsible for data collection and (pre-)processing, is very important to many large initiatives.<br>
ENVRIplus RIs can benefit from the subscription services, e.g., to create more elaborated data<br>
products by requesting data from other sources, and can optimise their internal workflows by<br>
signing up for automatic updates.<br>
Science Demonstrator 5 showcases a “sensor registry” that aims at supporting the management<br>
of sensors deployed for in-situ measurements. Common sensors or families of sensors are used<br>
across different research infrastructures, for example, oxygen optodes that are equipped on<br>
platforms in multiple research infrastructures. The goal of this work is to define common<br>
methods to access the sensor metadata in such cases. The sensor registry applies the design<br>
principle of data catalogue developed in WP8, and uses data technologies and standards from<br>
the OGC Sensor Web Enablement family including SensorML, Observations and Measurements<br>
(O&amp;M), and Sensor Observation Service (SOS). It brings together a marine domain<br>
implementation of these standards (the Marine SWE profile) developed by several European<br>
projects demonstrating the viability for future sensor and observation activities. The service can<br>
be integrated to various types of platforms, deep-sea observatories (e.g., EMSO), marine gliders<br>
(e.g., EuroGOOS) as well as solid earth (e.g., EPOS) or atmosphere observations (e.g., ICOS). It can<br>
also be used to track usage of specific sensor models (e.g., CO2) across the RI ‘s observation<br>
networks.<br>
<br>
Science Demonstrator 6 describes a service prototype that supports aerosol scientists in<br>
studying new atmospheric particle formation events by moving data analysis from local<br>
computing environments to interoperable infrastructures, thus harmonizing data analysis itself<br>
and more importantly the syntax and semantics of data derived from analysis. As researchers<br>
interpret primary data and thus gain information and transfer information into knowledge, we<br>
are studying and advancing in particular some technical aspects of a knowledge infrastructure<br>
i.e., a robust network of scientists, artefacts such as virtual research environments and research<br>
data, and institutions such as research infrastructures and e-Infrastructures that acquire,<br>
maintain and share scientific knowledge about the natural world. The science demonstrator<br>
showcases a possible architecture of a socio-technical infrastructure that “transforms data into<br>
knowledge.” The proposed approach highlights a range of novel possibilities, in particular<br>
enabling researchers to focus on data analysis and interpretation while leaving data access and<br>
transformation from and to systems to interoperable infrastructure. It significantly contributes to<br>
implementing the global agenda of FAIR data by promoting the notion of “FAIR by Design”,<br>
weaving data FAIRness into the fabric of infrastructures. It builds on the principle not to leave<br>
making data FAIR to researchers but to guarantee it by design of well-engineered infrastructures.<br>
The demonstrator is first and foremost of primary interest to a specific scientific community,<br>
namely the various aerosol research groups that study new particle formation events.

Science Demonstrator 7 illustrates how a LifeWatch researcher can easily upload and integrate<br>
an analysis algorithm in D4Science, and share it with other researchers in a VRE. The use case<br>
proposed an integration solution that links the D4Science/gCube VRE to the LifeWatch RI and to<br>
the EGI e-Infrastructure. This integration, for example, enables individual researchers to repeat<br>
and reuse algorithms at will, run trend analysis, and add new parameters and custom data. The<br>
VRE provides provenance registration that improves reproducibility and also allows retention of<br>
computation results in the user’s workspace. This facilitates editing and adaptation of<br>
algorithms, features that are not provided by the existing LifeWatch ICT.
description: Slightly updated from the version 2.0 (which was delivered as official report to the Commission).