New Storage Layout for Earth System Data (D4.2)
Creators: name: Luettgau, Jakob
name: Kunkel, Julian
name: Lawrence, Bryan
name: Jensen, Jens
name: Congiu, GIuseppe
name: Hua, Huang
name: Nassisi, Paola
Publisher: name: Zenodo
Year: 2017
Subjects: HPC
Earth simulation
storage environments
I/O middleware
earth system data

description: <strong>Abstract</strong>

Making the best use of HPC in Earth simulation requires storing and manipulating vast quantities of data.

Existing storage environments face usability and performance challenges for both domain scientists and the data centers supporting the scientists.

These challenges arise from data discovery/access patterns, and the need to support complex legacy interfaces. In the ESiWACE project, we develop a novel I/O middleware targeting, but not limited to, earth system data.

This deliverable sheds light upon the technical design of the ESD middleware, and the user perspective and implications when using the middleware.

Its architecture builds on well-established end-user interfaces but utilizes scientific metadata to harness a data structure centric perspective.

In contrast to existing solutions, the middleware maps data structures to available storage technology based on several parameters:

1) A data center specific configuration of available hardware with their characteristics;

2) The intended usage pattern explicitly provided by the user and implicitly by the structure of the data.

This allows to exploit performance characteristics of a heterogeneous storage environment more efficiently.

This deliverable provides the background on data representations and description formats commonly used in earth system modeling.

The document isolates the key requirements for an earth system middleware and collects numerous use-case outlining the benefit to existing and anticipated workflows and technologies.

Finally, a detailed initial design for the architecture of the earth system middleware is proposed and documented.

The document is not intended to describe all components completely but provides a high-level overview that is necessary to build a first prototype as it is planned in the next phase of the ESiWACE project.

During this development, the design will be adjusted to match the prototype; the final version of the design document will be delivered with the end of the project.

 

<strong>About this document</strong>

<strong>Work package in charge: </strong>WP4 Exploitability

<strong>Actual delivery date for this deliverable:</strong> July 2017

<strong>Dissemination level</strong>: PU

<strong>Lead author:  </strong>

Deutsches Klimarechenzentrum GmbH (DKRZ), Jakob Lüttgau

<strong>Other contributing authors:</strong>

Deutsches Klimarechenzentrum GmbH (DKRZ), Julian Kunkel

Science and Technology Facilities Council (STFC), Bryan Lawrence, Jens Jensen

The University of Reading (UREAD), Bryan Lawrence

Seagate Systems UK Limited (SEAGATE), Giuseppe Congiu, Huang Hua

Fondazione Centro Euro-Mediterraneo sui Cambiamenti Climatici (CMCC): Paola Nassisi
description: This project has received funding from the European Union's
Horizon 2020 Research and Innovation Programme
under Grant Agreement No 675191