# Presentation: 080707

## Distributed data access and processing in the C3-Grid* project 

**Stephan Kindermann**
- German Climate Computing Center – DKRZ

** ****C****ollaborative ****C****limate ****C****ommunity Grid Project (Part of D-Grid Initiative)*

##  C3Grid Overview: Partners, Goals.. 

-  C3Grid Data Networking Components:
-  ISO Discovery Metadata and Metadata Catalog
-  Data Access and *Preprocessing*
-  *Data Processing* and Metadata for Data Reuse
-  Interoperability
-  First Experience with EGEE collaborative environment
-  C3Grid as part of an interoperable environment

- _Overview_

## C3Grid:  Partners and Main Components

**World Data Centers**

**Research Institutes**

- Climate

- Mare

- RSAT

- PIK

- GKSS

- AWI

- MPI-M

**Universities**

- FU Berlin Uni Köln

- Data Access Interface

- DWD

- ISO Discovery Metadata

- Data +

- Metadata

- Data Analysis
- Workflow

- Data +

- Metadata

- Grid Access Interface

**ISO 19139**
**Discovery**
**Catalog**

- Result Data Products + Metadata

**C3Grid Data Providers**** **

**Collaborative Grid Workspace**

- (A)

- (B)

- (C)

- ?

- !

- IFM-Geomar

- DKRZ

- Portal

## C3Grid: Goals and Approaches

-  **Uniform Discovery of German climate related data providers**
-     ISO 19115/19139 metadata profile
-     OAI-PMH harvesting of metadata
-     Central metadata index for search
-     GridSphere based Portal

** ****Uniform data access and flexible *****data pre-processing***
-    Common data request web service
-    Grid based data delivery

**  ****Data processing and data reuse**
-    Portal integration of data processing workflows
-    Grid workspace with data/job co-scheduling
-    (Discovery-) Metadata generation part of workflows

## (A) Metadata for Data Discovery: Design and Implementation

- Data Access Interface

- ISO Discovery Metadata

- Data +

- Metadata

- Data Analysis
- Workflow

- Data +

- Metadata

- ISO 19139
- Discovery
- Catalog

- Result Data Products + Metadata

**C3Grid Data Providers **

- Collaborative Grid Workspace

- (A)

- (B)

- (C)

- ?

- !

## (A) C3Grid Metadata Profile of ISO 19115

**C3Grid Content Description (CF Variables)**

**C3Grid Access Description**

**C3Grid Data**
**Provenance Description**

**C3Grid **
**ISO 19139 **
**XML **
**Description**
**(ISO 19115)**

- Metadata

- Model output data

- .

- .

- .

- .

- .

- Observation data

- Analysis data

- DKRZ
- Archive

- CERA
- DB

- PIK
- Flat File

- Distributed C3Grid Storage

- Pangaea
- Data warehouse

- DWD
- DB

- C3Grid
- Flat File

- DLR

- ...

**C3Grid Profile**

**Common Syntax: **
**ISO 19139**

**Common Interpretation: **

- Discovery
- Metadata

- Discovery
- Metadata

- Discovery
- Metadata

- (WDC Climate)

- (WDC Mare)

- (WDC RSAT)

## (A) Discovery Metadata Harvesting

- Lucene+
- Index

- OAI-Harvester

- WS Interface

- OAI-PMH Server

- Portal

- ISO
- 19139
- XML
- description
- (ISO 19115)

-  Status:
-  Prototype ready
-    (Data providers: DKRZ, WDC Climate,
-       IFM-Geomar, PIK, GKSS) ,
-   about 300 MByte (30.000 XML Instances)
-     ISO Metadata files
-   describing > 100 TByte of data
-     accessible through C3 Portal

## Slide 8

## C3Grid Metadata Harvesting Approach

**web service **
**frontend**

**Portal**

**CERA**

**PANGAEA****®**

**Other **
**Data **
**Provider**

- Google-style and
**range queries**

- DIS

- Metadata1.xml, Metadata2.xml,
- Metadata3.xml, Metadata4.xml,
- ...

| Field | Term | Document |
| --- | --- | --- |
| identifier | ABC:123 | 2 |
| identifier | XYZ:223 | 6 |
| identifier | MI6:007 | 12 |
| abstract | region | 2,23,112 |
| abstract | pressure | 3,23 |
| abstract | humidity | 4,33,215 |
| min_lat | 030.43 | 1 |
| min_lat | -023.23 | 2 |
| data_uri | http://... | 4 |

- Apache Lucene index

- document cache

- indexing
- of
- selected
- fields

- OAI-PMH

**full-text**
**index**

**harvesting**
**backend**

## Central Index: Generic Framework

- From: U. Schindler Univ. of Bremen

-  Soon available on sourceforge.net !

-  Fast Range Queries
-  Java API + Web Service Interface

## (B) Data Access and Preprocessing

- World Data Centers

- Research Institutes

- University Partners

- Data Access Interface

- ISO Discovery Metadata

- Data +

- Metadata

- Data Analysis
- Workflow

- Data +

- Metadata

- ISO
- Discovery
- Catalog

- Result Data Products + Metadata

- C3Grid Data Providers

- Collaborative Grid Workspace

- (A)

- (B)

- (C)

- ?

- !

## (B) Data Access and Pre-Processing: Implementation

- C3 Grid Data Request Web Service

- DB

- Flat
- File

- Archive

- GridFTP Server

- C3Grid Work Space

- High variability of implementation complexity at individual data providers:
- simple DB call, no preproc.   access and preprocessing of terabytes of data .. !!

- Metadata
- generation

## (C)  Data Processing and Reuse

- World Data Centers

- Research Institutes

- University Partners

- Data Access Interface

- ISO Discovery Metadata

- Data +

- Metadata

- Data Analysis
- Workflow

- Data +

- Metadata

- ISO
- Discovery
- Catalog

- Result Data Products + Metadata

- C3Grid Data Providers

- Collaborative Grid Workspace

- (A)

- (B)

- (C)

- ?

- !

## (C) Data Processing

- Lucene+
- Index

- OAI-Harvester

- WS Interface

**C3Grid Workspace**

- XML

- XML

- wflow

- m_tool

- OAI-PMH Server

- Portal

- “quality check”

- GT 4 WS-GRAM

- job

- data + metadata

## Interoperability !?

##  Interoperability: Interfacing C3Grid

- Data

- repository

- OAI

- -

- PMH

- server

- ISO

- 19139

- Metadata

- Data

- Request

- WS

- WN

- WN

- compute

- grid

- workspace

- GridFTP

- server

- Adapter to

- propriatary

- (

- meta

- -

- )data

- access

- solutions

- data

- providers

- Data

- repository

- OAI

- -

- PMH

- server

- ISO

- 19139

- Metadata

- Data

- Request

- WS

- WN

- WN

- compute

- grid

- workspace

- GridFTP

- server

- Adapter to

- propriatary

- (

- meta

- -

- )data

- access

- solutions

- data

- providers

- OAI-PMH Harvesting Interface

- Data Request (Staging) Web Service Interface

- GT 4 Job Submission WSRF Interface

- C3Grid Workflow Submission Web Service Interface

- GridFTP Data Access Interface

- Well defined Interfaces:

- Data Discovery Web Service

##  Interoperability: Extending C3

-  Interoperability: Extending C3

-  Make external environment a C3Grid Data Provider
- Provide OAI-PMH server / make ISO metadata available
- Implement Data Request Web service
- Provide GridFTP Data Access

- (2) Provide Job Submission Interface
- Implement Data Request Web service triggering processing
- (Wrapper to translate C3Grid job descriptions and use general job submission interface (e.g. gLite / GT4 ..))

- Job
- Interface

- Prototype following this approach to interface EGEE infrastructure demonstrated
-  EGEE Conf. `06 demo: no portal integration, basic functionality
-  EGEE User Forum `07 manchester:  AMGA metadata server importing and exporting ISO 19139 xmls + portal
-  EGEE rehearsal demo `07 Geneva

- C3Grid

## Data Selection

## Data Access and Data Filtering

## Data Processing on EGEE: 

- Lucene+
- Index

- Data Processing on EGEE:

- OAI-Harvester

- WS Interface

**C3Grid Workspace**

- XML

- XML

- wflow

- m_tool

- OAI-PMH Server

- Portal

- “quality check”

- API

- Prototype based on AMGA and MySQL

- GT 4 WS-GRAM

**EGEE Infrastructure**

- AMGA
- Metadata
- Catalog

- ISO
- Fill

- Update interface

- ISO
- Export

- wflow

- GLite WMS

- job

- data + metadata

- job

- data

- metadata

## Data Processing Workflows

## Data Discovery

- OAI

- -

- PMH

- server

- ISO

- Data

- Request

- WS

- grid

- GridFTP

- server

- OAI

- -

- PMH

- server

- Data

- Request

- WS

- GridFTP

- server

- C3Grid

- C3

- Portal

- C3Grid

- DMS / Scheduler

- MDS

- Data Portal

- 1.b

-     1.a  OAI-PMH harvesting of ISO 19139 descriptions, or
-     1.b  call of C3Grid discovery web service interface, or
-     1.c  link to C3 Portal

- 2. Data Access / Generation of derived data products
-     2.a  call C3Grid workflow interface (data access + processing)
-       or  call data request web service at data provider (data access + preproc.)
-     2.b     gridFTP data delivery

- Interoperability: C3Grid as an „intelligent“ data provider

- 1.a

- 1.c

- EGEE /
-  D-Grid

- 2.a

- 2.b

## Open Issues / Next Steps 

- Common interoperable security (AA) infrastructure
- First version of Shibboleth enabled portal,  Shib. IdP at data providers
- Next: Grid Security Integration ( GridShib)
- Extend data basis / Integrate user experiences
- Metadata handling tools to ease metadata generation
- Service description (e.g. based on ISO 19119)
- De-composable workflows, workflow information service

## Appendix ....

## Security Aspect: C3Grid step 0  step 1 

##  Processing history 

- Index

- (C) Data Reuse of Analysis Results: Metadata Generation

- OAI-Harvester

- WS Interface

**C3Grid Workspace**

- wflow

- m_tool

- OAI-PMH Server

- Portal

- “quality check”

- API

- Prototype based on AMGA and MySQL

- Context description of
- Analysis Data:

-  Aggregation
-  Processing history

## (A) + (B) + (C): Implementation of Data Networking Components

- data

- metadata

- GridFTP
- Interface

- LuceneIndex

- Harvesting
- Interface

- Harvester

- Portal

- C3Grid Data Management
- / Scheduling Middleware

**Distributed C3Grid Workspace Resources**

- Analysis / Control

- ISO metadata

- transformation

- Data Request
- Interface

- Data Provider

- Harvesting
- Interface

- Preprocessing

- data

- OAI-PMH (open archives initiative protocol for metadata harvesting)

- WSDL defined community data request web service

- wflow

- WFlow Providers

- Resource Providers

- Data Providers

## EGEE Interoperability: Metadata and Data Access

- Data

- repository

- OAI

- -

- PMH

- server

- ISO

- 19139

- Metadata

- Data

- Request

- WS

- WN

- WN

- compute

- grid

- workspace

- GridFTP

- server

- Adapter to

- propriatary

- (

- meta

- -

- )data

- access

- solutions

- data

- providers

- Data

- repository

- OAI

- -

- PMH

- server

- ISO

- 19139

- Metadata

- Data

- Request

- WS

- WN

- WN

- compute

- grid

- workspace

- GridFTP

- server

- Adapter to

- propriatary

- (

- meta

- -

- )data

- access

- solutions

- data

- providers

- C3Grid Portal

- AMGA
- Metadata
- Catatalog
- (for ISO
- 19139)

- Storage
- Element

- Replica
- Catalog

- WNs

- WNs

- WNs

- CE

- A
- P
- I

-     UI

-  EGEE Demo Geneve 06
-  EGEE Demo Manchester 07

- A first protoype:

## C3Grid ISO 19115 / 19139 Profile

- <MD_Metadata .. http://www.isotc211.org/2005/gmd/gmd.xsd">
-    <fileIdentifier ../>
-    **<hiera****r****chy****L****evel ../> <parentIdentifier ../>**
-    <contact ../>
-    <dateStamp ../>
-    <identificationInfo>
-         **<****abstract ****../>**
-         <resourceFormat ../>
-         **<descriptiveKeywords  ../>**
-         <environmentDescription .. />
-         **<exten****t****> **
**              ****<geographicElement**** ../****>**
**              ****<****temporal****Element**** ../****>**
**              ****<****vertical****Element**** ../****>**
**        ****<exten****t/****>**
-    <identificationInfo/>
-    **<contentInfo>**
**      ****  ****<attributeDescription ..*****CF standard names*****..****/> **
**   ****<contentInfo****/****>**
-    **<distributionInfo**** *****..online*****,*****(******data****** ******item size, name******)***** ****..****/>**
-    **<d****ataQuality****Info**** *****..***
***                   *****lineage***** (history functionality).***
***                   *****source ***** (data ancestors)***** /****>**
-    <series> <DS_Series>
-          <composed_of>  .. <composed_of/>
-     <DS_Series/> <series/>
- </MD_Metadata>

**C3Grid ISO ****19115 / ****19139 Profile**

- aggregate

- C3Grid data request and delivery references

- Climate parameters  (physical content)

- Unique identification of resource including geographic, temporal, vertical extent

- C3Grid data processing history and data ancestor information

- Aligned with WMO ISO profile (v0.3)

- Fine granular decomposition (optional)

## Current situation: heterogenity hinders scientific work

- Collect & Prepare

- Visualize

- 4

- Analyse

- Find & Select

- Model Data

- Observation Data

**Analysis Dataset**

**Result Dataset**

- Scenario data

- 3

- 2

- 1

**„****I want to correlate *****model data***** from DKRZ with *****observation data***** from DWD and *****satellite data***** from DLR“**

** ****Contact each data provider**
** ****Learn their data search utilities**
** ****Find and select data**

** ****Get access rights for datasets at each data provider**
** ****Learn their data access / preprocessing services**
** ****Get access to sufficient storage facilities**
** ****Trigger preprocessing and download data**

** ****start analysis tools**
** ****produce undocumented data**

** ****copy to local resources**
** ****create visualization**

**„****has somebody done **
** ****similiar things i want ?? **
**Can i reuse data for ...??“**

## Motivation: Data handling challenges

- Data Collect & Prepare

- Visualize

- 4

- Analyse

** **
**Distributed Climate Data**

- Model Data

- Observation Data

**Analysis Dataset**

**Result Dataset**

- Scenario data

- 3

- B

**(A) find appropriate data **
-  annotation of data
-     (metadata)

- Metadata Collect & Prepare

**(B) data access**
-  distribution and heterogenity of
- data sources
-  size of data

- Data descriptions

- A

**(C) Data analysis**
-  data reuse
-  data management