# Presentation: 080706

##  C3-Grid* Federation System for Climate Data Handling 

**Stephan Kindermann**
- German Climate Computing Center – DKRZ

** ****C****ollaborative ****C****limate ****C****ommunity Grid Project (Part of D-Grid Initiative)*

##  C3Grid Overview: Architecture, Partners, Goals.. 

-  C3Grid Federation System Components:
-  C3Grid ISO Discovery Metadata and Metadata Catalog
-      A short interop. study: C3Grid ISO Metadata / Geonetwork
-  Data Access and *Preprocessing*
-  C3Grid Security

- _Overview_

-  C3Grid / IPCC ?

## C3Grid: Overview

- D-Grid
- (SRM,          d-cache,..)

**World Data Centers**

**Research Institutes**

- Climate

- Mare

- RSAT

- PIK

- GKSS

- AWI

- MPI-M

**Universities**

- FU Berlin Uni Köln

- Data Access Interface

- DWD

- ISO Discovery Metadata

- Data +

- Metadata

- Workflow

- Data +

- Metadata

- Grid Data / Job Interface

**ISO 19139**
**Discovery**
**Catalog**

- Result Data Products + Metadata

**C3Grid Data Providers**** **

**Collaborative Grid Workspace**

- (A)

- (B)

- ?

- !

- IFM-Geomar

- DKRZ

- Portal

- C3RC

## (A) Metadata for Data Discovery: Design and Implementation

- Data Access Interface

- ISO Discovery Metadata

- ISO 19139
- Discovery
- Catalog

**C3Grid Data Providers **

- (A)

- ?

## (A) Metadata – harvesting and lookup components

-  Fast Range Queries
-  Java API + Web Service Interface

-  made available on sourceforge.net
-      see also: http://www.panfmp.org

-  Technology
-     ISO 19115/19139 metadata profile
-     OAI-PMH harvesting  catalogue
-     lucene based catalogue search
-     GridSphere based portal

## (A) C3Grid ISO 19139 profile

- Design criteria:
- no schema extensions, profiling by restriction
- restriction using schematron constraints
- „the granularity of the discovery metadata should reflect the logical organization of the data repository at a sufficiently coarse grained level“ (1)
- CF based content description
- Link to resource metadata infrastructure
-     (GT4-MDS based)

- (1) Inspire: DT Metadata – Draft Implementing Rules for Metadata (version 2, 02/02/2007)

## (A) C3Grid ISO Profile

- Description at aggregate level (e.g. experiment)
-  Aggregate extent description
-      with multiple verticalExtent sections
-  Sub-selection in data request

## (A) C3Grid ISO Profile: CF usage

- Content description based on (extended) CF names
-  Link to corresponding vertical CRS

## (A) C3Grid ISO profile

- Data Distributor Info:
-  reference to C3Grid resource metadata catalog (MDS) (names  service endpoints)
-  (optional: service endpoints)

## (A) C3Grid ISO profile

- Data provenance description:
-  by now (data staging output): simple sequence of ProcessStep descriptions
-  later (c3grid processed data): combined Source/ProcessStep blocks + external data provenance store

## Slide 11

## C3Grid ISO Profile: A short geonetwork experiment

- Federation building:
-  OAI-PMH, WebDAV, Z39.50, geonet
-  Full ISO metadata support (ISO19139/19119)
-  OGC CSW 2.0 reference impl.
-  RSS and GeoRSS newsfeeds
-  SKOS based thesauri

-  adaptable to new schema`s
-  schematron constraint checking
*On roadmap:*
-  flexible ISO profile support
-  shibboleth integration

## C3Grid ISO Profile: A short geonetwork experiment

## Building complex metadata federations ...

- Harvesting via:
-  CSW
-  OAI-PMH
-  Geneonet
-  Web-Dav

## C3Grid ISO Profile: A short geonetwork experiment

- Import / Edit / Search: ok
- Missing:
-  content (CF) search
-  vertical search
-  temporal BBox search
-  data staging

## 
 complete portal protoype to seach, access (pre-process) data described by C3Grid ISO profile in 3 weeks based on geonetwork open source solution ..  

-  complete portal protoype to seach, access (pre-process) data described by C3Grid ISO profile in 3 weeks based on geonetwork open source solution ..

## Slide 19

## (B) Data Access and Preprocessing

- World Data Centers

- Research Institutes

- University Partners

- Data Access Interface

- ISO Discovery Metadata

- Data +

- Metadata

- Data Analysis
- Workflow

- Data +

- Metadata

- ISO
- Discovery
- Catalog

- Result Data Products + Metadata

- C3Grid Data Providers

- Collaborative Grid Workspace

- (A)

- (B)

- ?

- !

## (B) Data Access and Pre-Processing: Implementation

- Data Staging Request

- Data IDs

- Output Properties

- Selection:
-  lon, lat, alt
-  time
-  content: CF

- Data Staging Web Service

- DB

- Flat
- File

- Archive

- Distributed C3Grid Work Space

- Offer
- Time / resource
- estimation

-      skeleton impl

- status
- ..

- Provider staging jars

- Provider staging scripts

- MD DB

- WS GRAM

- JSDL based
- description

- Processing jobs

- Local resource manager

-  C3Grid Generation 1: secured plain web services
- (status)
-  C3Grid Generation 2: WSRF service interfaces (scheduled november 08)
-  Generation 2+: full PKI/SAML security stack

## C3Grid Middleware Components

**Scheduler:** Globus WSRF based, accepts WSL workflow description: compute tasks + data staging tasks

**Datamanagement:** Globus WSRF based, offer negotiation with scheduler, consistent view to distributed data, (later: *replica management, caching)*

- Globus MDS Resource Metadata Catalog: service registry, resource status

-  Dependency on Globus SW stack, no high level impl. support tools, impl. Globus 4.1.x migration ??, problems with delegation impl. (insufficient docu. and guidance)

## C3Grid Workflow Analysis

- analysis and preparation of workflows

- monitoring and management of workflow execution

- (individual) scheduling strategy to optimize the management

- Handler to facade single/ specific Tasks

- interaction an moitoring via WS Notification standard

## (C) Security Infrastructure

- Identity Provider
- Home Organisation

- Attribute Provider
- Virtual Organisation

- MyProxy

**Delegation Service**

**Grid Service**

**Grid Service**

- Grid
- Resource

- GRAM / DataRAM

- C3Grid Middleware

- GridShib
- SAML tools

- wflow
- client

- SLCS
- (CA)

- X509 Grid-proxy

- GridShib for GT

- policy

**Portal**

- <..SAML
- Assertions..>

- SAML

- SAML

- SAML

- SAML

- Personal /
- Group
- Account

- „Home attributes + VO attributes“

- DFN

- Browser

- Webstart
- app

- Shibb.
- login

## (C) Security Infrastructure

**Status:**
-  Shibb IdP`s running at core C3Grid partners
-  Online CA for short-lived credentials tested, set up & operated by DFN (the German NREN)
-  Online CA (DFN-SLCS) accreditation process with EUGridPMA started
-  SLC contain campus attributes as SAML assertion
-  Java Webstart app to bootstrap SLCS in development at DFN
-  GridShib SAML Tools (v0.6.0) tested
-  Prototype of shibbolized GridSphere portal tested
-  open issues with GT4 proxy-delegation implementation
**Next:**
-  Integration of components
-  Virtual home organization for C3 users without a Shibboleth IdP
-  Integration of VO attributes (shibbolized VOMS)

## C3Grid / IPCC Use Case

- (0) IPCC Metadata harvested / mirrored in CERA DB (WDCC)
-  Metadata visible in C3 Portal
-  User issues IPCC data import from external repository
-  User  OpenID IdP / + IPCC_Access role  external repos
-  Download  ??  C3 Repository
-  C3Grid grants access to users with IPCC_Access role
-       ‘grant procedure ?’: before each wflow exec. contact to IdP/AttributeService ?? or more offline method ?

- C3RC / C3 Workspace

- IPCC data import

- Wflow result publication

- Analysis wflow

## Appendix

## C3Grid Content Info (Version 2)

- <contentInfo> <MD_CoverageDescription>
- <attributeDescription> <gco:RecordType>
-      CF_name_with_attribute
- </gco:RecordType> </attributeDescription>
- <contentType> <MD_CoverageContentTypeCode codeList=_["http://wis.wmo.int/2006/catalogues/cf-standard-name-table.xml"](http://wis.wmo.int/2006/catalogues/cf-standard-name-table.xml)_
-      codeListValue="air_temperature">
-         air_temperature with a cell_methods attribute including time:mean (interval: 1 day)
- </MD_CoverageContentTypeCode> </contentType>
-  <dimension **xlink:href="****#verticalCRS_hPa****"**><MD_RangeDimension><descriptor>
-         <gco:CharacterString>K</gco:CharacterString>
-  </descriptor></MD_RangeDimension</dimension>
- </MD_CoverageDescription> </contentInfo>

## Security Aspect: C3Grid step 0  step 1 

## Slide 30

## Slide 31

##  Processing history 

- Index

- (C) Data Reuse of Analysis Results: Metadata Generation

- OAI-Harvester

- WS Interface

**C3Grid Workspace**

- wflow

- m_tool

- OAI-PMH Server

- Portal

- “quality check”

- API

- Prototype (Python)

- Context description of
- Analysis Data:

-  Aggregation
-  Processing history