# Presentation: 222876

## Development of a Prototype NOAA Gridwww-ad.fsl.noaa.gov/ac/schaffer/ngi.html

- Dan Schaffer, Paul Hyder Mark Govett (FSL)
- Chris Moore
-  (PMEL)
- Brian Gross, Ron Bewtra
- (GFDL)
- May 2005

## Objectives and Milestones

- Construct a prototype NOAA grid (mostly complete)
- Develop a NOAA certificate authority (partially complete)
- Develop a grid meta-scheduler (rudimentary version completed)
- Execute the ROMS, WRF and CM2 models on the NOAA grid machines (completed)
- Couple WRF/ROMS across
- NOAA grid (failed)
- TeraGrid (in progress)

- A Prototype NOAA Grid

## Expenditures

- FSL    – 2/3 of the combined HPCC and base funds spent
- PMEL – 2/3 of the funds spent
- GFDL - ???

- A Prototype NOAA Grid

## NOAA Grid Currently

**Ijet**

**FSL**
**Firewall**

**AD Cluster**

**Ijet Batch**
**Server**

**Beowulf**
**Cluster**

**Beowulf**
**Batch Server**

**GFDL**
**Firewall**

**Grid**
**Scheduler**

**Oshun**
**(PMEL)**

**AD Batch**
**Server**

- Glogin
- Gftp

- Gsubmit URL

- A Prototype NOAA Grid

## NOAA Grid Objective

**Boulder HPCS**

**Boulder Batch**
**Server**

**Princeton**
**HPCS**

**Princeton**
**Batch Server**

**NOAA**
**Firewall**

**Meta**
**Scheduler**

**PMEL Batch Server**

- Glogin
- Gftp

**PMEL Cluster**

**DC Area Batch Server**

**DC Area**
**HPCS**

- Gsubmit URL

**MyProxy CA**
**Server**

- CA Proxy

- A Prototype NOAA Grid

## Obstacles

- Unstable software
- Globus 3.2.1
- Third party packages that rely on it
- Resource allocation policy needed
- Model runs may need to be preemptible
- Security infrastructure
- Heterogeneous
- Adds complexity
- Reduces mutual trust
- Fluid : 2 PMEL nodes became inaccessible just after inclusion onto the grid

- A Prototype NOAA Grid

## NOAA Grid Secure Access Proposal

- A secure perimeter would be constructed around the NOAA HPCS’s and possibly smaller clusters
- Access to any of these nodes only possible with secure card (one-time-password) technology
- For now, **mutually acceptable**, heterogeneous technologies allowed
- Eventually, FIPS 201 token card readers

- A Prototype NOAA Grid

## NOAA Grid Secure Access Proposal

- Grid network traffic only permitted between these systems via a defined set of ports
- Security of prototype NOAA CA would be enhanced
- A permanent CA administrator would be identified
- Grid certificates stored in a MyProxy database on one or more highly secure nodes only accessible from grid nodes
- Certificates would never leave these nodes
- Users could download short-term certificate proxies to enable grid access

- A Prototype NOAA Grid

**Notes:**
CA admin verifies prospective users as is done currently by all HPCS site administrators

## NOAA Grid Secure Access Proposal

- Once on the grid, users could do the following without one-time-passwords
- Remote login to other grid nodes
- Transfer files between grid nodes
- Submit batch jobs to any grid clusters
- Proposal available on the grid project web-site

- A Prototype NOAA Grid

**Notes:**
These capabilities are analogous to what is allowed on the individual machines.  We’re just redefining the machine to be something bigger

## Why secure grid proposal won’t fly currently

- Technical Issues
- GFDL and FSL HPCS’s share filesystems with non-HPCS nodes
- Access to these HPCS’s from lab-internal user workstations does not require a one-time-password
- Remote login idle session timeouts are not implemented at any of these sites

- A Prototype NOAA Grid

## Why secure grid proposal won’t fly currently

- Management Issues
- Each HPCS security administrator would be at the mercy of the other site administrators
- What if someone forgets to apply a patch?
- What if someone forgets to lock down a node?
- What if the security team at one of the sites becomes short-staffed due to funding issues?
- A highly technically competent NOAA HPCS security administrator with ultimate responsibility (and budgeting authority) is needed

- A Prototype NOAA Grid

**Notes:**
Maybe someone from the current staff

## Meta-Scheduling

- Rudimentary scheduler in place
- Leveraged LeSC Globus/SGE job manager
- User requests specific clusters
- Input/Output file staging works
- Jobs specified using batch system independent XML
- No intelligence to choose optimal site automatically
- No ability to query queue status

- A Prototype NOAA Grid

## Meta-Scheduling

- Experiments made with 4 meta-schedulers
- Cluster Resources, Inc. SILVER
- Works with PBS, LoadLeveler, LSF but not SGE
- Successful demonstration on NOAA grid using PBS
- Support for SGE is in the works
- EPCC Job Scheduling Hierarchically (JOSH)
- Only works with SGE
- Only works on some grid nodes
- Freeware

- A Prototype NOAA Grid

## Meta-Scheduling

- Platform, Inc. funded Community Scheduler Framework (CSF)
- Does not work with Globus 3.2.1
- Needs plugins for batch systems other than LSF
- Freeware
- Condor-G
- Does not work with Globus 3.2.1
- Freeware

- A Prototype NOAA Grid

## Coupled Modeling

- Coupled WRF/ROMS between PMEL/FSL previously demonstrated
- Coupled WRF/ROMS between FSL/GFDL failed due to lack of cross-grid connectivity
- Additional firewall holes would be required
- Coupled WRF/ROMS across the TeraGrid nearly working (Chris Moore)

- A Prototype NOAA Grid

## Remaining Work

- Upgrade to Globus 4.X (said to be much more stable)
- Enhance CA security
- Implement true meta-scheduler
- Finish coupling WRF/ROMS across the TeraGrid
- Grid-enable MAPP EarthPort
- A java-based portal for managing testing process for WRF and other models
- Will add
- Single sign-on
- Meta-scheduling
- Grid file transfer GUI
- Upgrade NOAA grid to IPv6

- A Prototype NOAA Grid

## Grid File Transfer GUI

- A Prototype NOAA Grid

## Questions?

**www-ad.fsl.noaa.gov/ac/schaffer/ngi.html**

- A Prototype NOAA Grid