# Presentation: 044186

## NCCS User Forum

- 25 September 2008

## Agenda

**Welcome & Introduction**
**Phil Webster, CISTO Chief**
**Scott Wallace, CSC PM**

- Current System Status
- Fred Reitz, Operations Manager

- Compute Capabilities at the NCCS
- Dan Duffy, Lead Architect

- Questions and  Comments
- Phil Webster, CISTO Chief***	***

- User Services Updates
- Bill Ward, User Services Lead

- SMD Allocations
- Sally Stemwedel, HEC Allocation Specialist

- 9/25/08

## NCCS Support Services

**Large scale HEC computing cluster and on-line storage**
**Comprehensive toolsets for job scheduling and monitoring**

**Large capacity storage **
**Tools to manage and protect data**
**Data migration support**

**Help Desk**
**Account/Allocation support**
**Computational science support**
**User teleconferences**
**Training & tutorials**

**Interactive analysis environment**
**Software tools for image display**
**Easy access to data archive**
**Specialized visualization support**

**Capability to share data & results**
**Supports community-based development**
**Facilitates data distribution and publishing**

**Code repository for collaboration**
**Environment for code development and test**
**Code porting and optimization support**
**Web based tools**

**Internal high speed interconnects for HEC components**
**High-bandwidth to NCCS for GSFC users**
**Multi-gigabit network supports on-demand data transfers **

**NCCS Support Services**

**HEC Compute**

**Data Archival and Stewardship**

**Code Development & Collaboration**

**Analysis & Visualization**

**User Services**

**Data Sharing**

**High Speed Networks**

**DATA**

**Global file system enables data access for full range of modeling and analysis activities**

- 9/25/08

## Resource Growth at the NCCS 

- 9/25/08

## NCCS Staff Transitions

- New Govt. Lead Architect: Dan Duffy
- 	(301) 286-8830, _[Daniel.Q.Duffy@nasa.gov](mailto:Daniel.Q.Duffy@nasa.gov)_
- New CSC Lead Architect: Jim McElvaney
- 	(301) 286-2485, _[James.D.McElvaney@nasa.gov](mailto:James.D.McElvaney@nasa.gov)_
- New User Services Lead: Bill Ward
- 	(301) 286-2954, _[William.A.Ward@nasa.gov](mailto:William.A.Ward@nasa.gov)_
- New HPC System Administrator: Bill Woodford
- 	(301) 286-5852, _[William.E.Woodford@nasa.gov](mailto:William.E.Woodford@nasa.gov)_

- 9/25/08

## Key Accomplishments

- SLES10 upgrade
- GPFS 3.2 upgrade
- Integrated SCU3
- Data Portal storage migration
- Transition from Explore to Discover

- 9/25/08

## Integration of Discover SCU4

- SCU4 to be connected to Discover 10/1 (date firm)
- Staff testing and scalability 10/1-10/5 (dates approximate)
- Opportunity for interested users to run large jobs 10/6-10/13                    (dates approximate)

- 9/25/08

## Agenda

- Welcome & Introduction
- Phil Webster, CISTO Chief
- Scott Wallace, CSC PM

**Current System Status**
**Fred Reitz, Operations Manager**

- Compute Capabilities at the NCCS
- Dan Duffy, Lead Architect

- Questions and  Comments
- Phil Webster, CISTO Chief***	***

- User Services Updates
- Bill Ward, User Services Lead

- SMD Allocations
- Sally Stemwedel, HEC Allocation Specialist

- 9/25/08

## Explore Utilization Past Year

**67.0%**

**69.0%**

**85.6%**

**73.0%**

**613,975**
**CPU hours**

- 9/25/08

## Explore Availability

**8/13 – Hardware failure**

**8/13 – Facilities maintenance**

**5/14 – Hardware maintenance**

**May through August availability**
**13 outages**
**8 unscheduled**
**7 hardware failures**
**1 human error**
**5 scheduled**
**65 hours total downtime**
**32.8 unscheduled**
**32.2 scheduled**

**Longest outages**
**8/13 – Hardware failure, 19.9 hrs**
**E2 hardware issues after scheduled power down for facilities maintenance**
**System left down until normal business hours for vendor repair**
**Replaced/unseated PCI bus, IB connection**
**8/13 – Hardware maintenance, 11.25 hrs**
**Scheduled outage**
**Electrical work – facility upgrade**

**Explore Availability**

- 9/25/08

## Explore Queue Expansion Factor

**Run Time**

**Weighted over all queues for all jobs**
**(Background and Test queues excluded)**

**Explore Queue Expansion Factor**

- 9/25/08

## Discover Utilization Past Year

- 9/25/08

## Discover Utilization

- 9/25/08

## Discover Availability

**Discover Availability**

**May through August availability**
**11 outages**
**6 unscheduled**
**2 hardware failures**
**2 software failures**
**3 extended maintenance windows**
**5 scheduled**
**89.9 hours total downtime**
**35.1 unscheduled**
**54.8 scheduled**

**Longest outages**
**7/10 – SLES 10 upgrade, 36 hrs**
**15 hours planned**
**21 hour extension**
**8/20 – Connect SCU3, 15 hrs**
**Scheduled outage**
**8/13 – Facilities maintenance, 10.6 hrs**
**Electrical work – facility upgrade**

**7/10 – Extended SLES10 upgrade window**

**7/10 – SLES10 upgrade**

**8/20 – Connect SCU3 to cluster**

**8/13 – Facilities electrical upgrade**

- 9/25/08

## Discover Queue Expansion Factor

**Queue Wait Time + Run Time**
**Run Time**

**Weighted over all queues for all jobs**
**(Background and Test queues excluded)**

- 9/25/08

## Current Issues on Discover:Infiniband Subnet Manager

**Symptom:** Working nodes erroneously removed from GPFS following Infiniband Subnet problems with other nodes.
**Outcome:** Job failures due to node removal
**Status:** Modified several subnet manager configuration parameters on 9/17 based on IBM recommendations.  Problem has not recurred; admins monitoring.

- 9/25/08

## Current Issues on Discover:PBS Hangs

**Symptom:** PBS server experiencing 3-minute hangs several times per day
**Outcome:** PBS-related commands (qsub, qstat, etc.) hang
**Status: **Identified periodic use of two communication ports also used for hardware management functions.  Implemented work-around on 9/17 to prevent conflicting use of these ports.  No further occurrences.

- 9/25/08

## Current Issues on Discover:Problems with PBS –V Option

**Symptom:** Jobs with large environments not starting
**Outcome: **Jobs placed on hold by PBS
**Status: **Investigating with Altair (vendor). In the interim, requested users not pass full environment via –V, instead use –v or define necessary variables within job scripts.

- 9/25/08

## Current Issues on Discover:Problem with PBS and LDAP

**Symptom:** Intermittent PBS failures while communicating with LDAP server.
**Outcome: **Jobs rejected with bad UID error due to failed lookup
**Status:** LDAP configuration changes to improve information caching and reduce queries to LDAP server; significantly reduced problem frequency.  Still investigating with Altair.

- 9/25/08

## Future Enhancements

- Discover Cluster
- Hardware platform – SCU4 10/1/2008
- Additional storage
- Discover PBS Select Changes
- Syntax changes to streamline job resource requests
- Data Portal
- Hardware platform
- DMF
- Hardware platform

- 9/25/08

## Agenda

- Welcome & Introduction
- Phil Webster, CISTO Chief
- Scott Wallace, CSC PM

- Current System Status
- Fred Reitz, Operations Manager

**Compute Capabilities at the NCCS**
**Dan Duffy, Lead Architect**

- Questions and  Comments
- Phil Webster, CISTO Chief***	***

- User Services Updates
- Bill Ward, User Services Lead

- SMD Allocations
- Sally Stemwedel, HEC Allocation Specialist

- 9/25/08

## Overall Acquisition Planning Schedule2008 – 2009

**Jan**

**Feb**

**Mar**

**Apr**

**May**

**Jun**

**Jul**

**Aug**

**Sep**

**Oct**

**Nov**

**Dec**

**2009**

**Stage 1: Power & Cooling Upgrade**

**Stage 2: Cooling Upgrade**

**Storage Upgrade**

**Compute Upgrade**

**Facilities: E100**

**Write RFP**

**Issue RFP, Evaluate Responses, Purchase**

**Delivery & Integration**

**Write RFP**

**Issue RFP, Evaluate Responses, Purchase**

**Delivery**

**Explore Decommissioned**

**Stage 1: Integration & Acceptance**

**Stage 2: Integration & Acceptance**

**We are here!**

- 9/25/08

## What does this schedule mean to you?Expect some outages – Please be patient

**Jan**

**Feb**

**Mar**

**Apr**

**May**

**Jun**

**Jul**

**Aug**

**Sep**

**Oct**

**Nov**

**Dec**

**2009**

**Storage Upgrade**

**Compute Upgrade**

**Additional Storage On-line**

**Stage 1 Compute Capability**
**Available for Users**

**Discover Mods**

**GPFS 3.2 Upgrade (not RDMA)**

**SLES 10 Software Stack Upgrade**

**Stage 2 Compute Capability**
**Available for Users**

**Decommission Explore**

**DONE**

**DONE**

**DONE**

**Delayed**

- 9/25/08

## Cubed Sphere Finite VolumeDynamic Core Benchmark

**Non-hydrostatic, 10 KM resolution**
**Most computationally intensive benchmark**
**Discover Reference Timings**
**216 cores (6x6) – 6,466s**
**288 cores (6x8) – 4,879s**
**384 cores (8x8) – 3,200s**
**All runs made using ALL cores on a node.**

- 9/25/08

## Near Future

- Additional storage to be added to the cluster
- 240 TB RAW
- By the end of the calendar year
- RDMA pushed into next year
- Potentially one additional scalable unit
- Same as the new IBM units
- By the end of the calendar year
- Small IBM Cell application development testing environment
- 2 to 3 months

- 9/25/08

## Agenda

- Welcome & Introduction
- Phil Webster, CISTO Chief
- Scott Wallace, CSC PM

- Current System Status
- Fred Reitz, Operations Manager

- Compute Capabilities at the NCCS
- Dan Duffy, Lead Architect

- Questions and  Comments
- Phil Webster, CISTO Chief***	***

- User Services Updates
- Bill Ward, User Services Lead

**SMD Allocations**
**Sally Stemwedel, HEC Allocation Specialist**

- 9/25/08

## SMD Allocation Policy Revisions 8/1/08

- 1-year allocations only during regular spring and fall cycles
- Fall e-Books deadline 9/20 for November 1 awards
- Spring e-Books deadline 3/20 for May 1 awards
- Projects started off-cycle
- Must have support of HQ Program Manager to start off-cycle
- Will get limited allocation expiring at next regular cycle award date
- Increases over 10% of award or 100K processor-hours during award period need support of funding manager; email _[support@HEC.nasa.gov](mailto:support@HEC.nasa.gov)__ to request increase._
- Projects using up allocation faster than anticipated are encouraged to submit for next regular cycle.

- 9/25/08

## Questions about Allocations?

- Allocation POC
- 	Sally Stemwedel
- 	HEC Allocation Specialist    _[sally.stemwedel@nasa.gov](mailto:sally.stemwedel@nasa.gov)_
- 	(301) 286-5049
- SMD allocation procedure and
- 	e-Books submission link
- 	_[www.HEC.nasa.gov](http://www.hec.nasa.gov/)_

- 9/25/08

## Agenda

- Welcome & Introduction
- Phil Webster, CISTO Chief
- Scott Wallace, CSC PM

- Current System Status
- Fred Reitz, Operations Manager

- Compute Capabilities at the NCCS
- Dan Duffy, Lead Architect

- Questions and  Comments
- Phil Webster, CISTO Chief***	***

**User Services Updates**
**Bill Ward, User Services Lead**

- SMD Allocations
- Sally Stemwedel, HEC Allocation Specialist

- 9/25/08

## Explore Will Be Decommissioned

- It is a leased system
- e1, e2, e3 must be returned to vendor
- Palm will remain
- Palm will be repurposed
- Users must move to Discover

- 9/25/08

## Transition to Discover - Phases

- PI notified
- Users on Discover
- Code migrated
- Data accessible
- Code acceptably tuned
- Performing production runs

- 9/25/08

## Transition to Discover - Status

- 9/25/08

## Transition to Discover - Status

- 9/25/08

## Transition to Discover - Status

- 9/25/08

## Accessing Discover Nodes

- We are in the process of making the PBS select statement more simple and streamlined
- Keep doing what you are doing until we publish something better
- For most folks, changes will not break what you are using now

- 9/25/08

## Discover Compilers

**comp/intel-10.1.017**
- 	preferred
**comp/intel-9.1.052**
- 	if intel-10 doesn’t work
**comp/intel-9.1.042**
**	**only if absolutely necessary

- 9/25/08

## MPI on Discover

**mpi/scali-5**
- Not supported on new nodes
**-l select=<n>:scali=true **to get it
**mpi/impi-3.1.038**
- Slower startup than OpenMPI
- Catches up later (anecdotal)
- Self-tuning feature still under evaluation
**mpi/openmpi-1.2.5/intel-10**
- Does not pass user environment
- Faster startup due to built-in PBS support

- 9/25/08

## Data Migration Facility Transition

- DMF hosts Dirac/Newmintz (SGI Origin 3800s) to be replaced by parts of Palm (SGI Altix)
- Actual cutover Q1 CY09
- Impacts to Dirac users:
- Source code must be recompiled
- Some COTS must be relicensed
- Other COTS must be rehosted

- 9/25/08

## Accounts for Foreign Nationals

- Codes 240, 600, and 700 have established a well-defined process for creating NCCS accounts for foreign nationals
- Several candidate users have been navigated through the process
- Prospective users from designated countries must go to NASA HQ
- Process will be posted on the web very soon
- 	_[http://](http://securitydivision.gsfc.nasa.gov/index.cfm?topic=visitors.foreign)__[securitydivision.gsfc.nasa.gov/index.cfm?topic](http://securitydivision.gsfc.nasa.gov/index.cfm?topic=visitors.foreign)__[=](http://securitydivision.gsfc.nasa.gov/index.cfm?topic=visitors.foreign)__[visitors.foreign](http://securitydivision.gsfc.nasa.gov/index.cfm?topic=visitors.foreign)_

- 9/25/08

## Feedback

- Now – Voice your ...
- Praises?
- Complaints?
- Later – NCCS Support
- _[support@nccs.nasa.gov](mailto:support@nccs.nasa.gov)_
- (301) 286-9120
- Later – USG Lead (me!)
- _[William.A.Ward@nasa.gov](mailto:William.A.Ward@nasa.gov)_
- (301) 286-2954

- 9/25/08

## Agenda

- Welcome & Introduction
- Phil Webster, CISTO Chief
- Scott Wallace, CSC PM

- Current System Status
- Fred Reitz, Operations Manager

- Compute Capabilities at the NCCS
- Dan Duffy, Lead Architect

**Questions and  Comments****	**
**Phil Webster, CISTO Chief*****	***

- User Services Updates
- Bill Ward, User Services Lead

- SMD Allocations
- Sally Stemwedel, HEC Allocation Specialist

- 9/25/08

## Open DiscussionQuestionsComments