# Presentation: 658187

## NCCS User Forum

**11 December 2008**

## Agenda

**Welcome & Introduction**
**Phil Webster, CISTO Chief**

- Current System Status
- Fred Reitz, Operations Manager

- NCCS Compute Capabilities
- Dan Duffy, Lead Architect

- Questions and  Comments
- Phil Webster, CISTO Chief***	***

- User Services Updates
- Bill Ward, User Services Lead

- 12/11/08

## Agenda

- Welcome & Introduction
- Phil Webster, CISTO Chief

**Current System Status**
**Fred Reitz, Operations Manager**

- NCCS Compute Capabilities
- Dan Duffy, Lead Architect

- Questions and  Comments
- Phil Webster, CISTO Chief***	***

- User Services Updates
- Bill Ward, User Services Lead

- 12/11/08

## Key Accomplishments

- SCU4 added to Discover and currently running in “pioneer” mode
- Explore decommissioned and removed
- Discover filesystems converted to GPFS 3.2 native mode

- 12/11/08

## Discover Utilization Past Year

**Discover Utilization Past Year**

**67.1%**

**64.4%**

**73.3%**

**2,446,365**
**CPU hours**

**1,320,683**
**CPU hours**

**SCU3 cores added**

- 12/11/08

## Discover Utilization

- 12/11/08

## Discover Queue Expansion Factor

**Eligible Time + Run Time**
**Run Time**

**Weighted over all queues for all jobs**
**(Background and Test queues excluded)**

- 12/11/08

## Discover Availability

**September through November availability**
**13 outages**
**9 unscheduled**
**0 hardware failures**
**7 software failures**
**2 extended maintenance windows**
**4 scheduled**
**104.3 hours total downtime**
**68.3 unscheduled**
**36.0 scheduled**

**Longest outages**
**11/28-29 – GPFS hang, 21 hrs**
**11/12 – Electrical maintenance, Discover reprovisioning,18 hrs**
**Scheduled outage**
**10/1 – SCU4 integration, 11.5 hrs**
**Scheduled outage plus extension**
**9/2-3 – Subnet Manager hang, 11.3 hrs**
**11/6 – GPFS hang, 10.9 hrs**

**GPFS hang**

**Electrical maintenance, Discover reprovisioning**

**SCU4 integration**

**Subnet Manager hang**

**GPFS hang**

**GPFS hang**

**GPFS hang**

**SCU4 integration, Switch reconfiguration**

**Subnet Manager hang**

**Subnet Manager maint.**

**GPFS hang**

- 12/11/08

## Current Issues on Discover:GPFS Hangs

**Symptom:** GPFS hangs resulting from users running nodes out of memory.
**Impact: **Users cannot login or use filesystem.  System Admins reboot affected nodes.
**Status:** Implemented additional monitoring and reporting tools.

- 12/11/08

## Current Issues on Discover:Problems with PBS –V Option

**Symptom:** Jobs with large environments not starting
**Impact: **Jobs placed on hold by PBS
**Status: **Consulting with Altair. In the interim, don’t use –V to pass full environment, instead use –v or define necessary variables within job scripts.

- 12/11/08

## Resolved Issues on Discover:Infiniband Subnet Manager

**Symptom:** Working nodes erroneously removed from GPFS following Infiniband Subnet problems with other nodes.
**Impact:** Job failures due to node removal
**Status:** Modified several subnet manager configuration parameters on 9/17 based on IBM recommendations.  Problem has not recurred.

- 12/11/08

## Resolved Issues on Discover:PBS Hangs

**Symptom:** PBS server experiencing 3-minute hangs several times per day
**Impact:** PBS-related commands (qsub, qstat, etc.) hang
**Status: **Identified periodic use of two communication ports also used for hardware management functions.  Implemented work-around on 9/17 to prevent conflicting use of these ports.  No further occurrences.

- 12/11/08

## Resolved Issues on Discover:Intermittent NFS Problems

**Symptom:** Inability to access archive filesystems
**Impact:** hung commands and sessions when attempting to access $ARCHIVE
**Status: **Identified hardware problem with Force10 E600 network switch.  Implemented workaround and replaced line card. No further occurrences.

- 12/11/08

## Future Enhancements

- Discover Cluster
- Hardware platform
- Additional storage
- Data Portal
- Hardware platform
- Analysis environment
- Hardware platform
- DMF
- Hardware platform

- 12/11/08

## Agenda

- Welcome & Introduction
- Phil Webster, CISTO Chief

- Current System Status
- Fred Reitz, Operations Manager

**NCCS Compute Capabilities**
**Dan Duffy, Lead Architect**

- Questions and  Comments
- Phil Webster, CISTO Chief***	***

- User Services Updates
- Bill Ward, User Services Lead

- 12/11/08

## Very High Level of What to Expect in FY09

**Oct**

**Nov**

**Dec**

**Jan**

**Feb**

**Mar**

**Apr**

**May**

**Jun**

**Jul**

**Aug**

**Sep**

**Discover SW Stack Upgrade**

**Cluster Upgrade (Nehalem)**

**Analysis System**

**DMF from IRIX to Linux**

**Data Management Initiative**

**New Tape Drives**

**Major Initiatives**

**Other Activities**

**Discover FC and Disk Addition**

**Additional Discover Disk**

**Continued Scalability Testing**

**Delivery of IBM Cell**

- 12/11/08

## Adapting the Overall Architecture

- Services will have
- More independent SW stacks
- Consistent user environment
- Fast access to the GPFS file systems
- Large additional disk capacity for longer storage of files within GPFS
- This will result in
- Fewer downtimes
- Rolling outages (not everything at once)

- 12/11/08

## Conceptual Architecture Diagram

**Conceptual Architecture Diagram**

**GPFS I/O Servers**

**IB**

**Discover (batch)**
**Base**
**SCU1**
**SCU2**
**SCU3**
**SCU4**
**Viz**

**GPFS I/O Servers**

**IB**

**Analysis Nodes**
**(interactive)**

**SAN**

**GPFS I/O Servers**

**IB**

**FY09 Compute**
**Upgrade**
**(Nehalem)**

**Data Portal**

**GPFS I/O Servers**

**IB**

**SAN**

**Archive**
**DMF**

**SAN**

**10 GbE**
**LAN**

- 12/11/08

## What is the Analysis Environment?

- Initial technical implementation plan
- Large shared memory (256 GB at least) nodes
- 16 core nodes with 16 GB/core
- Interactive (not batch); direct logins
- Fast access to GPFS
- 10 GbE network connectivity
- Consistent software stack to Discover
- Independent of compute stack (coupled only by GPFS)
- Additional storage for staging from the archive specific for analysis
- Visibility and easy access to the archive and data portal (NFS)

- 12/11/08

## Excited about Intel Nehalem

- Quick Specs
- Core 7i – 45 nm
- 731 million transistors per quad-core
- 2.66 GHz to 2.93 GHz
- Private L1 cache (32 KB) and L2 (256 KB) per core
- Shared L3 cache (up to 8 MB) across all the cores
- 1,066 MHz DDR3 Memory (3 channels per core)
- Important Features
- Intel QuickPath Interconnect
- Turbo Boost
- Hyper-Threading
- Learn more at:
- _[http://www.intel.com/technology/architecture-silicon/next-gen/index.htm](http://www.intel.com/technology/architecture-silicon/next-gen/index.htm)_
- _[http://en.wikipedia.org/wiki/Nehalem_(microarchitecture)](http://en.wikipedia.org/wiki/Nehalem_(microarchitecture))_

- 12/11/08

## Nehalem versus Harpertown

- Single thread improvement (will vary based on application)
- Larger cache with the 8 MB shared cache across all processors
- Memory to processor bandwidth dramatically increased over the Harpertown
- Initial measurements have shown 3 to 4x memory to processor bandwidth increase

- 12/11/08

## Agenda

- Welcome & Introduction
- Phil Webster, CISTO Chief

- Current System Status
- Fred Reitz, Operations Manager

- NCCS Compute Capabilities
- Dan Duffy, Lead Architect

- Questions and  Comments
- Phil Webster, CISTO Chief***	***

**User Services Updates**
**Bill Ward, User Services Lead**

- 12/11/08

## Issues from Last User Forum:Shared Project Space

- Implementation of shared project space on Discover
- Status: resolved
- Available for projects by request
- Accessible via /share; usage deprecated
- Accessible via $SHARE; correct usage

- 12/11/08

## Issues from Last User Forum:Increase Queue Limits

- Increase CPU & time limits in queues
- Status: resolved

| Queue | Priority | Max CPUs | Max Hours |
| --- | --- | --- | --- |
| test | 101 | 2064 | 12 |
| general_hi | 80 | 512 | 24 |
| debug | 70 | 32 | 1 |
| general_long | 55 | 256 | 24 |
| general | 50 | 256 | 12 |
| general_small | 50 | 16 | 12 |
| background | 1 | 256 | 4 |

- 12/11/08

## Issues from Last User Forum:Commands to Access DMF

- Implementation **dmget** and **dmput**
- Status: test version ready to be enabled on Discover login nodes
- Reason for delay was that dmgets on non-dm files would hang
- There may still be stability issues
- E-mail will be sent soon notifying users of availability

- 12/11/08

## Issues from Last User Forum: Enabling Sentinel Jobs

- Running a “sentinel” subjob to watch a main parallel compute “subjob” in a single PBS job
- Status: under investigation
- Requires an NFS mount of data portal file system on Discover gateway nodes
- Requires some special PBS usage to specify how subjobs will land on nodes

- 12/11/08

## Other Issues:Poor Interactive Response

- Slow interactive response on Discover
- Status: resolved
- Router line card replaced
- Automatic monitoring instituted to promptly detect future problems

- 12/11/08

## Other Issues:Parallel Jobs > ~300-400 CPUs

- Some users experiencing problems running > ~300-400 CPUs on Discover
- Status: resolved
- “stacksize unlimited” in .cshrc file needed
- Intel mpi passes environment, including settings in startup files

- 12/11/08

## Other Issues:Parallel Jobs > 1500 CPUs

- Many jobs won’t run at > 1500 CPUs
- Status: under investigation
- Some simple jobs will run
- NCCS consulting with IBM and Intel to resolve the issue
- Software upgrades probably required
- Solution may fix slow Intel MPI startup

- 12/11/08

## Other Issues:Visibility of the Archive

- Visibility of the archive from discover
- Current Status
- Compute/viz nodes don’t have external network connections
- “Hard” NFS mounts guarantee data integrity, but if there is an NFS hang, the node hangs
- Login/gateway nodes may use a “soft” NFS mount, but risk of data corruption
- bbftp or scp (to Dirac) preferred over cp when copying data

- 12/11/08

## DMF Transition

- Dirac due to be replaced in Q2 CY09
- Interactive host for Grads, IDL, Matlab, etc.
- Much larger memory
- GPFS shared with Discover
- Significant increase in GPFS storage
- Impacts to Dirac users:
- Source code must be recompiled
- COTS must be relicensed/rehosted
- Old Dirac up until migration complete

- 12/11/08

## Help Us Help You

- Don’t use “PBS –V” (job hangs with error “too many failed attempts to start”)
- Direct stdout, stderr to specific files, or you will fill up the PBS spool directory
- Use an interactive batch session instead of an interactive session on a login node
- If you suspect your job is crashing nodes, call us before running again

- 12/11/08

## Help Us Help You (continued)

- Try to be specific when reporting problems, for example:
- If the archive is broken, specify symptoms
- If files are inaccessible or can’t be recalled, please send us the file names

- 12/11/08

## Plans

- Implement a better scheduling policy
- Implement integrated job performance monitoring
- Implement better job metrics reporting
- Or...

- 12/11/08

## Feedback

- Now – Voice your ...
- Praises?
- Complaints?
- Suggestions?
- Later – NCCS Support
- _[support@nccs.nasa.gov](mailto:support@nccs.nasa.gov)_
- (301) 286-9120
- Later – USG Lead (me!)
- _[William.A.Ward@nasa.gov](mailto:William.A.Ward@nasa.gov)_
- (301) 286-2954

- 12/11/08

## Agenda

- Welcome & Introduction
- Phil Webster, CISTO Chief

- Current System Status
- Fred Reitz, Operations Manager

- NCCS Compute Capabilities
- Dan Duffy, Lead Architect

**Questions and  Comments****	**
**Phil Webster, CISTO Chief*****	***

- User Services Updates
- Bill Ward, User Services Lead

- 12/11/08

## Open DiscussionQuestionsComments