# Presentation: 309612

## Introduction to the T3E

- Mark Durst
- NERSC/USG
- ERSUG Training, Argonne, IL
- 28 April 1999

## Outline

- Hardware and Configuration
- Programming Environment
- Planning Runs
- Monitoring Execution
- Accounting
- Additional Resources
- Elvis Impression

**Notes:**
The boring nuts and bolts one must know to run on the machine.

On supercomputers, things are not always working properly...

## NERSC T3E Configuration

- Commodity DEC Alpha EV-5 superscalar processor
- 450 MHz clock
- 900 Mflops/PE peak (only 5-10% typically achieved)
- Theoretical peak performance: 575 Gflops
- 256 MB memory per PE
- 692 PEs in 3 flavors
- 644 		Application
- 33		Command (ideally)
- 15		OS
- Access via telnet, ssh, FTP
- Connect to NERSC mass storage, AFS

**Notes:**
Peak is what made NERSC #5 six months ago (#12 now)

OS PEs irrelevant to the user
Command PEs what you compile, etc. on; shared
App PEs for parallel execution only; unshared

How some app procs flip back and forth

Mapped-out procs come from command PEs

## Interactive Environment

- UNICOS/mk
- Available shells: sh/ksh, csh, tcsh
- csh: no file completion
- tcsh not Cray-supported
- Home directories
- 2 GB file quota (with possible data migration)
- 3500 inode quota
- /usr/tmp
- Used both for batch and temporary user space
- 75 GByte quota, 6K inode quota
- Fastest transfer rates

**Notes:**
“mk” means that some of the OS (basically anything PE-specific) is resident on each PE, while most global stuff (anything requiring understanding of what’s going on elsewhere) runs on OS PEs

tcsh can crash the batch system; batch jobs are basically like shell scripts run at login; that’s why we don’t allow tcsh 

must point out the distinction between temporary user space (which is purgeable but persistent) and batch space (which is evanescent)

## modules

- modules manages user environment
- Paths
- Environment variables
- Aliases
- Cray’s PrgEnv is modules-driven
- Provided startup files are critical!
- Add to them, don’t clobber them
- Add to paths, don’t set them
- If you mess up, no compilers, etc.
- Largely automatic

**Notes:**
In addition to setting up modules, the startup files provide a per-session temporary directory, set needed environment variables, etc.

## More Fun with modules

-  module list (tells you what’s loaded)
-  module avail (lists them all)
- Other module subcommands
- load
- unload
- switch
- help
- Roll back compilers
- Test new versions
-  http://home.nersc.gov/software/os/modules.html

## Other modules

-  imsl (loads by default)
-  nag (loads by default)
-  scalapack (1.5)
-  GNU (prepends) and GNU.tools (appends)
-  tools (tcsh, bash)
-  netcdf
-  KCC (KAI C++ compiler)
-  USG
-  tedi

**Notes:**
USG as an example of something that adds to the module “space”; you can see more modules after loading it.

Could be used to simplify a group’s work (through the “use.own” module)

## Programming Environment

-  f90
-  cc/CC
-  cam (assembler)
-  cld (loader; usually unneeded)
-  pghpf
-  KCC (“module load KCC”)
-  totalview (debugger)
-  pat, apprentice (performance analysis)

**Notes:**
f90 -o myobj.o with no source will simply load

High Performance Fortran is the set of F90 extensions that provide language-level primitives for data decomposition and distribution of computational work.  Quite a different approach from communications libraries, and in general harder to get working well; but in principle allows the user to do less, the compiler more.

KCC from Kuck & Associates; many prefer their C++ implementation to Cray’s.

## f90

- Conforms to Fortran 90 standard
- Much “standard” f77 wasn’t
- User-defined and abstract types
- Array syntax
- Allocatable objects and pointers
- Additional intrinsics
-  cpp-like preprocessor

**Notes:**
Cray generally responsive to failures to conform

F90 standard includes the F77 standard as a subset.

Additional intrinsics include RNG, type property inquiries, etc.

f90’s preprocessor not to be confused with INCLUDE in F90

Also add’l “modern” programming stuff: (overloaded) modules, arbitrary structures (incl. as f’n returns), control constructs (DO/ENDDO, SELECT), source form...

## Important f90 options

-  -f: source form (fixed or free)
- Defaults: .f fixed, .f90 free
-  -c: Compile only
-  -o *name*: Name executable
- Overrides -c (use -b *name* instead)
-  -g, -G0, -G1: debugging
-  -O[0-3]: general optimization
-  -Ra, -Rb: Argument/Bounds checking
-  -dp: Double precision  64-bit single precision
-  -i 32 / -s default32: 32-bit integers / numbers
-  -ev: Static memory allocation

**Notes:**
Web slides say “F90”; should be “f90” (options to *compiler*, which is only lower case)

Makefiles often say -c -o <name>; regrettably, must replace this awkwardly with either no -o (luckily, most replace myname.f with myname.o, which is the default behavior), or -b <name> instead.

Also exist several specific kinds of optimization (inlining, loop unrolling, etc.), each of which can be set separately.  “Cannot” combine the two methods.

-dp: real DP “deferred”...can also go “down” to 32 bits...(integer only?)

## Executables: Malleable or Fixed

-  -X*npes* (e.g., -X64) creates “fixed” executable
- Always runs on same number of (application) processors
- Type ./a.out to run
-  -Xm or no -X option creates “malleable” executable
-  ./a.out will run on command PE
-   mpprun -n *npes* ./a.out runs on *npes* APP PEs

**Notes:**
Malleable runs on command PE with one proc.; still, you may want to specifically prevent a scalar executable from being “mall”d.

## Execution Model

- In F90, C, C++, all processors execute same program
- Can ask for:
- Process number (from zero up)
-   MY_PE() (F90)
-   _my_pe() (C/C++)
- Total number of PEs
-   NUM_PES()  (F90)
-   _num_pes() (C/C++)
- Above used to establish “master/slave” relationships
- Libraries still needed for communication

**Notes:**
For “embarrassingly parallel” programs, these may be enough.  

For most, no; but worth mentioning for use in quickie test programs

## Libraries

- MPI (Message-Passing Interface)
- PVM (Parallel Virtual Machine)
- SHMEM (SHared MEMory; non-portable)
- BLACS (Basic Linear Algebra Communication Subprograms)
- ScaLAPACK (SCAlable [parts of] LAPACK)
- NetCDF (NETwork Common Data Format)
- HDF (Hierarchical Data Format)
- LIBSCI (including parallel FFTs), NAG, IMSL

**Notes:**
MPI has the more solid implementation, and is less “frozen” in the world in general
PVM better for heterogeneous architectures and for spawning (though this is awkward T3E-only)
SHMEM, as the name implies, allows program to act more as if all memory were shared, with direct writes to remote memory; not only non-portable, but harder.
NETCDF?
Exists SPRNG, though not publicly up yet

## Archival Storage in HPSS

- High-Performance Storage System
- Designed for scalability & hierarchies
- User storage quotas exist
- Access via ftp or new hsi utility
- Two systems:
- hpss.nersc.gov (hsi hpss)
- archive.nersc.gov (hsi, hsi archive) contains old CFS files
- merger planned

**Notes:**
HSI utility looks a lot like old cfs utilities; is only way to do recursive stuff, most management (delete and move within ftp’s purview)

Don’t forget pftp hpss, jimmied to be more efficient (we hope) and to be passwordless (hsi is getting there, too)

## Networking Issues

- AFS
- Accounts must be requested
- Tiny local quotas
- Available on Crays through NFS/AFS gateway
- Non-trivial latencies
- Remote logins
-  .rhosts access not permitted;  no incoming “r- commands”
- ssh available
-  xterm only “backwards”

**Notes:**
30MB quota; explain latencies

Fire xterms back at your wkstn (though we’d prefer you generate an xterm elsewhere)

## Execution modes

- Interactive serial
- < 60 minutes
- on command PEs
- slightly reduced memory
- Interactive parallel
- < 30 minutes
- < 64 processors
- Batch

**Notes:**
Interactive is understood to be for debugging and housekeeping purposes; production computing is to be done in batch.  Violators can be serialized...

## Batch queues on mcurie.nersc.gov

- To see them: qstat -b
-  pe16 through pe512
- 4 hours “on the torus”
- Routine parallel jobs
-  serial_short: 4 hours on a single command PE
-  debug_small: 1⁄2 hour, up to 32 PEs
-  long128, gc128, gc256: 12-hour queues
-  64 PEs
-  gc queues restricted
- Largest queues shuffled in at night
- Other jobs checkpointed out
- Subject to change

## Example daily job mix

**Notes:**
This is a representative (though not exact) drawing of how things get shuffled in and out.

## Batch submission

- Jobs are shell scripts
-  cqsub submits, returns task ID; cqdel deletes
-  cqstatl/qstat gets status (many options)
- NQS parameters determine queue
- #QSUB -l mpp_p=... (number of PEs)
- #QSUB -l mpp_t=... (“parallel” time)
- for serial jobs:
- use #QSUB -q serial
*not* #QSUB -l mpp_p=1

**Notes:**
This is the twenty-second version of the batch talk.

## Pipe Queues

- You submit to pipe queues, not batch queues
- Use only pipe names in directives like:  #QSUB -q serial
- Group batch queues:
-  serial = serial_short
-  debug = debug_small
-  production = pe128 through pe512
-  long = long128, gc128, gc256
- 3 jobs per user in production + long
- 3 in serial, one in debug
- To see them: qstat -p

**Notes:**
If you don’t specify a batch queue, you’ll end up in “production”, which is probably not so bad.

## Scheduling Information

- Lots of NQS-related limits
- Queue run limits
- Queue “complex” run limits
- Global Resource Manager
- Fits jobs into contiguous sets of PEs
- Once started, jobs run to completion (mostly)
- First-fit algorithm lets small jobs trample big ones
-  grmview shows PE status, waiting jobs

**Notes:**
Scheduling is relevant to both interactive and batch computing (though, obviously, NQS limits don’t apply to interactive).

You don’t really care about GRM, but it can sometimes be the only explanation for “why am I not running?”

This, too, is a work in progress.

## Scheduling Information (cont’d)

-  pslist gives summary of GRM data
- No man page; pslist -h instead
- Checkpointing
- For system maintenance
- To run test and “grand challenge” jobs
- Shows “Hop” in qstat/cqstatl (held by operator)
-  mppview more nuts-and-bolts

**Notes:**
tstat is all you’ll ever really want.

Its picture includes the batch region and interactive region explicitly, and allows you to see holes.

## Accounting and allocations

- T3E allocations are in node-minutes
- setcub view repo=*reponame*
- setcub view user=*username*
-   newacct *reponame* switches repos interactively
- One login name per user; multiple repos
-  #QSUB -A *reponame *charges batch jobs
- Charging updated daily; enforcement manual

**Notes:**
ENFORCEMENT MANUAL!! 
ENFORCEMENT MANUAL!!

Even though allocations have come out in small hunks, we’ve still had accounts where users burned up everything in one wild night.

Regrettably, this is the PI’s responsibility.

## On-line Resources

- T3E pages under “Computers” at home.nersc.gov
- Read overview once, check “Changes” monthly
- Docs in Cray on-line system
-  http://www.cray.com/swpubs/
- “Topics” to T3E collection
- Many other docs (e.g., F90, C manual sets)
- Cray Web site, www.cray.com
- Technical documents, additional on-line docs
- NERSC T3E tutorials
- “Training”  “NERSC Tutorials”

## More on-line resources

- Other NERSC tutorials
- Using the Cray f90 compiler at NERSC
- Introduction to make
- NQE: Using the batch system
- Look over NERSC Web generally

## man pages

- cqsub
- cqstatl
- f90
- cc
- CC