# Presentation: 496836

**Getting Started at NERSC**

*Richard Gerber*
- NERSC User Services
-  ragerber@nersc.gov
- 510-486-6820

## Outline

- What is NERSC?
- Computing Facilities
- Connecting to NERSC
- Software Environment
- Compilers & Compiling
- Running Jobs
- Debugging, Profiling, & Optimization
- Accounts & Accouting
- Documentation & Getting Help

## What is NERSC?

**N**ational **E**nergy **R**esearch **S**cientific **C**omputing Center
- National center for DOE Office of Science scientific computing
- Located at Berkeley Lab & Oakland Scientific Facility
- Hardware, User Support, Research
- Serves 2,000 users at national laboratories, universities, and industries

## Computing Facilities

- IBM SP
- No. 5 most powerful supercomputer in world
- 188 POWER 3 16-way SMP nodes; 3,008 processors
- 4.5 Tflops peak
- 5 Tbytes memory (1.7 Gbytes/processor)
- 20 Tbytes disk (6.65 Gbytes/processor)
- IBM “Colony” switch for internode communication
- _[http://hpcf.nersc.gov/computers/SP/](http://hpcf.nersc.gov/computers/SP/)_

## Computing Facilities

- HPSS mass file storage
- All NERSC users have HPSS account
- High-speed access to/from Seaborg
- Currently 450 Tbytes in HPSS
- _[http://](http://hpcf.nersc.gov/storage/hpss/)__[hpcf](http://hpcf.nersc.gov/storage/hpss/)__[.](http://hpcf.nersc.gov/storage/hpss/)__[nersc](http://hpcf.nersc.gov/storage/hpss/)__[.](http://hpcf.nersc.gov/storage/hpss/)__[gov](http://hpcf.nersc.gov/storage/hpss/)__[/storage/](http://hpcf.nersc.gov/storage/hpss/)__[hpss](http://hpcf.nersc.gov/storage/hpss/)__[/](http://hpcf.nersc.gov/storage/hpss/)_
- Linux math & SGI visualization servers
- Mathematica, MATLAB, Maple, ...
- AVS, NCAR, IDL, ...
- http://hpcf.nersc.gov/computers/servers/
- NERSC users can request accounts at http://hpcf.nersc.gov/forms/server-request.html

## Connecting to NERSC 

- You must have a NERSC account/password
- Passwords - NERSC Account Support: 486-8612
- Log in using SSH-enabled software
- First-time login on Seaborg:
**UNIX:**** ssh –l nersc_username sadmin.nersc.gov**
- Change password (**passwd** command)
- 1 hour later (max) connect to **seaborg.nersc.gov**
- Reference
- Seaborg: _[http://hpcf.nersc.gov/help/access](http://hpcf.nersc.gov/help/access)_
- HPSS: _[http://hpcf.nersc.gov/storage/hpss](http://hpcf.nersc.gov/storage/hpss)_

## SP Software Environment

- UNIX (IBM AIX)
- Default shell – **csh**
- See: _[http://hpcf.nersc.gov/software/os/shells/](http://hpcf.nersc.gov/software/os/shells/)_
- Parallel Operating Environment (POE)
- IBM software to manage parallel system
- Non-IBM tools & utilities in “modules”
- See: _[http://hpcf.nersc.gov/software/modules.html](http://hpcf.nersc.gov/software/modules.html)_
- List of all Seaborg software
- _[http://hpcf.nersc.gov/software/ibm/](http://hpcf.nersc.gov/software/ibm/)_

## Compilers

- Fortran
- Series of “compilers”: **xlf, xlf90, mpxlf** ...
- See: _[http://hpcf.nersc.gov/software/ibm/xlf.html](http://hpcf.nersc.gov/software/ibm/xlf.html)_
- HPF : _[http://hpcf.nersc.gov/software/pg/pghpf/](http://hpcf.nersc.gov/software/pg/pghpf/)_
- C/C++
- Series of “compilers”: **xlc, xlC, mpcc**, ...
- See: _[http://hpcf.nersc.gov/software/ibm/xlc.html](http://hpcf.nersc.gov/software/ibm/xlc.html)_
- KCC, but Intel has dropped support & development
- GNU: _[http://hpcf.nersc.gov/software/tools/GNU.html#gcc](http://hpcf.nersc.gov/software/tools/GNU.html#gcc)_
- Make sure you can access enough memory
- _[http://hpcf.nersc.gov/software/ibm/sp_memory.html](http://hpcf.nersc.gov/software/ibm/sp_memory.html)_

## Compiling Parallel Programs

- Most codes use MPI for communication
- Must use IBM “parallel” compilers
**mp****xlf, ****mp****xlC, ****mp****xlf90**, ...
**E.g.:**** ****mpxlf90 ... my_first_mpi.f**
- See: _[http://hpcf.nersc.gov/software/libs/mpi/](http://hpcf.nersc.gov/software/libs/mpi/)_
- OpenMP
- Creates threads on a single node
- Use “thread-safe” compilers
**xlf****_r****, mpxlf90****_r**, ... and **–qsmp=omp**
- IBM C++ OpenMP support coming soon...
- See: _[http://hpcf.nersc.gov/computers/SP/openmp.html](http://hpcf.nersc.gov/computers/SP/openmp.html)_
- HPF (see previous reference)

## Running Jobs

- Consider Seaborg configuration
- 16 processors/node = 16 MPI tasks/node maximum
- You have exclusive access to a node
- 16-64 Gbytes memory/node
- 2 network interface cards/node
- 2 communications libraries
- IP – standard Internet Protocol commonly used by ethernet
- US – “User Space” takes advantage of high-speed switch
- File Systems: **$HOME** and **$SCRATCH**
- Quotas: **$HOME**: 10 GB, **$SCRATCH**: 122 GB
- Check disk usage with **myquota** command
- See: _[http://hpcf.nersc.gov/running_jobs/ibm/](http://hpcf.nersc.gov/running_jobs/ibm/)_

## Interactive Jobs

- Interactive Parallel Jobs
- Limited to  30 minutes wallclock time, 16 nodes
**%**** <poe> ./a.out -nodes *****N***** –procs *****NP***
**poe** command is optional, a.out is name of your executable
- NP must not be greater than N*16 or job will not run
- Can also specify **–****tasks_per_node** and one of    **–****nodes** or –**procs**
- Options unknown to **poe** are passed (silently) to your program
- Use **–retry SECS –retrycount N **if machine is busy
- _[http://hpcf.nersc.gov/running_jobs/ibm/#interactive](http://hpcf.nersc.gov/running_jobs/ibm/#interactive)_

## Batch Jobs

- Production runs should be done in batch
- IBM’s batch system software is called LoadLeveler
- You first create a batch script that contains AIX commands and LoadLeveler directives.
- Lines that contain LoadLeveler directives begin with **#@**
- Use LoadLeveler directives to specify run time, number of tasks and nodes, network adapter usage, and run “class”

## LoadLeveler Directives I

- Always include these directives
**#@class = class_name**
- where class_name is one of** regular, debug, premium, low, regular_long**
- Normally use regular, except debug for debugging
**#@node = *****num_nodes***
**#@tasks_per_node = *****16***
- This can be less than 16, but you are always charged for 16
**#@network.MPI = csss,not_shared,us**
**#@wall_clock_limit = *****HH:MM:SS***
**Maximums: 8:00:00, except debug: 00:30:00, regular_long: 24:00:00, **take advantage of backfill
**#@queue**
- Always the last directive

## LoadLeveler Directives II

- Additional suggested directives
**#@shell = /usr/bin/csh**
- Or your favorite shell
**#@job_type = parallel**
**#@environment = COPY_ALL**
- Imports shell environment variables into batch environment
**#@notification = always**
- Sends email to you when job starts and finishes
**#@account_no = mp309**
**#@output = $(host).$(jobid).$(stepid).out**
**#@error = $(host).$(jobid).$(stepid).err**
- Gives output and error files unique names

## Example Batch Script

**#@ class = regular     **
**#@ shell = /usr/bin/csh**
**#@ node = 8**
**#@ tasks_per_node =  16**
**#@ network.MPI = csss,not_shared,us **
**#@ wall_clock_limit = 2:00:00**
**#@ notification = always**
**#@ job_type = parallel**
**#@ output = $(host).$(jobid).$(stepid).out**
**#@ error = $(host).$(jobid).$(stepid).err**
**#@ account_no = mp309**
**#@ environment = COPY_ALL**
**#@ queue**
**./a.out**
**exit**

## Batch Script Notes

- Do not explicitly use **poe** or **poe** options on command line
- Submit script with **llsubmit** command
**% ****llsubmit *****script_name***
- Monitor with** ****llqs**** **command or at _[http://hpcf.nersc.gov/cgi-bin/qstat/llq_seaborg](http://hpcf.nersc.gov/cgi-bin/qstat/llq_seaborg)_  (web login: nuser, password: oer_user)
- Reference & batch script generator:
- _[http://hpcf.nersc.gov/running_jobs/ibm/batch.html](http://hpcf.nersc.gov/running_jobs/ibm/batch.html)_
- http://hpcf.nersc.gov/forms/ll_script.html

## Debugging & Profiling

- Debugger – TotalView
- Compile your code with –g compiler option
**%****module load totalview**
**%****totalview poe –a ./a.out –nodes 1 –procs 16**
- See: http://hpcf.nersc.gov/software/ibm/totalview/
- Profiling – hpmcount, hpmlib & poe+
- POWER3 has hardware counter; can get Mflops, etc.
- Runtime utilities: hpmcount, poe+
**% ****poe+ ./a.out**
- Performance library API: hpmlib
- See: _[http://hpcf.nersc.gov/software/ibm/hpmcount/](http://hpcf.nersc.gov/software/ibm/hpmcount/)_

## Optimization

- Compiler options can provide a great deal of code optimization
- For production codes, we recommend
**-O3 –qtune=pwr3 –qarch=pwr3 –qstrict**
- See _[http://hpcf.nersc.gov/computers/SP/options.html](http://hpcf.nersc.gov/computers/SP/options.html)_
- Detailed information can be found in “Power 3 Introduction and Tuning Guide” at _[http://hpcf.nersc.gov/computers/SP/sg245155.pdf](http://hpcf.nersc.gov/computers/SP/sg245155.pdf)_

## Programming Libraries I

- ESSL
- Highly tuned serial scientific library routines
**% ****xlf90 ... file.f –lessl**
- _[http://hpcf.nersc.gov/software/ibm/#essl](http://hpcf.nersc.gov/software/ibm/#essl)_
- PESSL
- Tuned parallel scientific library routines
**% ****mpxlf90 ... file.f –lpessl**
- _[http://hpcf.nersc.gov/software/ibm/#pessl](http://hpcf.nersc.gov/software/ibm/#pessl)_
- MASS
- Optimized replacements for simple math functions (e.g. sine)
**% ****module load mass**
**%** **xlf ... file.f $MASS**
**%** **xlc ... file.c $MASS -lm**
- http://hpcf.nersc.gov/software/libs/math/MASS/

## Programming Libraries II

- Math
- NAG, IMSL, FFTs, etc.
- _[http://hpcf.nersc.gov/software/libs/math/](http://hpcf.nersc.gov/software/libs/math/)_
- Visualization
- NCAR
- http://hpcf.nersc.gov/libs/graphics/
- IO
- HDF, netCDF, etc.
- http://hpcf.nersc.gov/software/libs/io/

## Accounts & Charging I

- Repositories
- Resource allocations are made to repositories
- Allocations reset each FY, starting Oct. 1
- CS267 account is called **mp309**
- Machine Charges
- Historical “strange” charging algorithm
- Charge = wallclock * nodes * 16 * 2.5 * priority
- E.g.: 1 hour running on 1 node in the regular class:
- (1) * (1) * (16) * (2.5) * (1.0) = **40 MPP Hours**
- NIM (NERSC Information Management)
- Check your usage at _[http://nim.nersc.gov/](http://nim.nersc.gov/)_
- NIM password is different from Seaborg password
- Call 486-8612 for NIM password

## Accounts & Charging II

- HPSS
- Charge for storage and transfer to HPSS
- Check usage at _[http://hpcf.nersc.gov/accounts/storage/](http://hpcf.nersc.gov/accounts/storage/)_
- (Don’t be too concerned about “SRU”s)
- Passwords
- Separate passwords for Seaborg, NIM, HPSS, servers
- Call NERSC Account Support: 486-8612
- Log into **sadmin.nersc.gov** to change Seaborg password
- Policies
- No sharing of accounts or passwords
- _[http://hpcf.nersc.gov/policy/](http://hpcf.nersc.gov/policy/)_

## Documentation

- HPCF Web Site
- “High Performance Computing Facility”
- _[http://hpcf.nersc.gov/](http://hpcf.nersc.gov/)_
- Site for all “user” information
- IBM Documentation Server
- Difficult to find relevant information in IBM docs
- NERSC tries to integrate IBM docs into single useful site
- _[http://usgibm.nersc.gov/](http://usgibm.nersc.gov/)_

## Getting Help

- Consulting Staff
- 10 consultants
- Help for all technical questions
- Account Support Staff
- All account issues, including passwords
- Help Desk
- 8-5 Pacific time, M-F
- Consultants: 486-8611, _[consult@nersc.gov](mailto:consult@nersc.gov)_
- Account support: 486-8612, support@nersc.gov
- Online Help Desk - http://help.nersc.gov/
- Use NIM usernames/passwords
- FAQs and submit questions online