# Presentation: 153151

## ORNL XT3 Tutorial

- Mark Fahey
- faheymr@ornl.gov

## Agenda

- ORNL XT3
- XT3 programming environment
- Application development, compilers, libraries
- Catamount programming considerations
- Running an application: yod, PBSPro
- Watching an application
- Debugging an application: Totalview
- Profiling an application: PAPI, CrayPat, Apprentice2

## Acknowledgement

- Research sponsored by the Mathematical, Information, and Computational Sciences Division, Office of Advanced Scientific Computing Research, U.S. Department of Energy, under Contract No. DE-AC05-00OR22725 with UT-Battelle, LLC.

## Cray XT3

- 3rd generation MPP platform
- purpose-built for HPC applications
- designed for large problems
- engineered to deliver superior performance for massively parallel applications, including:
- scalable processing elements, high performance AMD Opteron processors and memory
- high bandwidth, low latency (eventually) interconnect
- MPP optimized operating system
- standards-based programming environment
- sophisticated RAS and system management features
- high speed, highly reliable I/O system (eventually)

## ORNL XT3 (Jaguar)

**Compute PE       (5212) **
**Login PE           (6)**
**Network PE         (2)**
**System PE          (3)**
**lustre PE         (50)**

**Service Partition (Full Linux kernel, Suse 9, kernel 2.4), 4 GB RAM**

**Compute Partition (Catamount kernel)**
**2 GB RAM**

***Specialized Linux nodes***

**Compute PE: Nodes to run massively parallel application, light weight kernel Catamount**
**Login PE: to work for users in Linux based environement: compiling, editing, PBS resource managers,..**
**Network PE: provide TCP/IP connections to external systems **
**System PE: running global system services: boot, administration, ....**
**lustre PE: handling I/O to the global, parallel file system from applications and storage devices**

## ORNL XT3 compute node

- 2.4 GHz AMD Opteron, model 150
- Peak of 4.8 Gflops/s
- 64KB L1 data cache and 64KB L1 instruction cache, both 2-way assoc.
- 1MB L2 cache
- 2 GB memory
- Catamount Kernel

## HPCC numbers on 128p*

| Test | measure | XT3
2.4ghz | XD1
2.4ghz | X1E
1.13ghz | Power4+
1.7ghz |
| --- | --- | --- | --- | --- | --- |
| HPL | Gflops | 0.499 | 0.502 | 1.31 | 0.53 |
| PTRANS | GB/s | 31.5 | 13.5 | 56.4 | 7.77 |
| STREAM | GB/s | ~4.5 | ~4.5 | 32.5 | 12 |
| RndAccess | GUPS | 0.0045 | 0.067 | 1.77 | 0.00283 |
| Glbl. FFT | Gflops | 38.8 | 35.5 | 15.1 | 4.38 |
| RndRingB | GB/s | 0.276 | 0.259 | 0.592 | 0.747 |
| RndRingL | usec | 43.7 | 2.06 | 13.9 | 7.94 |

- *124 MSPs on X1E

## Programming Environment

- What are defaults?
- >module list
- Currently Loaded Modulefiles:
-  1) acml/2.5           7) xt-pe/1.1.08       13) xt-catamount/1.1.08
-  2) pgi/6.0.1          8) xt-libsci/1.1.08   14) xt-boot/1.1.08
-  3) totalview/6.8.0-0  9) xt-mpt/1.1.08      15) xt-crms/1.1.08
-  4) papi/3.0.8.1      10) xt-service/1.1.08  16) xt-lustre-ss/1.1.08
-  5) elf/0.8.5         11) xt-libc/1.1.08     17) PrgEnv/1.1.08
-  6) dwarf/4.2.0       12) xt-os/1.1.08       18) xt-pbs/5.3.1

## Prog. Env. (cont.)

- What is available?
- Shortened list below
- >module avail
- ------------------------- /opt/modulefiles -------------------------
- PrgEnv/1.1.06         gnet/2.0.5            xt-libc/1.1.08
- acml/2.5              papi/2.3              xt-libsci/1.1.08
- acml-gnu/2.5          papi/3.0.8.1(default) xt-lustre-ss/1.1.08
- acml-mp/2.5           pgi/6.0.4             xt-mpt/1.1.08
- dwarf/4.2.0(default)  pgi32/6.0.4           xt-os/1.1.08
- elf/0.8.5(default)    pkg-config/0.15.0     xt-pe/1.1.08
- gcc/3.2.3             xt-boot/1.1.08        xt-service/1.1.08
- gcc-catamount/3.3     xt-catamount/1.1.08
- glib/2.4.2            xt-crms/1.1.08

## Prog. Env. (cont.)

- module tool used on XT3 (and other NCCS resources) to handle different versions of packages
-       (compiler, MPICH, tools,...):
-       **e.g.:** **module load compiler1  **
**      ****e.g.: module switch compiler1 compiler2**
**      ****e.g.: module load totalview**
**       ****.....**
- takes care of changing of PATH, MANPATH, LM_LICENSE_FILE,.... environment
- users should not set these environment variables in shell startup files, makefiles, batch scripts, etc....
- keep things flexible to other package versions

## Compilers

- Portland Group
- Fortran 95, Fortran 77, C, and C++
- pg[f77,f95,cc,CC] can be used to generate executables for service nodes
- ftn, f77, cc, and CC drivers used to generate catamount executables
- the compiler drivers take care of loading appropriate libraries
- (-lmpich, -lsci, -lacml, -lpapi)
- Gnu
- gcc/3.2.3 can be used to generate executables for service nodes
- gcc-catamount/3.3 is only for internal use yet, the appropriate handling with other packages like mpt, sci,... has to be found  (end of the year)
- so stay with the modules provided per default so far (this may change with newer compiler versions, additional packages in the future)

## Hello world

**cat >hello.f**
**      ****include ‘mpif.h’**
**      ****call MPI_Init( MPI_COMM_WORLD, ie )**
**      ****call MPI_Comm_rank( MPI_COMM_WORLD, me, ie )**
**      ****print *, “Hello world, from “, me**
**      ****call MPI_Finalize( MPI_COMM_WORLD )**
**      ****end**
**% ftn -o hello --target=catamount hello.f**
**% yod -sz 2 hello**
**Hello world, from 0**
**Hello world, from 1**
- No flags are needed for include or libraries for mpich, SciLib, ACML or PAPI.  Try “-dryrun” if you’re curious what is hidden under the covers.
- The compiler driver scripts are part of the PrgEnv module, and are named “cc”, “CC”, “f77” or “ftn”.

- It is by far easiest to use the wrapper scripts “cc”, “ftn”, etc. when compiling or linking to a Catamount executable.  These default to parameters suitable for a Catamount target, and they tell you that:
**/opt/xt-pe/1.0/bin/snos64/ftn: INFO: catamount target is being used**
- You can eliminate this warning with --target=catamount
- If you do intend to build compute only (no MPI) objects, you can carefully use the base names “pgcc”, “pgCC”, etc. Do not use gcc or g++ directly unless you intend to build a Linux object or executable.

## Libraries

- MPI-2 message passing library for communication between nodes
- derived from MPICH-2, implements MPI-2 standard, except for support of dynamic process functions
- SHMEM
- GNU C library
- subset of glibc, version 2.3.2 is implemented on Catamount
- optimized libraries:
- 64 bit AMD Core Math library (ACML) version 2.5
- Level 1,2,3 of BLAS, LAPACK, FFT
- SciLib:  Scalapack, BLACS, SuperLU
- FFTW, NetCDF
- Will be modules eventually
- For now, get them from /apps/
- , HDF5, libsz
- HDF5 and libsz in ~larkin/xt3/jaguar
- Will be in /apps and modules

## Programming Considerations

- glibc on Catamount does not support:
- pipes, sockets, TCP/IP communication
- MPI and SHMEM are the only mechanism to communicate
- no PVM
- dynamic process controll (e.g. exec, popen, fork, system)
- dynamic loading of executable code (static executables)
- no pthreads
- no OPENMP
- no /proc/cpuinfo or /proc/meminfo files on compute nodes, just on login nodes
- If you are not sure if something is supported, try man pages, e.g.:
- “man mmap” will say
**not supported on Cray XT3 compute partitions**

## Programming Considerations (cont.)

- XT3 macros
- Standard preprocessor macro for XT3 to run under catamount is:
-  __QK_USER__
- Also defined is
- __LIBCATAMOUNT__
- __x86_64__ gets defined by the PGI compiler
- Use “touch foo.h; cpp –dM foo.h” to see predefined compiler macros
- Won’t show the 1st two macros though

## MPICH_IGNORE_CXX_SEEK

- There is a name conflict between stdio.h and MPI C++ binding in relation to the names SEEK_SET, SEEK_CUR, SEEK_END
- Solution right now, either
- Undef the symbols before including mpi.h,
- Change the order of includes, mpi.h before stdio.h or iostream, or
- If MPI I/O flags are not of use, simply compile with
- -DMPICH_IGNORE_CXX_SEEK

## System call not supported

- During link, you will see messages like:
- /opt/pgi/6.0.1/linux86-64/6.0/lib/libpgf90rtl.a(system3f.o)(.text+0x24): In function `system_':
- : warning: system is not implemented and will always fail
**WRITE(LINE,'(A,A)') 'rm -f ',FILEN(IA:IE)‘**
**#if defined(CRAY_XT3)**
**          ****LINE=FILEN(IA:IE)//'\0'**
**          ****call f2c_removefile(LINE)**
**#else**
**          ****CALL SYSTEM(LINE)**
**#endif**
**void f2c_removefile_(const char *filename)**
**{**
**#include <stdio.h>**
**int rc;**
**rc = remove(filename);**
- }
- Do this also for copy file, remove directory, create directory,.....

## Example

- #if defined(CRAY_XT3)
-           call makedirectory(DIRN)
- #else
-           CMD= ‚‘mkdir ‘ // DIRN
-           CALL SYSTEM(CMD)
- #endif
- void makedirectory_ (char *dirname){
-   mkdir(dirname,0755);
- }

## Timer Support

- There is no real distinction made between “elapsed time” and “cpu time” for Catamount.
- The getrusage() call returns user time and system time equal
- The commonly used timers “clock”, “etime” and “times” cause dummy stubs to be loaded along with a warning message.
- timing.c:70: warning:**   clock** is not implemented and will always fail
- timer.c:35: warning: **times** is not implemented and will always fail
**you still produce an executable!**
- Instead, use:
- Fortran: cpu_time() intrinsic returns processor seconds
- Also portable, MPI_Wtime() is a useful elapsed-time clock
- gettimeofday() works, but is the usual Unix/Linux low-accuracy (100Hz rate).
- dclock() is another C/Fortran cpu time clock, with a nominal resolution of 100 nanoseconds.  However it is based on the processor clock, with CPU frequency uncalibrated.  It may be off as much as +/-50 micro-seconds per second
- Available later (end of 2005):  etime, dtime, clock

## Example

- extern "C" {
- #include <catamount/dclock.h>
- }
- #include <stdio.h>
- #include <unistd.h>
- double dclock (void);
- main(){
- double start_time, end_time, elapse;
-    **start_time =dclock() ;**
-    sleep(5) ;
-    **end_time = dclock() ;**
-    elapse = end_time - start_time ;
-    printf("elapse %f \n",elapse) ;
- }

## Example

**Examples dclockf.F90:**
- program main
-    real*8 start_time, end_time, elapse,dclock
-    **start_time =dclock()**
-    call sleep(5)
-    **end_time = dclock()**
-    elapse = end_time - start_time ;
-    write(*,*)"elapse ",elapse
- end
- ftn dclockf.F90 -o dclockf
- /opt/xt-pe/1.0/bin/snos64/ftn: INFO: catamount target is being used
- dclockf.F90:

## Creating Catamount Executables

- do not use pgf90, mpif90,...
- To make sure that you are creating a Catamount executable and not a Linux executable for service nodes!!
- > ftn -O3  -I../COMMON -I../LIB  -Dparallel   -Mpreprocess -Mfree  -c
-      /opt/xt-pe/1.0/bin/snos64/ftn: INFO: **catamount target is being used**
- we are cross-compiling:
- compiling an executable for a Catamount node on a LINUX kernel node
**man pgf90** or **man pgcc** or **man pgCC** to get general PGI compiler options

- With initial releases it is not possible to build Linux (SIO node) MPI sets, or mixes of SIO/Catamount nodes

## Configure on the XT3

- setenv CC cc setenv CXX CC setenv F77 ftn setenv F90 ftn
- ./configure  <-options>
- Where options could be depending on configure script:
**--****host=x86_64-unknown-linux-gnu**
- cross-compile to build programs, to prevent execution of Catamount executable on Linux kernel node
**--enable-static**
- build only statically linked executables

## IO considerations

- I/O support for compute node applications is limited: only Fortan, C, C++ I/O call, MPI and (later SHMEM I/O) functions are allowed
- No I/O with named pipes on Catamount
- a special file that acts as a buffer to connect processes on the same machine
- can be read from and written to just as though they were ordinary files
- I/O is offloaded to the service I/O nodes
- yod application launcher handles stdin, stderr and stdout
- I/O is forwarded as RPC requests to the launching yod process
- By default stdio is unbuffered: this results in bandwidth limitation since it is offloaded to yod
- there are routines to improve this with buffering I/O: setvbuf

## Lustre

- If Lustre is configured on your system, this is of course the better option. We have a “xt-lustre-ss” module as a default, no special action is required.
- So “-llustre” is done automatically by ftn and cc
- “How can I tell if Lustre is configured on my system?”
- Look for mounts of the form:
**% df -t lustre**
**Filesystem                1K-blocks     Used  Available Use% Mounted on**
**7:/nid00007-mds/client    846749008 93780876  709955704 12% /lus/nid00007**
**135:/nid00135_mds/client 3289403772  2651208 3119660552  1% /lus/nid00135**
- The XT3 Programming Environment guide (Cray document S-2396-11) has a large section describing exactly how to make sure you are getting parallel I/O.  See section 4.3

## Compiler flags

- -Mlist                          : create a listing file
**Preprocessor Options:**
- -Mpreprocess             : run the preprocessor on Fortran files (default on .F, .F90, or .fpp files)
**Optimization Options:**
- -fast                                    : chooses generally optimal flags for the target platform
- -fastsse                              : Chooses generally optimal flags for a processor that supports
-                                              the SSE and SSE2 instructions.
- -Mnodepchk                       : don't assume that potential data dependencies exist
- -Minline=levels:number     : number of levels of inlining are performed
**Debug Options:**
- -g                         : Generate symbolic debug information
- -Mbounds            : Array bounds checking
- -Mchkptr              : Check for unintended de-referencing of null pointers

## Compiler flags (cont.)

**Language Options:**
- -Mfree                       : process Fortran source using free form
-                                     specifications
- -Mnomain                  : useful for using the ftn driver to link programs
-                                     with the main program written in C or C++ and one
-                                     or more subroutines written in Fortran.
- -byteswapio              : can be used to read big-endian files in Fortran.
-                                    XT3 is little endian.
- -Mflushz                    : Set SSE to flush-to-zero mode
- -i8                              : treat INTEGER variables in Fortran as eight bytes
- -r8                             : treat REAL variables in Fortran as eight bytes
**                                   ****(be careful using these in general, **
**                                   ****since this information can not be passed**
**                                   ****through library calls)**

## SSE

**SSE** means "Streaming SIMD Extensions„, which enhance the x86 instruction set (actual now SSE2):
- 8 new 128-bit SIMD floating-point registers that can be directly addressed;
- 50 new instructions that work on packed floating-point data;
- + some MMX instructions for multimedia, video, mpeg
- SSE2:
- for 32bit variables: vector instructions with vector length 4;
- for 64bit variables: vector instructions with vector length 2!

## Vectorize on XT3?

- Is it important to vectorize? What does AMD say?
- “Strongly recommended for single precision arithmetic.”
- Less important for double precision
- Others have found it is helpful most of the time
- What vectorizes? And what does not?
- Only Inner stride-1 loops vectorize
- Loops with if’s do not
- Loops with any indirect or none unit stride references do not
- Outer stride-1 loops do not
- May have problems with large body loops
- -Mvect=nosizelimit

## Running Jobs

## yod

- Instead of mpirun, use the yod command
- yod running on service node to take over runtime components which require a full Linux operating system
-               yod –size #ncpus exec
-       or     yod –sz #ncpus exec
- Launching several executables using the same MPI_COMM_WORLD
-      process communicator
-      > cat loadfile
-      #loadfile
-      yod –sz 8   progr1
-      yod –sz 10 progr2
-      > yod –F loadfile

## Interactive work

- Even for 1-processor test runs, use yod
- > make ex1
- > cc     ex1.c   -o ex1
- > ./ex1
- Segmentation fault
- > yod -sz 1 ./ex1
- Done!
- Yod must be used even for 1 CPU jobs!

## Batch jobs

- Here is an example script
**> cat job.pbs**
**#!/bin/ksh**
**#PBS -l size=64,walltime=12:00:00**
**#PBS -j oe**
**#PBS -N myjob**
**cd $PBS_O_WORKDIR**
**yod -sz $PBS_NNODES executable arguments**
**> qsub job.pbs**
- For other options specific to the XT3, see the pbs_resources_xt3 (7B) man page

## qstat

- To see the status of running, held or pending batch jobs, use “qstat -a” command. See the qstat man page for details of other options.
- fm9@jaguar2> qstat -a
- nid00004: ORNL/CCS
-                                            Time In Req'd  Req'd   Elap
- Job ID Username Queue    Jobname    SessID  Queue  Nodes  Time  S Time
- ------ -------- -------- ---------- ------ ------- ------ ----- - -----
- 881    larkin   short    LSMS_128    19728  000:43    128 02:00 R 00:42
- 882    larkin   short    VASP+profi  19844  000:38    128 02:00 R 00:37
- Total generic compute nodes allocated: 256

## xtshowmesh

- Available compute nodes:       0 interactive,  4935 batch
- YODS LAUNCHED ON CATAMOUNT NODES
- Job ID     User       Size   Start            yod command line and arguments
- --- ------ --------  -----  ---------------  ----------------------------------
-  b  3094   larkin      128  Jul 19 10:27:50  yod -sz 128 ../vasp.4.6/vasp+pat
-  a  3096   larkin      128  Jul 19 10:58:52  yod -sz 128 -D all ../lsms_main+u

## fm9@jaguar2> xtshowmesh
Compute Processor Allocation Status as of Tue Jul 19 11:01:53 2005

                Cabinet  0                 Cabinet  1                 
                    1         2                1         2       
   Node-> 012345678901234567890123   012345678901234567890123   
  Row   0 LLLLLLLL||||||||||||||||   LLLL||||||||LLLLLLLLL|||
        1         ||||||||||||||||       ||||||||         |||
        2         ||||||||||||||||       ||||||||         |||
        3 LLLLLLLL||||||||||||||||   LLLL||||||||LLLLLLLLL|||
        4 ||||||||||||||||aaaaaaaa   ||||||||||||||||||||||||
        5 ||||||||||||||||aaaaaaaa   ||||||||||||||||||||||||
        6 ||||||||||||||||aaaaaaaa   ||||||||||||||||||||||||
        7 ||||||||||||||||aaaaaaaa   ||||||||||||||||||||||||
        8 aaaaaaaaaaaaaaaaaaaaaaaa   ||||||||||||||||||||||||
        9 aaaaaaaaaaaaaaaaaaaaaaaa   ||||||||||||||||||||||||
       10 aaaaaaaaaaaaaaaaaaaaaaaa   ||||||||||||||||||||||||
       11 aaaaaaaaaaaaaaaaaaaaaaaa   ||||||||||||||||||||||||
       12 bbbbbbbbbbbbbbbbbbbbbbbb   ||||||||||||||||||||||||
       13 bbbbbbbbbbbbbbbbbbbbbbbb   ||||||||||||||||||||||||
       14 bbbbbbbbbbbbbbbbbbbbbbbb   ||||||||||||||||||||||||
       15 bbbbbbbbbbbbbbbbbbbbbbbb   ||||||||||||||||||||||||
[... rest deleted ...]

- Compute Processor Allocation Status as of Tue Jul 19 11:01:53 2005
-                 Cabinet  0                 Cabinet  1
-                     1         2                1         2
-    Node-> 012345678901234567890123   012345678901234567890123
-   Row   0 LLLLLLLL||||||||||||||||   LLLL||||||||LLLLLLLLL|||
-         1         ||||||||||||||||       ||||||||         |||
-         2         ||||||||||||||||       ||||||||         |||
-         3 LLLLLLLL||||||||||||||||   LLLL||||||||LLLLLLLLL|||
-         4 ||||||||||||||||aaaaaaaa   ||||||||||||||||||||||||
-         5 ||||||||||||||||aaaaaaaa   ||||||||||||||||||||||||
-         6 ||||||||||||||||aaaaaaaa   ||||||||||||||||||||||||
-         7 ||||||||||||||||aaaaaaaa   ||||||||||||||||||||||||
-         8 aaaaaaaaaaaaaaaaaaaaaaaa   ||||||||||||||||||||||||
-         9 aaaaaaaaaaaaaaaaaaaaaaaa   ||||||||||||||||||||||||
-        10 aaaaaaaaaaaaaaaaaaaaaaaa   ||||||||||||||||||||||||
-        11 aaaaaaaaaaaaaaaaaaaaaaaa   ||||||||||||||||||||||||
-        12 bbbbbbbbbbbbbbbbbbbbbbbb   ||||||||||||||||||||||||
-        13 bbbbbbbbbbbbbbbbbbbbbbbb   ||||||||||||||||||||||||
-        14 bbbbbbbbbbbbbbbbbbbbbbbb   ||||||||||||||||||||||||
-        15 bbbbbbbbbbbbbbbbbbbbbbbb   ||||||||||||||||||||||||
- [... rest deleted ...]

## Killing a job

- Eventually,
- “qdel <jobid>”
- Where <jobid> is qstat output
- Today,
- Run “xtps -Y” to get yod job id
- Then, “xtkill -9 <pid>”
- Why:
- because qdel is not properly killing yod processes

## Error handling

- “User error”, that is, can generate SIGSEGV and related signals. Generally these will produce a call trace-back for the faulting process.
- Other options are covered in the “man core” man page; it is possible to produce a core file for the first faulting process or all of them (be cautious of this option!).
- man core

## Tools

- Open up tools presentation

## Simple optimization examples

- Open up optimization examples presentation

## XT3 MPI – Implications for Apps

- Sophisticated MPI apps structured to overlap medium-long (10-100s of KB) messages with computation should do better on XT3 than typical competing solutions (gige/IB clusters)
- Typical compute/communicate MPI apps (majority of MPI codes?) _may_ do okay.  However only better than competition to the extent that XT3 bandwidth exceeds that of competing products
- MPI apps that do a lot of short messages probably won’t do much better than IB clusters

## XT3 Gotchas (hardware related)

- Don’t expect industry leading MPI latency with XT3.  DMA-only based method+Portals overhead will prevent this.
- Currently using interrupt-based method for doing portals tag matching.  Very poor latency until matching is done in firmware (accelerated portals).
- Many-to-one communication patterns will be problematic for large jobs due to 256 entry limit in CAM table

## XT3 Gotchas (MPI specific)

- Be aware that XT3 MPI is tuned to treat messages up to 128000 bytes as short messages.  Could end up doing a lot of *memcpy’s* from short message buffers to application buffers if not careful
- Try to prepost buffers, especially for large messages.  Structure algorithms to insure that MPI_Irecv call on recv side will happen BEFORE sender will send data
- Avoid using barriers
- Don’t use derived types
- Collectives not optimized for mesh topology
- May be better to use point-to-point MPI calls

## XT3 MPI environment variables

| environment variable | description | default |
| --- | --- | --- |
| MPICH_MAX_SHORT_MSG_SIZE | Maximum message size to use EAGER protocol where sender sends message header+data to receiver. | 128000 bytes |
| MPICH_UNEX_BUFFER_SIZE | Size of each of the three unexpected message buffers maintained by each process. | 2 MB |
| MPICH_PTL_UNEX_EVENTS | Specifies size of event queue associated with short message buffers.  Basically maximum number of short messages that can be outstanding at a given process. | 2048 |

## Preinstalled Packages

- Physically in /apps/,
- some will be available by default, some will be modules
- FFTW
- NetCDF 3.6.0
- HDF5 1.6.4
- Not in /apps yet, in ~larkin/xt3/jaguar
- libsz and libz here too
- Coming
- PETSC 2.3
- SUPERLU
- SUPERLU_DIST

## Status of M3D on Jaguar

- Waiting on petsc