# Presentation: 868366

- Migrating from the Crays to the IBM SP
- July 30, 2002
- David Turner
- NERSC User Services Group
- dpturner@nersc.gov
- 510-486-4027

## Introduction

- Scope
- Introduce Cray users to IBM SP system
- Compare hardware, environment, software
- Discuss performance expectations
- Examples
**Not** an MPI tutorial
**Not**  a detailed decommissioning plan

## Cray Vector Computers

- 1978: Cray1, 138 Mf/s, serial #6, CTSS
- 1981: second Cray1
- 1984: XMP, 2 CPUs
- 1985: Cray2, 4 CPUs
- 1988: Cray2, 4 CPUs
- 1990: Cray2, 8 CPUs
- 1991: Cray3 contract cancelled, C90 ordered
- 1992: YMP (8 CPUs), until C90 (16 CPUS/s) delivered
- 1996: J90 cluster
- 1999: SV1 upgrade

## Cray MPP Computers

- 1994: T3D
- 1996: T3E-600, 128 CPUs
- 1997: T3E-900, 512 CPUs
- 1998: Two T3Es merged into single machine, 692 CPUs

## Cray Retirement

- T3E hardware paved way for SP
- Software environment a bigger issuer

## Processor Comparison

- SV1
- Cray vector; 300 MHz; 1.2 Gf/s
- 20 or 24 CPUs per chassis; 68 total
- Total user flops: 82 Gf/s
- T3E
- DEC Alpha EV5 microprocessor; 450 MHz; 900 Mf/s
- 644 application, 33 command, 15 OS; 692 total; 512 user maximum
- Total user flops: 580 Gf/s
- SP
- IBM POWER3 microprocessor; 375 MHz; 1.5 Gf/s
- 16 CPUs/node; 184 compute (2944 CPUs); 3 login; 16 GPFS; 5 spares; 208 total; 2048 user maximum
- Total user flops: 4.4 Tf/s

## Memory Comparison

- SV1
- 8 GB/chassis; 4 GB user max
- 256 KB cache
- T3E
- 256 MB/PE; 128 GB user max
- 8 KB data cache, 96 KB secondary cache
- SP
- 16 GB/node (64 nodes have 32 GB, 4 nodes have 64 GB); 2 TB user max
- 64 KB L1 data cache (128 B/line), 8 MB L2 cache

## Disk Comparison

- SV1
- 134 GB home; 5 GB quota; 5000 inode quota
- 332 GB, 334 GB, 385 GB /tmp; no quotas, but 80 GB batch limit
- T3E
- 176 GB home; 2 GB quota; 3500 inode quota
- 1.5 TB /tmp; 75 GB quota, 6000 inde quota
- SP
- 6.9 TB home; 10 GB quota; 15000 inode quota
- 12.7 TB /scratch; 120 GB quota; 25000 inode quota

## File Systems

- Crays
- $TMPDIR
- Deleted at end of session
- /tmp
- Persistent, but not permanent
- SP
- $SCRATCH
- Persistent, but not permanent
- /tmp
_***DO NOT USE***_; can crash system

## Operating Systems

- SV1
- Unicos 10.0.1.1
- T3E
- Unicos/mk 2.0.5.59
- SP
- AIX 4.3

## Accounts

- SV1
- Multiple usernames (one per repository)
- T3E
- Single username, multiple repositories
- SP
- Single username, multiple repositories
- Change password on sadmin.nersc.gov, wait one hour to propagate
*Accounts must be requested!*

## Access Issues

- Might need
**ssh –P seaborg**
**scp –L seaborg:*****file1***** *****file2***
- Or add to **~/.ssh/config**:
**UsePrivilegedPort no**

## Software Environment

- Crays
- modules for Cray and third-party software
- sh, csh, ksh; tcsh and bash through modules
- multiple versions of compilers, etc.
- recommend users customize *extension* files
**.cshrc.ext, .login.ext, .profile.ext, ...**
- SP
- modules for third-party software
- sh, bsh, csh, ksh, tcsh, bash
- current version only of compilers, etc.
- users *must* customize extension files

## Mathematical Libraries I

| Name | SV1 | T3E | SP |
| --- | --- | --- | --- |
| Aztec (Linear Algebra) | No | Yes | Yes |
| SciLib | Yes | Yes | No |
| ESSL | No | No | Yes |
| FFTW (Fourier Transforms) | No | Yes | Yes |
| Harwell | Yes | No | No |
| IMSL | Yes | Yes | Yes |
| LAPACK | Yes | Yes | Yes |
| MASS (Tuned for SP) | No | No | Yes |
| METIS (Graph Partitioning) | No | No | Yes |

## Mathematical Libraries II

| Name | SV1 | T3E | SP |
| --- | --- | --- | --- |
| NAG | Yes | Yes | Yes |
| NAG Parallel | No | No | Yes |
| PESSL (Parallel ESSL) | No | No | Yes |
| PETSc | No | Yes | Yes |
| ScaLAPACK | No | Yes | Yes |
| SLATEC | Yes | No | No |
| SuperLU (Linear Algebra) | Yes | Yes | Yes |
| VSS (Sparse Solver) | Yes | No | No |

## Other Libraries

| Name | Description | SV1 | T3E | SP |
| --- | --- | --- | --- | --- |
| CERNLIB | Physics | No | Yes | Yes |
| HDF | I/O | Yes | Yes | Yes |
| HDF5 | I/O | Yes | Yes | Yes |
| MPI | Message Passing | Yes | Yes | Yes |
| MPI I/O | I/O | No | Yes | Yes |
| NCAR | Graphics | Yes | No | Yes |
| netCDF | I/O | Yes | Yes | Yes |

## Scientific Applications

| Application | Description | SV1 | T3E | SP |
| --- | --- | --- | --- | --- |
| Amber | Molecular Dynamics | Yes | Yes | Yes |
| ANSYS | Structural Analysis | Yes | No | No |
| CCM3 | Climate Modeling | Yes | No | Yes |
| CSM | Climate Modeling | Yes | No | No |
| CHARMM | Molecular Dynamics | Yes | Yes | Yes |
| Gaussian 98 | Molecular Electronic Structure | Yes | No | Yes |
| GAMESS | Molecular Electronic Structure | Yes | No | *** |
| ITS | Nuclear Transport | Yes | No | No |
| MAFIA | Electomagnetic Field Simulation | Yes | No | No |
| MCNP | Nuclear Transport | Yes | No | No |
| MCTDH | Time-dependent Electronic Structure | Yes | No | Yes |
| MOLPRO | Molecular Electronic Structure | Yes | No | Yes |
| NASTRAN | Structural Analysis | Yes | No | No |
| NWChem | Molecular Electronic Structure | Yes | Yes | Yes |

## Programming Tools I

- SV1
- Debugging
**totalview**
- Performance
**hpm**
**atexpert** (no longer supported)
- T3E
- Debugging
**totalview**
- Performance
**pat**
**apprentice** (no longer supported)

## Programming Tools II

- SP
- Debugging
**totalview**
**pdbx**
**pedb**
**zerofault**
**assure**
- Performance
**hpmcount / poe+**
**papi**
**tau**
**vampir**
**xprofiler**

## Compilers

- SV1
**f90, cc, CC**
- T3E
**f90, cc, CC, **
**pghpf, KCC**
- SP
**xlf, xlf90, xlf_r, xlf90_r,****mpxlf, mpxlf90, mpxlf_r, mpxlf90_r**
**xlc, ...**
**pghpf, KCC**

## Compiler Options I

| Cray f90 | IBM xlf90 | Meaning |
| --- | --- | --- |
| -p | -I | .mod search directory |
| -f fixed | -qfixed | F77-style source |
| -N 72|80|132 | -qfixed=72|80|132 | Number of columns |
| -eI | -u | “implicit none” |
| -m4 | -w | Suppress warnings |
| -r | -qsource | Produce listing |
| -m{0-4} | -qflag={I,L,W,E,S,U} | Message level |
| -Omsgs,negmsgs | -qreport=smplist | Parallelization report |

## Compiler Options II

| Cray f90 | IBM xlf90 | Meaning |
| --- | --- | --- |
| -Rbc | -C | Array bounds checking |
| -Ra | -qextchk | Argument checking |
| -e0 | -qinitauto | Initialize locals to 0 |
| -eR | -qrecur | Recursive subroutines |
| -Oieeeconform | -qfloat=rndsngl | IEEE-conformant math |
| -ef | -p, -pg | Profiling |
| -O{0-3} | -O{2-4} | Optimization level |
| -ev | -qsave | Static locals |
| -Otask{0-3} | -qsmp=nested_par | Automatic parallelization |

## Execution Models

- SV1
- Sequential
- OpenMP (or Autotasking)
- MPI
- T3E
- Sequential
- MPI (or shmem)
- SP
- Sequential
- MPI (or LAPI)
- OpenMP (or Pthreads)
- Mixed (e.g., OpenMP and MPI)

## Batch Systems

- Cray
- NQE/NQS
**cqsub, cqstatl, cqdel**
- IBM SP
- LoadLeveler
**llsubmit, llqs, llcancel**

## Batch Categories

- SV1 queues
- st_80, st_256, mt_512
- 120 CPU-hour limit; low, regular, premium
- T3E queues
- pe16, pe32, pe64, pe128, pe256, pe512
- 6 MPP-hour (wall) limit; low, regular, premium
- long64, long128, long256
- 24 MPP-hour (wall) limit; low, regular, premium
- SP classes
- low, regular, premium
- 2048 processor, 8 hour (wall) limits
- regular_long
- 512 processor, 24 hour (wall) limits

## Matrix Multiply (Naive)

**      ****subroutine seqaxb(a, b, c, n)**
**      ****integer n**
**      ****real*8 a(n, n), b(n, n), c(n, n)**
**      ****integer i, j, k**
**c**
**      ****do i = 1, n**
**        ****do j = 1, n**
**          ****do k = 1, n**
**            ****c(i, j) = c(i, j) + a(i, k) * b(k, j)**
**          ****enddo**
**        ****enddo**
**      ****enddo**
**c**
**      ****return**
**      ****end**

## MM Options

- SV1
**-O3**
- T3E
**-O3**
- SP
**-O3 –qstrict –qtune=pwr3 –qarch=pwr3**

## MM Results

-           **        ****SV1        T3E         SP **
**     ****N    sec   Mf/s    sec     sec   Mf/s**
**   ****100  0.005  331.7   0.04    0.02  123.5**
**   ****200  0.027  595.3   0.07    0.08  200.0**
**   ****300  0.085  639.4   0.15    0.26  209.1**
**   ****400  0.163  792.4   0.29    2.91   44.7**
**   ****500  0.309  816.7   0.53   13.33   19.2**
**  ****1000  2.276  889.3   3.60  110.40   18.3**
**  ****1100  3.273  823.4   4.94  148.04   18.1**
**  ****1200  3.910  895.1   6.30  192.88   18.0**
**  ****1300  5.270  844.5   8.17  247.17   17.9**
**  ****1400  6.206  894.8   9.70  308.13   17.9**
**  ****1500  8.139  839.4  12.38  379.76   17.9**

## Better Matrix Multiply

**      ****subroutine seqaxb(a, b, c, n)**
**      ****integer n**
**      ****real*8 a(n, n), b(n, n), c(n, n)**
**      ****integer i, j, k**
**c**
**      ****do k = 1, n**
**        ****do j = 1, n**
**          ****do i = 1, n**
**            ****c(i, j) = c(i, j) + a(i, k) * b(k, j)**
**          ****enddo**
**        ****enddo**
**      ****enddo**
**c**
**      ****return**
**      ****end**

## Better MM Results

**             ****SV1        T3E       SP**
**     ****N    sec   Mf/s    sec    sec   Mf/s**
**   ****100  0.006  327.4   0.04   0.01   51.2**
**   ****200  0.028  573.1   0.07   0.05  287.1**
**   ****300  0.077  705.9   0.15   0.16  323.7**
**   ****400  0.172  754.3   0.29   0.37  335.2**
**   ****500  0.316  800.6   0.54   0.80  309.1**
**  ****1000  2.365  855.7   3.60  10.64  186.9**
**  ****1100  3.334  808.4   4.94  15.78  168.3**
**  ****1200  4.080  857.7   6.20  22.00  156.7**
**  ****1300  5.436  818.7   8.31  29.20  150.1**
**  ****1400  6.382  870.3   9.69  37.27  146.9**
**  ****1500  8.191  834.0  12.34  46.22  145.8**

## Large Memory Options

**% ./mm 3000**
**1525-108 Error encountered while attempting to allocate a data object.  The program will stop.**
- Could also fail with “**segmentation violation**”
- Probably need:
**-bmaxdata:0x80000000 –bmaxstack:0x10000000**
- Local variables might also need:
**SAVE X(MAXBIG), Y(MAXBIG), Z(MAXBIG)**

## Message Passing MM

**do i = 1, numblk**
**  ****call seqaxb(a, b, c, blksiz)**
**  ****call mpi_sendrecv_replace(a, blksizsq, MPI_REAL8, **
**                            ****rankwest, WEST,**
**                            ****MPI_ANY_SOURCE, WEST,**
**                            ****. . .)**
**  ****call mpi_sendrecv_replace(b, blksizsq, MPI_REAL8, **
**                            ****ranknorth, NORTH,**
**                            ****MPI_ANY_SOURCE, NORTH,**
**                            ****. . .)**
**enddo**

## Message Passing Results

| PEs | N | Mf/s |
| --- | --- | --- |
| 4 | 400 | 18.2 |
|  | 800 | 132.8 |
|  | 1600 | 508.1 |
|  | 3200 | 497.1 |
| 16 | 800 | 132.9 |
|  | 1600 | 886.8 |
|  | 3200 | 2096.5 |
|  | 6400 | 854.9 |
| 64 | 1600 | 944.7 |
|  | 3200 | 5579.0 |
|  | 6400 | 9700.5 |
|  | 12800 | 3478.4 |

## Resources

- Web
- http://hpcf.nersc.gov
- http://hpcf.nersc.gov/computers/crayretire.html
- http://hpcf.nersc.gov/computers/SP/
- http://hpcf.nersc.gov/computers/SP/craytoSP.html
- http://hpcf.nersc.gov/computers/SP/xlf_cf90_flags.html
- http://hpcf.nersc.gov/software/ibm/sp_memory.html
- http://hpcf.nersc.gov/running_jobs/ibm/batch.html
- http://usgibm.nersc.gov
- Consulting
- http://help.nersc.gov
- consult@nersc.gov
- 1-800-66-NERSC, menu option 3
- 1-510-486-8600, menu option 3

## Resources

- Web
- http://hpcf.nersc.gov
- http://hpcf.nersc.gov/computers/crayretire.html
- http://hpcf.nersc.gov/computers/SP/
- http://hpcf.nersc.gov/computers/SP/craytoSP.html
- http://hpcf.nersc.gov/computers/SP/xlf_cf90_flags.html
- http://hpcf.nersc.gov/software/ibm/sp_memory.html
- http://hpcf.nersc.gov/running_jobs/ibm/batch.html
- http://usgibm.nersc.gov
- Consulting
- http://help.nersc.gov
- consult@nersc.gov
- 1-800-66-NERSC, menu option 3
- 1-510-486-8600, menu option 3

## Resources

- Web
- http://hpcf.nersc.gov
- http://hpcf.nersc.gov/computers/crayretire.html
- http://hpcf.nersc.gov/computers/SP/
- http://hpcf.nersc.gov/computers/SP/craytoSP.html
- http://hpcf.nersc.gov/computers/SP/xlf_cf90_flags.html
- http://hpcf.nersc.gov/software/ibm/sp_memory.html
- http://hpcf.nersc.gov/running_jobs/ibm/batch.html
- http://usgibm.nersc.gov
- Consulting
- http://help.nersc.gov
- consult@nersc.gov
- 1-800-66-NERSC, menu option 3
- 1-510-486-8600, menu option 3