# Presentation: 809847

## Single Node Optimization on the NERSC SP

- October 13, 2003
- Michael Stewart
- NERSC User Services Group
- _[pmstewart@lbl.gov](mailto:pmstewart@lbl.gov)_
- 510-486-6648

## Introduction

- Why is optimization particularly important on the SP?
- The most useful compiler arguments and libraries for code optimization.
- Examples of the effects of various optimization techniques on various public benchmark codes.

## IBM Default:  No Optimization!

- Default compilation without any arguments:  no optimization!
- Can have very bad consequences:
-          do i=1,bignum
-              x=x+a(i)
-          enddo
*bignum* stores of x are done with the default, no optimization arguments specified.
- When optimized, **store motion **is done:  intermediate values of x are kept in registers and the actual store is done only once, outside the loop.

## Recommended Optimization Flags

- For all compiles - Fortran, C, C++:
**-O3  -qstrict  -qarch=pwr3  -qtune=pwr3**
- Compromise between minimizing compile time and maximizing compiler optimization.
- With these options, optimization only done within a procedure (e.g. subroutine, function).
- Numerical results bitwise identical to those produced by unoptimized compiles.
- Drawback:  does not optimize complex or even very simple nested loops well.

## Numeric Arguments: -O2 and –O3

**-O0 **and **-O1 **not currently supported.
**-O2**:  Intermediate level producing numeric results equal to those produced by an unoptimized compile.
**-O3**:
- More memory and time intensive optimizations.
- Can change the semantics of a program to optimize it so numeric results will not always be equal to those produced by an unoptimized compile unless **-qstrict** is specified.
- No POWER3 specific optimizations.
- Not very good at loop oriented optimizations.
- Most benchmarks achieve 90% or better of their maximum possible performance at the **–O3** level.

## Numeric Arguments: -O4

- Equivalent to “**–O3** **–qarch=auto** **–qtune=auto **
**    ****-qcache=auto** **-qipa** **-qhot**”.
- Inlining, loop oriented optimizations, and additional time and memory intensive optimizations.
- Performs interprocedural optimizations.
- Option should be specified at link time as well as compile time.
- If you are experimenting try “-**O4 –qnohot**” as well as             **“-O4**”, since most of the compilation time is due to **-qhot**.

## Numeric Arguments: -O5

- Equivalent to “**–O4** **–qipa=level=2**”.
- Full interprocedural optimization in addition to **–O4** optimizations.
- Option should be specified at link time as well as compile time.
- If you are experimenting, try  “-**O5 -qnohot**” as well as       “**–O5**”.

## -qstrict:  Equality of Numeric Results

- Semantics of a program are not altered regardless of the level of optimization, so numeric results are identical to those produced by unoptimized code.
- Inhibits optimization (in principle) - does not allow changes in the order of evaluation of expressions and prevents other significant types of optimizations.
- In practice, this option rarely makes a difference at the **–O3** level and can even improve performance.
- No equivalent on the Crays.

## -qarch:  Processor Specific Instructions

**-qarch=pwr3**:  Produces code with machine instructions specific to the POWER3 processor that can improve performance on it.
- Codes compiled with **-qarch=pwr3** may not run on other types of POWER or POWERPC processors.
- The default at the **–O2** and **–O3** levels is **-qarch=com** which produces code that will run on any POWER or POWERPC processor.
- Default for **–O4** and **–O5** is **–qarch=auto **(=pwr3) on seaborg.
- When porting codes from other IBM systems to seaborg, make sure that the **–qarch** option is either **pwr3** or **auto**.

## -qtune:  Processor Specific Tuning

**-qtune=pwr3**:  Produces code tuned for the best possible performance on a POWER3 processor.
- Does instruction selection, scheduling and pipelining to take advantage of the processor architecture and cache sizes.
- Codes compiled with **-qtune=pwr3** will run on other POWER and POWERPC processors, but their performance might be much worse than it would be without this option specified.
- Default is for no specific processor tuning at the **–O2** and **–O3** levels, and for tuning for the processor on which it is compiled at the **–O4** and **–O5** levels.

## -qhot:  Loop Specific Optimizations

- Now works with C/C++ as well as Fortran.
- Loop specific optimizations:  padding to minimize cache misses, "vectorizing" functions like sqrt, loop unrolling, etc.
- Works best on loop dominated routines, if the compiler has some information about loop bounds and array dimensions.
- Operates by transforming source code:  **-qreport=hotlist **produces a (somewhat cryptic) listing file of the loop transformations done when **-qhot** is used.
- Can double or triple compile time and may even slow code down at run time.
- Included by default with **–O4** or **–O5** compiles.

## -qipa:  Interprocedural Analysis

- Examines opportunities for optimization across procedural boundaries even if the procedures are in different source files.
**Inlining** - Replaces a procedure call with the procedure itself to eliminate call overhead.
**Aliasing **- Identifying different variables that refer to the same memory location to eliminate redundant loads and stores when a program's context changes.
- Can significantly increase compile time.
- Many suboptions (see man page).
- 3 ipa numeric levels:  **-qipa=level=*****n***.

## -qipa=level Optimizations

- Determines the amount of interprocedural analysis done.
- The higher the number the more analysis and optimization done.
**-qipa=level=0**:  Minimal interprocedural analysis and optimization.
**-qipa=level=1 **or **-qipa**:  Inlining and limited alias analysis.   **(-O4**)
**-qipa=level=2**:  Full interprocedural data flow and alias analysis.  **(-O5)**

## ESSL Library

- Single most effective optimization:  replace source code with calls to the highly optimized Engineering and Scientific Subroutine Library (ESSL) .
- The ESSL library is specifically tuned for the POWER3 architecture and has many more optimizations than those that can be obtained with **–qarch=pwr3** and **–qtune=pwr3**.
- Contains a wide variety of linear algebra, Fourier, and other numeric routines.
- Supports both 32 and 64 bit executables.
- Not loaded by default, must specify the **–lessl** loader flag to use.

## -lesslsmp: Multithreaded ESSL Library

- When specified at link time ensures that the multi-threaded versions of the essl library routines will be used.
- No change required to source code.
- Can give significant speedups if not all processors of a node are busy.
**Important**:  Default for a program linked with **–lesslsmp** is to use 16 threads when run on seaborg.  Change the number of threads by setting the **OMP_NUM_THREADS **environment variable to the desired number of threads.

## Fortran Intrinsics

- Fortran intrinsics like matmul and dot_product are multi-threaded by default at run time when a “thread safe” compiler (**_r** suffix) is used to compile the code.
- 16 threads are used by default at run time.
- Can control the number of threads used at run time by setting the environment variable XLFRTEOPTS=intrinthds=***n*** where ***n*** is the number of threads desired.
- The non-thread safe compilers produce code that is single threaded at run time.
- The performance of both the single and multi-threaded versions of the intrinsics are significantly worse than their **ESSL** equivalents.

## -qessl:  Optimize Fortran Intrinsics

**-qessl**:  replace Fortran intrinsics with the equivalent routine from the ESSL library.
- Must link with **–lessl** (single threaded) or **–lesslsmp **(multi-threaded).
- For the multi-threaded version it uses the same number of threads at run time as any ESSL or OpenMP routine in the code:  16 by default or the value of the environment variable **OMP_NUM_THREADS**.

## MASS Math Library

- The Mathematical Acceleration SubSystem (MASS) consists of libraries of tuned mathematical intrinsic functions.
- Highly optimized versions of these functions:  sqrt, rsqrt, exp, log, sin, cos, tan, atan, atan2, sinh, cosh, tanh, dnint, x**y.
- Usage:
**module load mass**    (**mass_64** for 64-bit applications)
**xlf progf.f -o progf $MASS**
**cc progc.c -o progf $MASS -lm**

## Other Useful Compiler Options

**-qsmp=auto **– Automatic parallelization.  The compiler will attempt to parallelize the code.  (16 threads by default at run time or the value of the environment variable **OMP_NUM_THREADS**).
**-Q+*****proc*** – Inline specific procedure *proc*.
**-qmaxmem=*****n*** – Limits the amount of memory used by the compiler to *n* kilobytes.  Default *n*=2048. *n*=-1 memory is unlimited.
**-C** or **–qcheck** – Check array bounds.
**-g** – Generate symbolic information for debuggers.
**-v** or **–V** – Verbosely trace the progress of compilations.

## Example:  Matrix Multiply(1)

- Multiply two 1000 by 1000 real*8 matrices.
- Directly:  **-O3 –qarch=pwr3 –qtune=pwr3 –qstrict**
- Fortran:   c(i,j)=c(i,j)+a(i,k)*b(k,j)
- C:   c[i][j]=a[i][k]*b[k][j]+c[i][j]
- Performance depends on the order of the index variables.
-                ijk     ikj    jik    jki     kij     kji
- Fortran   17       9    16   237       9    198 MFlops
- C            16   209    18       9   200        9 MFlops
- Add **–qhot** to compile and performance differences disappear: Both Fortran and C:  560 MFlops for all indices.

## Example:  Matrix Multiply(2)

- Add **–qsmp=auto** to compile and run dedicated with 16 threads.
- Fortran: 5060 MFlops.
- C: 5460 MFlops.
- Fortran Intrinsic **matmul**:
- Unthreaded compile or 1 thread:  946 MFlops.
- Threaded compile (default 16 threads):  14,690 MFlops.
- -qessl –lessl (1 thread):  1210 MFlops.
- -qessl –lesslsmp: (16 threads) 18,323 MFlops.
- ESSL routine DGEMM:
- -lessl:  1290 MFlops.
- -lesslsmp: (16 threads) 19,696 MFlops.

## Fortran Livermore Loops

- 24 numeric kernels.
- Written in “fairly straightforward, uncomplicated Fortran 77”.
- 8 different summary statistics returned including mean, minimum, and maximum MFlops for each kernel.

## Fortran Livermore Loops Timings

| Optimization Level | Compile Time | Mean    Min    Max  MFlops |
| --- | --- | --- |
| unoptimized |   1.45 |  47          11        141 |
| -O2 |   8.15 | 244         22       994 |
| -O3 –qarch=pwr3
 –qtune=pwr3 -qstrict | 13.54 | 322         60      1296 |
| -O3 –qarch=pwr3
 –qtune=pwr3 | 14.33 | 338         60      1298 |
| -O3 –qarch=pwr3
 –qtune=pwr3 -qhot | 65.29 | 314         58       996 |
| -O4 | 78.14 | 321         58       998 |
| -O5 | 89.16 | 331         58      1072 |
| -O4 -qnohot | 33.07 | 339         58      1294 |
| -O5 -qnohot | 48.01 | 363         58      1341 |

## NAS Kernels

- Seven Fortran Kernels described at _[http://www.netlib.org/benchmark/nas-doc](slide24.xml)_.
- MXM – Matrix-matrix multiply.
- CFFT2D – Complex 2D FFT.
- CHOLSKY – Cholesky decomposition.
- BTRIX – Tridiagonal solver.
- GMTRY – Gaussian elimination.
- EMIT – Creates new vortices according to certain boundary conditions.
- VP – Invert 3 pentadiagonals simultaneously.

## NAS Kernels

| Optimization Level | Compile time (secs) | Average MFlops |
| --- | --- | --- |
| unoptimized | .79 | 30.7 |
| -O2 | 2.92 | 73.6 |
| -O3 –qarch=pwr3
 –qtune=pwr3 -qstrict | 7.86 | 77.0 |
| -O3 –qarch=pwr3
 –qtune=pwr3 | 8.14 | 77.1 |
| -O3 –qarch=pwr3
 –qtune=pwr3 -qhot | 41.49 | 90.2 |
| -O4 | 44.87 | 89.5 |
| -O5 | 43.17 | 89.7 |
| -O4 -qnohot | 15.30 | 78.2 |
| -O5 -qnohot | 19.53 | 78.4 |

## Individual NAS Kernels MFlops

| Opt | MXM | FFT | CHOL | BT | GM | EM  | VP |
| --- | --- | --- | --- | --- | --- | --- | --- |
| None | 53.3 | 34.4 | 18.3 | 56.5 | 12.6 | 97.1 | 25.2 |
| -O2 | 235.2 | 162.2 | 135.9 | 209.8 | 17.5 | 153.2 | 37.0 |
| -O3 -qstrict | 763.7 | 145.7 | 191.3 | 217.9 | 17.5 | 150.2 | 36.4 |
| -O3 | 763.1 | 146.4 | 206.3 | 221.3 | 17.4 | 156.2 | 35.9 |
| -O3 -qhot | 425.6 | 164.8 | 210.0 | 286.4 | 17.6 | 472.0 | 57.0  |
| -O4 | 501.6 | 151.6 | 212.4 | 284.3 | 17.5 | 475.1 | 56.6 |
| -O5 | 421.3 | 160.8 | 211.2 | 302.4 | 17.5 | 470.6 | 57.0 |
| -O4 -qnohot | 761.5 | 148.5 | 206.8 | 220.1 | 17.5 | 153.2 | 37.4 |
| -O5 -qnohot | 748.8 | 152.5 | 209.9 | 219.9 | 17.5 | 156.8 | 37.2 |

## EMIT NAS Kernels with MASS library

| Optimization Level | MFlops w/o MASS | MFlops w/MASS |
| --- | --- | --- |
| unoptimized |  97.1 | 121.4 |
| -O2 | 153.2 | 224.4 |
| -O3 –qarch=pwr3
 –qtune=pwr3 -qstrict | 150.2 | 224.6 |
| -O3 –qarch=pwr3
 –qtune=pwr3 | 156.2 | 228.0 |
| -O3 –qarch=pwr3
 –qtune=pwr3 -qhot | 472.0 | 476.7 |
| -O4 | 475.1 | 478.7 |
| -O5 | 470.6 | 479.3 |
| -O4 -qnohot | 153.2 | 227.6 |
| -O5 -qnohot | 156.8 | 227.6 |

## SMG2000 ASCI Purple Benchmark

- Parallel semicoarsening multigrid solver.
- SPMD code written in ISO-C using MPI.
- Run with 8 MPI processes on 1 node.
- Timings are MFlops returned by poe+.

## SMG2000 Timings

| Optimization Level | Compile Time | MFlops |
| --- | --- | --- |
| unoptimized |  23.7 | 109.0 |
| -O2 |  61.6 | 283.5 |
| -O3 –qarch=pwr3 –qtune=pwr3 -qstrict |  75.4 | 281.9 |
| -O3 –qarch=pwr3 –qtune=pwr3 |  75.1 | 291.2 |
| -O3 –qarch=pwr3 –qtune=pwr3 –qhot | 173.1 | 291.2 |
| -O4 | 175.1 | 290.2 |
| -O5 | 167.8 | 291.5 |
| -O4 -qnohot | 123.6 | 299.4 |
| -O5 -qnohot | 106.9 | 298.2 |

## sPPM ASCI Purple Benchmark

- Solves a 3D gas dynamics problem on a uniform Cartesian mesh, using a simplified version of the PPM (Piecewise Parabolic Method) .
- Distributed hybrid MPI/OpenMP code written in the late 1990's in Fortran 77 with some C routines.
- Version tested was 1 MPI process and 16 OpenMP threads.
- Timings are MFlops returned by poe+.

## sPPM Timings

| Optimization Level | Compile Time | MFlops |
| --- | --- | --- |
| unoptimized |    9.7 | 1147 |
| -O2 |   69.4 | 3043 |
| -O3 –qarch=pwr3 –qtune=pwr3 -qstrict | 126.9 | 2973 |
| -O3 –qarch=pwr3 –qtune=pwr3 | 117.7 | 2727 |
| -O3 –qarch=pwr3 –qtune=pwr3 –qhot | 236.9 | 4578 |
| -O4 | 314.3 | 4521 |
| -O5 | 404.1 | 4737 |
| -O4 -qnohot | 212.7 | 2953 |
| -O5 -qnohot | 220.1 | 3183 |

## References

- See the web page _[http://hpcf.nersc.gov/computers/SP/options.html](http://hpcf.nersc.gov/computers/SP/options.html)_ for an expanded version of this presentation along with many references.

## Finis

- End of this presentation.