# Presentation: 153124

## ASCI Red Math Libraries

- What Libraries Exist
- Libcsmath
- Libwc
- Others

## What Math Libraries Exist

- Libcsmath (“Comp-Sci MATH”)
- Level 1, 2, and 3 BLAS
- 1D FFTs
- Partial Man pages in R2.8!
- LAPACK
- BLACS
- NX (An integer sum bug has been fixed in R2.8)
- MPI
- libwc (write-combine Cougar libraries)
- ScaLAPACK (Parallel Linear Alegra Package)
- PBLAS (Parallel BLAS)

## LIBCSMATH

- Level 1, 2, and 3 BLAS.  1D FFTs
- /usr/lib/libcsmath_r.a, /usr/lib/libcsmath_cop.a, /usr/lib/libcsmath.a
- _r: Tries to split the BLAS/FFTs for you using the compiler
- _cop: Tries to split the BLAS/FFTs for you using “cop”
- All versions reentrant except on some level-2,level-3 complex BLAS
- If you do your own parallelism, you will want to explicitly use libcsmath.a.
- The official C interface to the BLAS is included.
- Linking with -mp or -Mconcur automatically gives you _r.a.
- Dual processor versions are enabled with -proc 2 on the yod line
- $TFLOP_XDEV/tflops/lib
- See the release notes and upcoming man pages (R2.8)
- Works with OSF and Cougar
- http://www.cs.utk.edu/~ghenry/distrib
- Can find a linux version here (around 16000 licenses.)
- Other interesting kernels:
- copsync()  ; xdgemm() ; transposition routines

## Libcsmath R2.8 enhancements

- C = A*B case where the number of columns of C (n) is small.
- All DGEMM cases where K is small
- K=24 GEMM cases
- K>64 GEMM cases
- GEMM cases where the number of rows (M) is 2.
- More prefetching done on columns of B
- Enhancements to other level-3 kernels
- Faster handling of smaller BLAS

## LIBWC

- Write Combine Library:
- Using the “write combine” method of accessing memory as opposed to “write back”.  Write combine buffers a single cache line and then writes it directly to memory, instead of loading it first into cache or keeping it in cache awhile like write back.
- Write Combine library for Cougar
- Takes advantage of new Xeon core features
- Applicable to any memory-write bound kernel.
- Please contact us if you have a use for a tuned kernel of this nature.
- Three versions (like libcsmath), but only available on Cougar
- The compiler does not automatically bring in one or another unlike libcsmath
- Link with -lwc and you must use -wc on the yod line
- libwc versions of: memcpy, dcopy, dzero, memset, memmove, bcopy
- Designed for large (1 Mbyte) memory writes.
- Other interesting kernels: (these can be used by anything!)
- flush_caches() (flushes the caches on one or both processors)
- use_write_combine() (returns 1 if it is safe to use write combine)
- touch1( array, size_of_array_in_bytes) (C and Fortran versions)

## LAPACK, PBLAS, ScaLAPACK

- /usr/lib/scalapack or $TFLOP_XDEV/tflops/lib/scalapack
- liblapack.a, libtmglib.a, libpblas.a, libscalapack.a, libtools.a, libredist.a
- ScaLAPACK and PBLAS depend on BLACS or BLACS_MPI
- Sample link lines:
- L/usr/lib/scalapack -ltmglib -llapack
- L/usr/lib/scalapack -lscalapack -lpblas -ltools -lredist
- L/usr/lib/scalapack -lblacsF77init_MPI -lblacs_MPI -lblacsF77init_MPI -lmpi
- L/usr/lib/scalapack -lblacsCinit_MPI -lblacs_MPI -lblacsCinit_MPI -lmpi
- L/usr/lib/scalapack -lblacs -lnx
- Recent BLACS Integer sum bug fix found in release R2.8!

## A new Optimization Tool

- Not yet available: only in Alpha on Janus right now.
- Optimizes your (F77) subroutine by trying different optimization strategies and returning to you the assembly code corresponding to the optimal one.
- You must provide a “greg_timer()” and “greg_initialize()” routines and link against the library.
- The routine greg_timer() calls “opcode_routine()” instead of the target routine to be optimized.
- The application runs on Janus for anywhere from a minute to a day or more depending on input options.

## Example

-        subroutine target_routine_to_optimize(A, B, C, M, N)
-         double precision A(*), B(*), C(*), SUM1
-         integer M, N, I, J
-         sum1 = 0.d0
-         do I = 1, M
-                 do J = 1, N
-                         sum1 = sum1 + A((I-1)*M+J) + B((I-1)*M+J)
-                 enddo
-                 C(I) = sum1
-         enddo
-         return
-         end
- /* New auxillary routines */
- #define ARRAY_SIZE 1024
- int N=ARRAY_SIZE, M=ARRAY_SIZE;
- double A[ARRAY_SIZE*ARRAY_SIZE];
- double B[ARRAY_SIZE*ARRAY_SIZE];
- double C[ARRAY_SIZE];
- greg_initialize()
- {
-    int I;
-    for ( I = 0 ; I < ARRAY_SIZE*ARRAY_SIZE ; I++ ) { A[I]=0.0; B[I]=0.0; }
-    for ( I= 0 ; I< ARRAY_SIZE ; I++ ) C[I] = (double) I;
-    flush_cache();
- }
- greg_timer()
- {
-    extern void (*opcode_routine)();
-    (*opcode_routine)(A, B, C, &N, &M);
- }