# Presentation: 801063

## Introduction to J90 Optimization

**David Turner**
**NERSC User Services Group**
**3/17/1999**

**Notes:**
Outline:
Why are people looking at PC clusters?
Why should NERSC be interested in PC clusters?
Who is doing PC clusters?
  Mention nersc/lbl/millennium/beowulf/sandia

Hardware/software issues
  hardware
  	processor
 	network
 software
 	operating system
	cluster software
	tools
Extreme Linux
Support issues (mention fnal)
What is NERSC doing?

## Philosophical Ramblings

**Design for optimization?**
**Where to start?**
**When to stop?**

## J90 Potential 

- STREAM benchmark results
- Sustainable memory bandwidth
**(http://www.cs.virginia.edu/stream)**
- John McCalpin, SGI
**                  ****bytes/iter   FLOPS/iter**
**COPY  **
**a(i)=b(i)           16           0**
**TRIAD **
**a(i)=b(i)+q*c(i)    24           2**

## STREAM Results

_**Machine     ncpus     COPY    TRIAD   MFLOPS**_
**Cray_C90       16 105497.0 103812.0   8651.0**
**Cray_C90        8  55071.9  63229.6   5269.1**
**Cray_C90        1   6965.4   9500.7    791.7**
**Cray_J932      16  16298.2  14995.9   1249.7**
**Cray_J932       8   9995.2   8941.3    745.1**
**Cray_J932       1   1433.6   1270.0    105.8**
**Cray_T3E-900   16   7497.0   8828.0    735.7**
**Cray_T3E-900    8   3747.0   4471.0    372.6**
**Cray_T3E-900    1    484.0    568.0     47.3**
**SGI_Origin_2K  16   5560.0   5240.0    436.7**
**SGI_Origin_2K   8   2570.0   2740.0    228.3**
**SGI_Origin_2K   1    332.0    358.0     29.8**
**Sun_UE_10000   16   2371.0   2905.0    242.1**
**Sun_UE_10000    8   1271.0   1546.0    128.8**
**Sun_UE_10000    1    164.0    202.0     16.8**

## STREAM Results (cont.)

_**Machine                    COPY    TRIAD   MFLOPS**_
**Cray_C90                 6965.4   9500.7    791.7**
**Cray_J932                1433.6   1270.0    105.8**
**Compaq_AlphaServer_DS20  1077.0   1323.0    110.2**
**IBM_RS6000-397            778.8    882.4     73.5**
**Cray_T3E-900              484.0    568.0     47.3**
**SGI_Origin_2K             332.0    358.0     29.8**
**Generic_440BX_400         304.0    315.4     26.3**
**Sun_Ultra2-2200           228.5    189.9     25.9**
**Sun_UE_10000              164.0    202.0     16.8**
**Apple_Mac_G3_266          137.1    137.1     11.4**

## Tools

**ja**
**./name**
**ja -cst -n name**
**hpm **
**prof**
**flowview**
**atexpert**

## Program “SLOW”

**PROGRAM SLOW**
**  ****IMPLICIT NONE**
**  ****INTEGER, PARAMETER :: DIMSIZE=8000000**
**  ****REAL, DIMENSION(DIMSIZE) :: X, Y, Z**
**  ****INTEGER:: I, J**
**  ****X = RANF()**
**  ****Y = RANF()**
**  ****DO J = 1, 10**
**    ****DO I = 1, DIMSIZE**
**      ****Z(I)=LOG(SIN(X(I))**2+COS(Y(I))**4)**
**    ****END DO**
**    ****PRINT *, Z(DIMSIZE-1)**
**  ****ENDDO**
**  ****STOP**
**END PROGRAM SLOW**

## No Optimization

_**f90 -O0 -r6 -O,msgs,negmsgs -o slow slow.f90**_
**x = RANF()**
**cf90-6204 f90:VECTOR SLOW,File = slow.f90, Line=8 **
**  ****A loop starting at line 8 was vectorized.**
**y = RANF()**
**cf90-6204 f90:VECTOR SLOW,File = slow.f90, Line=9 **
**  ****A loop starting at line 9 was vectorized.**

## Moderate Optimization

_**f90 -O1 -r6 -O,msgs,negmsgs -o slow slow.f90**_
**do j = 1, 10**
**cf90-6286 f90:VECTOR SLOW,File = slow.f90,Line=10 **
**A loop starting at line 10 was not vectorized because it contains input/output operations at line 14.**
**DO i = 1, DIMSIZE**
**cf90-6204 f90:VECTOR SLOW,File = slow.f90,Line=11 **
**A loop starting at line 11 was vectorized.**
**z(i) = LOG(SIN(x(i))**2 + COS(y(i))**4)**
**cf90-6001 f90:SCALAR SLOW,File=slow.f90,Line=12 **
**  ****An exponentiation was replaced by optimization.  This may cause numerical differences.**

## High Optimization

_**f90 -O3 -r6 -O,msgs,negmsgs -o slow slow.f90**_
**cf90-6502 f90:TASKING SLOW,File=slow.f90,Line=10 **
**A loop starting at line 10 was not tasked because it contains input/output operations at line 14.**
**cf90-6403 f90:TASKING SLOW,File=slow.f90,Line=11 **
**  ****A loop starting at line 11 was tasked.**

## Optimization Results

_**Opt NCPUS   Elapsed       User        Sys**_**    **
**0          768.7530   583.6793     7.1886  **
**1           89.0162    82.1009     1.1936  **
**2          104.7003    81.5687     1.0003**
**3       1  107.0177    81.6185     1.2994**
**3       2   44.6562    81.7050     1.4069 **
**3       3   41.3401    81.5320     1.3099**
**3       4   24.8146    81.8099     1.2968**

## 2 CPU Speedup

**(Concurrent CPUs * Connect seconds = CPU seconds)**
** ****---------------   ---------------   -----------**
**               ****1 *          5.4300 =      5.4300**
**               ****2 *         38.1300 =     76.2600**
**(Concurrent CPUs * Connect seconds = CPU seconds)**
**      ****(Avg.)           (total)         (total)**
** ****---------------      --------------   -----------**
**            ****1.88 *         43.5600 =     81.6900**

## 3 CPU Speedup

**(Concurrent CPUs * Connect seconds = CPU seconds)**
** ****---------------   ---------------   -----------**
**               ****1 *          9.2200 =      9.2200**
**               ****2 *         13.5500 =     27.1000**
**               ****3 *         15.0700 =     45.2100**
**(Concurrent CPUs * Connect seconds = CPU seconds)**
**      ****(Avg.)           (total)         (total)**
** ****---------------      --------------   -----------**
**            ****2.15 *         37.8400 =     81.5300**

## 4 CPU Speedup

**(Concurrent CPUs * Connect seconds = CPU seconds)**
** ****---------------   ---------------   -----------**
**               ****1 *          2.0400 =      2.0400**
**               ****2 *          1.7700 =      3.5400**
**               ****3 *          5.3200 =     15.9600**
**               ****4 *         15.0700 =     60.2800**
**(Concurrent CPUs * Connect seconds = CPU seconds)**
**      ****(Avg.)           (total)         (total)**
** ****---------------      --------------   ----------**
**            ****3.38 *         24.2000 =     81.8200**

## Using flowtrace/flowview

**f90 -O1 -ef -o slow slow.f90**
**./slow**
**flowview -Luch > slow.flow**
**Routine      Tot Time  Percentage Accum%**
**------------ --------  ---------- -------**
**SUB2         5.66E+01       69.02   69.02  **
**SUB1         2.43E+01       29.63   98.65  **
**SLOW         1.11E+00        1.35  100.00**

## Using prof

**f90 -O1 -l prof -o slow slow.f90**
**./slow**
**prof -x ./slow > slow.prof**
**profview slow.prof**

## Scalar Optimization

**Subroutine or function inlining**
**Fast (32-bit) integers **
**-Oallfastint**
** ****-Ofastint**
**Use INTERFACE specifications if passing array sections**

## Vector Computing

## Inhibitors to Vectorization

**Function or subroutine references**
**Inline**
**Push loop**
**Split loop**
**Backwards data dependencies**
**Reorder loop, use temporary vector**** **
**I/O statements**
**Character or bit manipulations**
**Branches into loop or backward out of loop**

## Nonvectorizable Code

**DO I = 1, N**
**  ****CALL CALC(X(I), Y(I), Z(I))**
**ENDDO**
**...**
**SUBROUTINE CALC(X, Y, Z)**
**Z = ALOG(SQRT((SIN(X) * COS(Y)) ** X))**
**RETURN**
**END**

## Inlining

**DO I = 1, N**
**  ****Z(I) = ALOG(SQRT((SIN(X(I))*COS(Y(I)))**X(I)))**
**ENDDO**

## Pushing

**CALL CALC(X(I), Y(I), Z(I), N)**
**...**
**SUBROUTINE CALC(X, Y, Z, N)**
**DIMENSION X(N), Y(N), Z(N)**
**DO I = 1, N**
**  ****Z(I) = ALOG(SQRT((SIN(X(I))*COS(Y(I)))**X(I)))**
**ENDDO**
**RETURN**
**END**

## Splitting

**DO I = 1, N**
**  ****A(I) = ABS(CALC(C(I)))**
**  ****B(I) = A(I) ** T * SQRT(C(I))**
**  ****A(I) = SIN(ALOG(C(I)))**
**ENDDO**

## Splitting (cont.)

**EXTERNAL CALC**
**DO I = 1, N**
**  ****A(I) = ABS(CALC(C(I)))**
**ENDDO**
**DO I = 1, N**
**  ****B(I) = A(I) ** T * SQRT(C(I))**
**  ****A(I) = SIN(ALOG(C(I)))**
**ENDDO**

## Scalar Recurrence

**DIMENSION A(1000), C(1000)**
**DO J = 1, M**
**  ****S = BB**
**  ****DO I = 1, N**
**    ****S = S * C(I)**
**    ****A(I) = A(I) + S**
**  ****ENDDO**
**ENDDO**
**<cf90-8135,Scalar,Line=7> Loop starting at line 7 was unrolled 16 times**.

## Scalar Recurrence (cont.)

**DIMENSION A(1000), C(1000), S(1000)**
**DO I = 1, M**
**  ****S(I) = BB**
**ENDDO**
**DO I = 1, N**
**  ****DO J = 1, M**
**    ****S(J) = S(J) * C(I)**
**    ****A(I) = A(I) + S(J)**
**  ****ENDDO**
**ENDDO**
**Loop starting at line 5 was unrolled 2 times.**
**A loop starting at line 5 was vectorized.**
**A loop starting at line 9 was vectorized.**

## Compiler Directives

**CDIR$ *****directive***** **
**!DIR$ *****directive***** **
**VECTOR, NOVECTOR**
**Turn vectorization on or off until end of program unit.**
**IVDEP**
**Ignore vector dependencies in next loop.**

## Parallel Computing

**Multitasking, microtasking, autotasking, parallel processing, multiprocessing, etc.**
**One possible goal:  parallelize outer loop(s), vectorize inner loop(s)**

## Data “Scoping” 

**DIMENSION A(N)**
**SUM = 0.0**
**DO I = 1, N**
**  ****TEMP = DEEP_THOUGHT(A,I)**
**  ****SUM = SUM + TEMP * A(I)**
**ENDDO**
**A, N**     **Shared, read-only everywhere**
**I, TEMP**  **Private, read-write everywhere**
**SUM **     **Shared, read-write everywhere**

## Compiler Directives

**DIMENSION A(N)**
**SUM = 0.0**
**!MIC$ DOALL SHARED(A,N),PRIVATE(I,TEMP)**
**DO I = 1, N**
**  ****TEMP = DEEP_THOUGHT(A,I)**
**!MIC$ GUARD**
**  ****SUM = SUM + TEMP * A(I)**
**!MIC$ ENDGUARD**
**ENDDO**

## Threshold Test

**DIMENSION A(N)**
**SUM = 0.0**
**!MIC$ DOALL VECTOR**
**!MIC$ IF(N.GT.1000)**
**!MIC$ SHARED(A,N),PRIVATE(I,TEMP)**
**DO I = 1, N**
**  ****TEMP = DEEP_THOUGHT(A,I)**
**!MIC$ GUARD**
**  ****SUM = SUM + TEMP * A(I)**
**!MIC$ ENDGUARD**
**ENDDO**

## atexpert

**f90 -eX -O3 -r6 -o slow slow.f90**
**setenv NCPUS 1**
**./slow**
**atexpert**