# Presentation: 486437

## Profiling Tools on the NERSC Crays and IBM/SP

- NERSC User Services

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

## Outline

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**Outline**

- Profiling Tools on NERSC platforms
- Cray PVP (killeen, seymour)
- Cray T3E (mcurie)
- IBM/SP (gseaborg)
- UNIX profiling/performance analysis tools
- References

## Why Profile?

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**Why Profile?**

- Characterise application :
- Is code cpu bound?
- Is code I/O bound?
- Is code memory bound?
- Analyse communication patterns - D.M. codes
- Focus optimisation effort ... and ultimately..
- Improve performance and resource utilisation

## Cray PVP/T3E - Application Characterization

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**Cray PVP/T3E - Application Characterization**

- Job accounting (ja)
- ja
- ./a.out
- ja -st -n a.out - see next slide for sample output
- Look out for :
- Maximum Memory Used > available memory
- Total I/O wait time (locked+unlocked) > 50% User CPU time
- Multitasking breakdown for parallel codes

## Job accounting : summary report

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**Job accounting : summary report**
- Elapsed Time : 8 Seconds 	User CPU Time : 35.5939 Seconds
- Multitasking/ Multistreaming Breakdown
- (Concurrent CPUs * Connect seconds = CPU seconds)
- 		1 * 0.0100 	=		0.0100
- 		2 * 0.0100	 = 		0.0200
- 		3 * 0.0600 	= 		0.1800
- 		4 * 8.8500 	= 	          35.4000
- 	(Avg.) 		(total) 				(total)
- 	3.99 	* 	8.9300 	= 		35.6100
- System CPU Time : 0.1226 Seconds
- I/O Wait Time (Locked) : 0.0000
- I/O Wait Time (Unlocked) : 0.0000
- CPU Time Memory Integral : 5.3854 Mword-seconds
- Data Transferred : 0.0001 MWords
- Maximum memory used : 0.4746 MWords

## HPM - Hardware Performance Monitor

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**HPM - Hardware Performance Monitor**

- Helps locate CPU related code bottlenecks
- reports use of vector registers, instruction buffers, memory ports
- hpm {options} ./a.out {prog_arguments}
- options = -g2 -> memory access information
- options = -g3 -> vector register information
- Look for :
- Ratio of **Floating Ops/CPU second** to **CPU mem**. **references per sec** should reflect the FpOps in the code

## Sample hpm output :  (hpm -g0 ./a.out)

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**Sample hpm output :  (hpm -g0 ./a.out)**
- Million inst/sec (MIPS) :       7.67    Instructions     :  274017290
- Avg. clock periods/inst :      26.06
- % CP holding issue      :      94.02    CP holding issue : 6714667737
- Inst.buffer fetches/sec :       0.04M   Inst.buf. fetches:        1420802
- Floating adds/sec       :      15.40M   F.P. adds        :      550002417
- Floating multiplies/sec :      24.36M  F.P. multiplies  :     870004996
- Floating reciprocal/sec :       0.28M   F.P. reciprocals :      10000042
- Cache hits/sec          :       0.00M     Cache hits       :          45893
- CPU mem. references/sec :      34.64M
- CPU references   :     1236978495
- Floating ops/CPU second:       40.5M

## Cray PVP : CPU Bound Codes: prof/profview 

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**Cray PVP : CPU Bound Codes: prof/profview **

- Instruments code to provide % cpu time in function calls
- f90 -lprof prog.f90
- ./a.out -> generates prof.data
- prof -st ./a.out > prof.report
- Chart (over) indicates relative distribution of CPU execution  time by function call
- prof -x a.out > pgm.prof
- profview pgm.prof

## Profview - Sample Output 

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**Profview - Sample Output**

## I/O and Memory Bound Codes : procstat/procview 

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**I/O and Memory Bound Codes : procstat/procview **

- procstat -m -i -R a.raw a.out
- procview a.raw
- I/O Analysis :
- Reports, Files -> All User Files (Long Report)
- Bytes Processed or I/O Wait Time
- Memory Analysis :
- Reports -> Processes -> Maximum Memory Used (Long Format)

## procview indicates which files consume most real time for I/O processing

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**I/O Bound Codes : procview**

- procview indicates which files consume most real time for I/O processing

## Memory Bound Codes : procview

- “High” (> 10% Elapsed Time) Time to complete Memory requests may indicate memory bound code
- Use Graphs option to produce plot of Memory use over elapsed time of application

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

## ATExpert - Autotasking Prediction

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**ATExpert - Autotasking Prediction**

- Analysis of source code to predict autotasking performance on dedicated Cray PVP
- f90 -eX -O3 -r4 -o {prog_name} prog.f90
- ./a.out
- atexpert -> shows predicted speed-up

## ATExpert Sample output

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**ATExpert Sample output**

- Indicates predicted speed-up of 4.3 on dedicated 8 processor PVP when source code is autotasked

## Also available on Cray PVP 

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

- Also available on Cray PVP

- Flowtrace/flowview
- times (using Operating System timers) subroutines and functions during program execution
- jumptrace/jumpview
- provides exact timing in function/subroutine by analysis of machine instructions in program
- perftrace/perfview
- times subroutines/functions based on statistics gathered from HPM tool

## Cray T3E - Apprentice

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**Cray T3E - Apprentice**

- Locate performance problems /inefficiencies
- MPI and shared memory performance, load balance and communication,  memory use
- Provides hardware performance information and tuning recommendations (Displays -> Observations)
- Compile/link
- f90 -o {prog} -eA {prog_name.f90} -lapp
- cc -o {prog} -happrentice {prog_name.c} -lapp
- Run code to generate app.rif

## Output from :

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

- apprentice app.rif

## Cray T3E - PAT

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**Cray T3E - PAT**

- Generates profile of CPU time in functions; load balance across PEs; h/w counter info.
- Compile and Link with PAT library
- f90 -o exe -lpat {source.f} pat.cld
- Run program as normal
- mpprun -n {procs} {exe} -> generate exe.pif
- pat executable exe.pif

## Profile based on relative CPU time in function calls

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

- Load Balance Histogram for routine “COLL”

## Cray T3E - ACTS/TAU 

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**Cray T3E - ACTS/TAU**

- Performance analysis of distributed/shared memory applications (C++ in particular)
- module load tau
- instrument programs with TAU macros
- add $(TAU_DEFS), $(TAULIBS) to compile/link
- run application; view tracefile with pprof, VAMPIR
- Reference
- _[http://acts.nersc.gov/tau](http://acts.nersc.gov/tau)_
- _[http://hpcf.nersc.gov/training/classes/Teleconf/1999july/Wu](http://hpcf.nersc.gov/training/classes/Teleconf/1999july/Wu)_

## Cray T3E - Vampir 

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**Cray T3E - Vampir**

- Analysis of message passing characteristics - generates display of MPI activity over instrumented time period (e.g. sender, receiver, message size, elapsed time)
- module load VAMPIR; module load vampirtrace
- Facility to instrument with VAMPIRtrace calls
- Generate trace file using TAU or VAMPIRtrace
- Reference :
- _[http://hpcf.nersc.gov/software/tools/vampir.html](http://hpcf.nersc.gov/software/tools/vampir.html)_

## IBM/SP - Xprofiler

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**IBM/SP - Xprofiler**

- Graphical interface for gprof profiles of parallel applications
- Compile **and** link code with “-g -pg”
- poe ./a.out -procs {n}
- generates gmon.out.{n} file for each process
- may introduce significant (upto factor of 2) overhead
- (In $TMPDIR)  xprofiler ./a.out gmon.out.*
- Report menu provides (gprof) text profile
- Source statement profiling shown

## NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER 

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

## Statement level profile available by clicking on relevant function graphical output - use Show Source Code option

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

## IBM/SP - Visualization Tool (VT)

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

- Message passing trace visualization
- Realtime system activity monitor (limited)
- MPI load balance overview :
- poe ./a.out -procs {n} -tlevel=3
- copy a.out.trc to $TMPDIR
- (In $TMPDIR) Invoke vt
- In trace visualization mode, “Play” a.out.trc
- see next slide for sample of Interprocessor Communication during program execution

## NATIONAL ENERGY RESEARCH SCIENTIFIC COMPUTING CENTER 

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

## IBM/SP : system_stats

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

- IBM Internal Tool
- module load sptools
- instrument code with system_stats() call
- Link with $(SPTOOLS), run code as normal
- Sample output

- Summary of the utilization of system resources:
- node      hostname     wall(s)    user(s)     sys(s)    size(KB)  pswitches
-   0        gs01015      16.80      13.18       0.04        2748        2138
-   1        gs01015      16.80      16.07       0.04        2744        1868
-   2        gs01003      16.80      16.62       0.04        2740        1870
-   3        gs01003      16.80      16.56       0.03        2732        1841

## IBM/SP - trace-mpi 

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

- IBM/SP - trace-mpi

- IBM Internal tool - Quantitative information  on MPI calls
- module load USG ; module load trace-mpi
- Fortran - add $(TRACE_MPIF) to build
- C - add $(TRACE_MPI) to build
- poe ./a.out -procs {n} - generates mpi.trace_file for each process (executable** must** call MPI_Finalize)
- summary mpi.trace_file.{n}  (see over)
- Useful check for load balance :
- grep “Total Communication” mpi.trace.file.*

## MPI message-passing summary for mpi.trace_file.3

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

**MPI message-passing summary for mpi.trace_file.3**
- MPI Function             #calls      Avg Bytes      Time (sec)
- -------------------------------------------------------------
- MPI_Allreduce:             9355           8.0           3.596
- MPI_Barrier:                       3           0.0            0.017
- MPI_Bcast:                        66           5.8           0.013
- MPI_Scatter:                      31        1008.0        0.088
- MPI_Comm_rank:               1             0.0           0.000
- MPI_Comm_size:                1           0.0           0.000
- MPI_Isend:                   43023        2003.7         0.893
- MPI_Recv:                   43023         2003.7         7.481
- MPI_Wait:                    43023        2003.7         3.739
- Total Communication Information:
- WALL = 15.8277, CPU = 15.53, MBYTES = 258.72
- The total amount of wall time                        = 26.229613

## Upcoming on the SP

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

- ACTS/TAU (C/C++)
- currently being ported to the IBM/SP
- VAMPIR
- has been ordered, awaiting delivery
- Performance Monitor Toolkit (HPM)
- should be available with Phase II system(requires AIX 4.3.4)
- Also, see Performance API project:
- _[http://icl.cs.utk.edu/projects/papi](http://icl.cs.utk.edu/projects/papi)_

## General/UNIX Profiling Tools

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

- Command line profilers and system analysis
- prof/gprof (enabled for MPI on IBM/SP)
- csh time command : time ./a.out
- vmstat -> look for high paging over extended time period (application may require more memory)
- Fortran/C function timers
- getrusage
- rtc, irtc
- etime, dtime, mclock
- MPI_Wtime

## Reference Material

**N****ATIONAL**** E****NERGY**** R****ESEARCH**** S****CIENTIFIC**** C****OMPUTING**** C****ENTER**** **

**	**

- NERSC web pages
- _[http://hpcf.nersc.gov/software/tools](http://hpcf.nersc.gov/software/tools)_
- Cray PVP/Cray T3E
- _[http://www.cray.com/swpubs](http://www.cray.com/swpubs)_
- Optimizing Code on Cray PVP Systems
- Cray T3E C, Fortran Optimization Guides
- IBM/SP
- _[LLNL Workshop on Performance Tools](http://www.llnl.gov/computing/tutorials/workshops/workshop/performance_tools)_