# Workload Characterization Project Checklist

### Meta Information

Name of code: V2D

Data collector: Richard Gerber

Interview history (include user names, affiliation, username, contact
info):

  ---------- ----------------- ---------------- -------------- ----------------------------
  **Name**   **Affiliation**   **Project(s)**   **Username**   **Contact info (phone,
                                                               email)**

  Doug       SUNY-Stonybrook   Terascale        swesty         Douglas.Swesty@sunysb.edu,
  Swesty                       Supernova                       631-632-8055
                               Initiative                      
                               SciDAC Project                  

                                                                    
  ---------- ----------------- ---------------- -------------- ----------------------------

  --------------------------------------------------------------------------------
  **Project**   **PI**           **Project        **% Project**    **Notes**
                                 Allocation                        
                                 (hours)**                         
  ------------- ---------------- ---------------- ---------------- ---------------
  TSI - MP124   Anthony          300000           40                    
                Mezzacappa                                         

                                                                        
  --------------------------------------------------------------------------------

Total estimated code hours (allocated):      

Office/Program:

Office 2:

Science Category: (     )

Information sources (put links to docs here):

###      

### Code information

What is the purpose of this code: To solve the integro-partial

differential equations of either Newtonian or relativistic
radiation-hydrodynamics as simulations of the convective epoch of
post-bounce supernovae.

High level code features (check all that apply)

+-----------------------------------+-----------------------------------+
| Algorithm                         | Direct solver                     |
|                                   |                                   |
| Structured mesh                   | Dense linear algebra              |
|                                   |                                   |
| Unstructured mesh                 | Sparse linear algebra             |
|                                   |                                   |
| Adaptive mesh refinement          | Heavy use of integer/logical      |
|                                   | arithmetic                        |
| No mesh                           |                                   |
|                                   | Monte Carlo                       |
| Iterative solver                  |                                   |
|                                   | FFT                               |
| Particle methods                  |                                   |
|                                   | Multigrid                         |
+-----------------------------------+-----------------------------------+

Algorithm notes:

     

Planned additions/changes to code over next several years:

AMR, more microphysics (may not affect code much)

Relationship (if any) to other codes:

interralated suite of 1D, 2D and 3D codes

Other notes on the code including version number:

2.53

Name of executable (how can we identify this job in LL logs?): v2d

### Code Performance

Performance data

  ------------- ------------- --------- ------------------ ----------- ------------ ------------- ------------
  **Machine**   **Problem\    **\#      **Mflop/s/proc**   **Gflop/s   **Mem/proc   **Wall hours  **Method**
                Type/Size**   procs**                      total**     (MB)**       (typical)**   

  Seaborg                     256       177                45.39       74           48            

  Seaborg                     512       177                90.72       69           48            

  Seaborg                     1024      136                139.34      73           48            

                                                                                                  

                                                                                                  

                                                                                                  

                                                                                                  

                                                                                                  

                                                                                                  
  ------------- ------------- --------- ------------------ ----------- ------------ ------------- ------------

Measurement includes:

Other measurement info:      

What is the code's "sweet spot", namely the number of processors that
maximizes science productivity? Why is this the sweet spot?

1024, optimal throughput while still maintaining high efficiency,
relevant problem size

If not all runs are at the sweet spot, what is the primary reason for
changing the number of processors? effectively all runs are at \"sweet
spot\"

Solving a different problem

Solving a bigger problem

Faster turnaround because of faster processing

Faster turnaround because of smaller queue wait times

Ability to get extra time (i.e. reimbursement program)

Other (Describe:      )

### Limits to performance

What factors limit code performance/scaling (beyond raw processor
performance)

Check all that apply.

+--------------------+-------+----------------------------------------+
| **Factor**         | **How | **Explanation**                        |
|                    | kn    |                                        |
|                    | own** |                                        |
+--------------------+-------+----------------------------------------+
| **Application      |       |                                        |
| based**            |       |                                        |
+--------------------+-------+----------------------------------------+
| Algorithm scaling  |       | inner product bottleneck               |
+--------------------+-------+----------------------------------------+
| Load imbalance     |       |                                        |
+--------------------+-------+----------------------------------------+
| Small problem size |       |                                        |
|                    |       |                                        |
| (Larger problem    |       |                                        |
| not needed for     |       |                                        |
| Science goals)     |       |                                        |
+--------------------+-------+----------------------------------------+
| Small problem      |       |                                        |
| size\              |       |                                        |
| (Limited by        |       |                                        |
| allocation from    |       |                                        |
| doing larger       |       |                                        |
| problems)          |       |                                        |
+--------------------+-------+----------------------------------------+
| Other              |       |                                        |
+--------------------+-------+----------------------------------------+
| **Machine based**  |       |                                        |
+--------------------+-------+----------------------------------------+
| Point-to-point     |       |                                        |
| latency            |       |                                        |
+--------------------+-------+----------------------------------------+
| Point-to-point     |       |                                        |
| bandwidth          |       |                                        |
+--------------------+-------+----------------------------------------+
| All-to-all         |       |                                        |
| performance\       |       |                                        |
| (large messages)   |       |                                        |
+--------------------+-------+----------------------------------------+
| Allreduce          |       | inner product bottleneck, ie. latency  |
| performance\       |       |                                        |
| (small messages)   |       |                                        |
+--------------------+-------+----------------------------------------+
| Broadcast          |       |                                        |
| performance        |       |                                        |
+--------------------+-------+----------------------------------------+
| Barrier            |       |                                        |
| performance        |       |                                        |
+--------------------+-------+----------------------------------------+
| Available memory   |       |                                        |
+--------------------+-------+----------------------------------------+
| MPI-IO I/O         |       | via parallel hdf5 (at greater than     |
|                    |       | 1024)                                  |
+--------------------+-------+----------------------------------------+
| Posix I/O (one     |       |                                        |
| file for several   |       |                                        |
| processes)         |       |                                        |
+--------------------+-------+----------------------------------------+
| Posix I/O          |       |                                        |
| (multiple files)   |       |                                        |
+--------------------+-------+----------------------------------------+
| External Network   |       |                                        |
+--------------------+-------+----------------------------------------+
| Pre/post           |       |                                        |
| processing         |       |                                        |
+--------------------+-------+----------------------------------------+
| Available disk     |       | would like more scratch space for long |
| space              |       | runs                                   |
+--------------------+-------+----------------------------------------+
| Bug                |       |                                        |
+--------------------+-------+----------------------------------------+
| Other              |       |                                        |
+--------------------+-------+----------------------------------------+

What machine features not currently available on the SP would the code
be able to take good advantage of if they were available

  ---------------- ---------------------- -------------------------------
  **Feature**      **State**              **Notes**

  Vector procesors                             

  Global shared                                
  memory\                                 
  (OpenMP or                              
  threads)                                

  Fast shmem and                          via turboMPI; for latency
  barrier                                 reduction

  Other                                   high memory bandwidth,
                                          low-latency inner products
  ---------------- ---------------------- -------------------------------

### Communication requirements

\[This information will be collected separately, through instrumentation
of code\]

### 

### I/O requirements

+------------------+----+--------+-------+----+-----------------+---+
| **type**         | *  | **I/O  | **M   | *  | **type of       | * |
|                  | *t | lib    | ode** | *t | data**          | * |
| **(in/out)**     | ot | rary** |       | yp |                 | n |
|                  | al |        |       | ic |                 | o |
|                  | \# |        |       | al |                 | t |
|                  | of |        |       | si |                 | e |
|                  | f  |        |       | ze |                 | s |
|                  | il |        |       | in |                 | * |
|                  | es |        |       | GB |                 | * |
|                  | in |        |       | of |                 |   |
|                  | t  |        |       | ea |                 |   |
|                  | yp |        |       | ch |                 |   |
|                  | ic |        |       | fi |                 |   |
|                  | al |        |       | le |                 |   |
|                  | r  |        |       | ** |                 |   |
|                  | un |        |       |    |                 |   |
|                  | ** |        |       |    |                 |   |
+------------------+----+--------+-------+----+-----------------+---+
|                  | 1  |        |       | 0  |                 |   |
|                  | 00 |        |       | .1 |                 |   |
|                  | 00 |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
+------------------+----+--------+-------+----+-----------------+---+
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
+------------------+----+--------+-------+----+-----------------+---+
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
+------------------+----+--------+-------+----+-----------------+---+
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
+------------------+----+--------+-------+----+-----------------+---+
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
|                  |    |        |       |    |                 |   |
+------------------+----+--------+-------+----+-----------------+---+

Definitions for mode column:

> collector: one process collects data and writes a file or one process
> reads data and broadcasts
>
> multi-collector: as collector, but for several application subgroups
>
> embarassingly parallel: every process writes or reads its own file
>
> cooperative: several processes write/read cooperatively to/from a
> single file

As you scale problem size, where do you expect I/O bottlenecks to appear
(if anywhere): 1024

Code is self-checkpointing

Code requires global filesystem

### Memory requirements

How much memory is used per processor (not per node) for typical
production runs

  ---------------------------------- ------------------------------------
  **number of processors**           **memory/processor(GB)**

  1024                               75

                                          
  ---------------------------------- ------------------------------------

### Code base

  -------------------------------- -------------- -------------- -----------
  **Feature**                      **required**   **optional**   **notes**

  C                                                                   

  Fortran 90                                                          

  C++                                                                 

  MPI                                                                 

  LAPI                                                                

  OpenMP                                                              

  Other (Describe:      )                                             
  -------------------------------- -------------- -------------- -----------

Year current code base originally started:      

Number of lines of code (excluding 3^rd^ party packages):      

Number of man-years spent developing code:      

Who is responsible for maintaining/improving code: Doug Swesty/Eric Myra

Who uses the code (outside of the projects listed above):      

External library/package/software requirements (everything beyond a
compiler and MPI-1)

(Don't forget MPI-I/O, BLAS)

  --------------------- --------------- ------------- ------------- ---------------
  **Library/package     **Required?**   **Used if     **Pre/post    **Notes**
  name/software**                       available**   processing\   
                                                      only**        

  HDF5                                                              v1.4
                                                                    compatibilty
                                                                    needed

  MASS                                                                   

                                                                         

                                                                         
  --------------------- --------------- ------------- ------------- ---------------

Pre/post processing capabilities of machine

  -------------------------------------------- -------------- ------------- -----------
  **Feature**                                  **required**   **desired**   **notes**

  Serial pre/post-processing on compute                                          
  machine                                                                   

  Parallel pre/post-processing on compute                                        
  machine                                                                   

  Ability to run Perl/Shell on compute nodes                                     
  -------------------------------------------- -------------- ------------- -----------

What types of systems has code been recently run on?

IBM SP

Linux Cluster

SGI Origin/Altix

Cray X1 or other vector-based machine

Other (Describe: Will be running on X1 (cf John Shalf))

Other requirements:      

### Information on filling out the form. 

How to use this checklist

-   One checklist per code

-   Fill in as much information as possible from ERCAP requests before
    contacting user

-   Include links to original source material. Filename of this doc is
    CODE.doc; source material in folder CODE.files

"% Project" means the fraction of project allocation used by this code.
ERCAP is unreliable here, so you may need to ask the PI. Sometimes a
particular code is run by a particular user, so you may be able to look
just at that user's usage.

Revision History:

  --------- ------------ ----------- --------------------------------------
  Version   Date         By          Comments

  1.0       01/02/2004   Bill Saphir Initial version

  1.1       01/01/2005   Bill Saphir comments from Francesca

  1.2       01/06/2004   Bill Saphir meeting comments

  1.3       01/16/2004   All         fixes to form

  1.4       1/28/2004    Bill        fixed second science category drop
                                     down

  1.5       2/3/2004     Bill        changes from 2/2 meeting
  --------- ------------ ----------- --------------------------------------