## The Chemistry Performance Database and Computational Quality of Service (CQoS) 

### Introduction

The goal of a chemistry database is to provide chemists more insight
into the performance of computations provided by GAMESS, MPQC and
NWChem. These packages usually provide more than one method for a
computation to be used under different circumstances. Currently the
decision to use one method instead of another one is determined by
chemists -- based upon their experience. A performance database that
collects and analyzes performance data for each quantum chemistry
computation may provide useful information for chemists to select a good
method to provide the expected performance.

CQoS research can directly benefit from this performance database. A
chemistry component usually wraps a computation provided in the original
package; if we cannot get insight into the performance of the original
codes, it's not likely we can predict the performance of the component
on top of the code. By providing more understanding of a computation
wrapped within a component, it may also help in designing tuning rules.

### Outline of the Project

![](media/image1.png){width="5.9527777777777775in"
height="4.8381944444444445in"}

To develop a database that can also support CQoS research, we refer to
the process of developing an automatic tuning system (adaptive computing
system). The developing process can be roughly divided into four stages
as shown in Figure 1. In the first stage, methods capable of providing
good performance must be collected or developed. In the second stage,
performance evaluations of each computation are conducted to acquire
performance data. The third stage is to analyze the collected
performance data and explore the parameters that have significant impact
to the overall performance. The last stage is to develop tuning rules.
The whole process is an iterative process in that for the developed
tuning rules, we will need to go through performance evaluation, data
analysis and may have to develop new tuning rules again.

Mapping the above four stages onto developing a chemistry database for
CQoS, we outline the required tasks in the following. In the first
stage, the focus is not on collecting/developing capable
implementations, but on developing mechanisms to interoperate
computations from different packages so one package can seamlessly use
computations provided by another package. The overhead of incorporating
CCA framework into the original package, and using computations from
different packages should be minimized. These are the goals of the
current CCA chemistry project.

The challenges in the second stage are: 1. Conduct performance
evaluations in proper granularity, 2. Collect and organize performance
data in a database so chemists can access this data easily. GAMESS (and
maybe the other two chemistry packages too) currently provides
coarse-grain level performance data, e.g., the run time of a whole
computation. For CQoS, more detailed performance data such as cache
utilization, communication latency, time spent in I/O etc., may be
needed. Performance tools such as TAU have the capability to instrument
source code and acquire fine-grain performance data; some manual work is
required for integrating TAU performance functions with GAMESS. It would
be better if acquiring/maintaining performance data can be handled
mostly by a system instead of asking every chemist to
produce/handle/maintain their performance data manually.

The third stage involves analysis of performance data collected in the
second stage. The issues include exploring: 1.parameters that have
significant impact to the performance of a computation, 2.the
relationships between these parameters, and 3.the relationships between
application characteristics and system characteristics. Performance
tools such as PerfExplorer provides help to explore the relationships
between the above parameters. Hardware information such as communication
or IO bandwidth are important to chemistry computations, thus data
generated by benchmarking suites may need to be incorporated into the
performance database since TAU only extracts basic system
characteristics such as cache size and CPU type during the performance
evaluation process.

The last stage is to develop tuning rules. Performance modeling that
predicts the runtime of a computation on a certain platform is a common
approach. Quantum chemistry computations usually involve a large number
of parameters and performance modeling for quantum chemistry
computations is regarded as extremely difficult (quotes from very
experienced computational chemists: "The holy grail problem in QC",
"Next to impossible"). A more practical approach may be to develop
heuristic tuning rules base on statistical data analysis, and this may
be the method we will try first. Another challenge in this stage is to
incorporate tuning rules into the original codes. It is very likely that
the original codes have to be modified to incorporate tuning mechanisms,
and we want such changes to be as few as possible.

The development of GAMESS involves many chemists over more than two
decades, and the large number of computations provided by GAMESS implies
that a chemist usually knows only some of them. To build a practical
performance database for chemistry computations we will need
participation of chemists. Our goal is to develop a system to help
chemists deal with procedures in each stage, reduce their efforts to
facilitate research progress, and hopefully help CQoS research for
large-scale scientific packages in the long run.

### Approaches and Current Progress

The tasks in Figure 1 would be easier to develop if we consider only one
computation, and one chemist in the whole development process. For
GAMESS, we need those tasks for many computations and many chemists, so
we prefer a systematic design. The simplest approach is to have every
chemist sends their GAMESS input files to a poor computer guy (a.k.a.
me), and let him run these computations, collect performance data,
upload performance data to the database, conduct performance analysis
and send emails of results back to chemists. It sounds scary; a lot of
jobs have to be handled manually and can be error-prone. We prefer to
make the whole process as automatic as possible.

Figure 2 shows the flow of data and the dependence between tasks, which
is what we used in developing the automation procedures. Most tasks in
Figure 2 used data provided by their previous tasks. For example, we
must have source code instrumented properly before we can conduct
performance evaluations to collect data, and performance analysis cannot
be processed without the performance database being properly built. Here
we describe our approaches and current progress for underdevelopment
tasks.

1.  Source code instrumentation: A more flexible program to instrument
    GAMESS is required. For example, to get the total time used in I/O,
    or time spent in communication 'within' certain parts of a
    computation (not the whole computation). Current automatic
    instrumentation programs cannot provide these capabilities, and we
    have developed supplemental programs for this purpose.

2.  Data collecting process: One major challenge here, from data
    collecting to performance analysis, is to make the whole process as
    trivial as possible to chemists. Our approach is that, chemists need
    only to provide their input files and indicating certain system
    settings, and then they can wait for the results of performance
    analysis. We have a front-end program (a web page
    <http://www.scl.ameslab.gov/Projects/borges/QCUpload.php>) that
    allows chemists to upload their input files and select the type of
    package to run the computation. The details of how to extract
    metadata, conduct performance evaluation and handle the database is
    a block box (handled by back-end programs) to chemists. The back end
    programs retrieve the input files form the front end, process the
    computation, extract the metadata from input files (and maybe GAMESS
    output log files), and upload performance data and metadata.

> ![](media/image2.png){width="5.997916666666667in"
> height="4.565277777777778in"}

3.  Performance database: MySQL is used as the underlying database. On
    top of MySQL we use PerfDMF to provide data uploading and
    controlling interface without using SQL function calls directly.
    PerfDMF is required if we want to use PerfExplorer for performance
    analysis. There are certain restrictions in the database schema
    provided by PerfDMF, and we need to arrange our data according to
    these restrictions in order to conduct the desired performance
    analysis provided by PerfExplorer.

4.  Application characteristics/System characteristics: Application
    characteristics are actually metadata extracted from GAMESS input
    files (or output log files from computations). Basic system
    characteristics such as CPU type and cache size are available when
    conducting performance evaluation using TAU. Some more complex
    system characteristics such as communication bandwidth and memory
    bandwidth can only be acquired by using hardware benchmarking
    suites. For example, NetPIPE can be used to extract communication
    bandwidth. Again, these data must be arranged properly in the
    PerfDMF for PerfExplorer to provide the desired analysis.

5.  Performance analysis: There are quite a lot of things to do to
    conduct performance analysis and get the results we need for
    developing tuning rules. We will need more capabilities provided by
    tools available today. Details are in the following section.

6.  Tuning rules development: Nothing at the moment. We need useful data
    in the database and the analysis procedures constructed first in
    order to develop tuning rules.

### Performance Analysis Issues

Figure 3 is a SCF computation on two different platforms. The simple
case study can give a glimpse of what kind of analyses are required, and
how difficult it may be to develop tuning rules.

![](media/image3.png){width="5.9875in" height="4.092361111111111in"}

On both platforms, our experiment varies combination of number of nodes,
and number of processors per node used in the computation. The left
panel shows the results on Seaborg, and the right panel shows the
results on Bassi.

On Seaborg, the cost of parallel write (PWRT) dramatically increases
when we use 16 processors per node; possibly caused by I/O channel
contention. On Bassi the ratio of PWRT is trivial. On the other hand,
the ratio of the latency of global sum (GSUM) in the overall cost
increases when we use more processors per node, but this latency is a
small portion when running the computation on Seaborg even when using 16
processors per node. In fact, the latency of GSUM on Bassi using 8
processors per node is more than that of using 16 processors per node on
Seaborg, even though Bassi is a newer machine.

Based on the two performance figures, consider the information we may
need to develop tuning rules:

1.  Under what circumstance does PWRT affect the overall performance?
    What is the relationship between the bandwidth of I/O channels and
    the amount of data being written per processor (or the overall data
    being written by all processors)? How about developing an equation
    to describe this relationship?

2.  The same set of questions for GSUM, substituting I/O with
    communication.

3.  This computation uses the CONVENTIONAL method, which means the
    result of integral computation is written to disk to be used in
    later iterations. GAMESS also provides a DIRECT method in which
    integrals are computed on the fly in each iteration. In order to
    decide scenarios of using conventional vs. direct methods, we may
    need to figure out the relationships between CPU speed to compute an
    integral, I/O bandwidth, data amount, etc.

4.  This is just a simple case. How many experiments do we need to
    explore parameters, subroutines that have significant impact on the
    overall performance for a computation? And for ALL computations
    provided by GAMESS?

Consider the scope of GAMESS, we can expect that there will be a huge
amount of performance data if we want to explore the answers to the
above questions. Handling these performance data manually can be
problematic.

![](media/image4.png){width="5.997916666666667in"
height="4.625694444444444in"}