# Presentation: 532341

## The ACTS CollectionBuilding a Reliable Software Infrastructure for Scientific Computing

**Osni Marques and Tony Drummond**
**Lawrence Berkeley National Laboratory**
**[*****oamarques,ladrummond*****]*****@lbl.gov***

**SC 2003, November 15-21 ****Phoenix, AZ**

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Outline

- Keeping the pace with the software and hardware
- Hardware evolution
- Performance tuning
- Software selection
- Other challenges
- The DOE ACTS Collection Project
- Goals
- Current features
- Gallery of applications
- Related activities
- Lessons learned
- Who benefits?

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## High Performance Computers (Sustainable Performance)

- ~ 20 years ago **** 1x106 Floating Point Ops/sec (Mflop/s)
- Scalar based
- ~ 10 years ago **** 1x109 Floating Point Ops/sec (Gflop/s)
- Vector & Shared memory computing, bandwidth aware
- Block partitioned, latency tolerant
- ~ Today **** 1x1012 Floating Point Ops/sec (Tflop/s)
- Highly parallel, distributed processing, message passing, network based
- data decomposition, communication/computation
- ~ 10 years away **** 1x1015 Floating Point Ops/sec (Pflop/s)
- Many more levels of memory hierarchy, combination of grids&HPC
- More adaptive, latency and bandwidth aware, fault tolerant, extended precision, attention to SMP nodes

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Tuning pays off!

**Automatic Tuning**
- PHiPAC: *www.icsi.berkeley.edu/~bilmes/phipac*
- ATLAS: *www.netlib.org/atlas*
- XBLAS: *www.nersc.gov/~xiaoye/XBLAS*
- Sparsity: *www.cs.berkeley.edu/~yelick/sparsity*
- FFTW: *www.fftw.org*

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## What About Software Selection? 

- Use a direct solver (*A*=*LU*) if
- Time and storage space acceptable
- Iterative methods don’t converge
- Many *b*’s for same *A*
- Criteria for choosing a direct solver
- Symmetric positive definite (SPD)
- Symmetric
- Symmetric-pattern
- Unsymmetric
- Row/column ordering schemes available
- MMD, AMD, ND, graph partitioning
- Hardware

- Build a preconditioning matrix *K* such that *Kx=b* is much easier to solve than *Ax=b* and *K* is somehow “close” to *A* (incomplete *LU* decompositions, sparse approximate inverses, polynomial preconditioners, preconditioning by blocks or domains, element-by-element, etc). See *Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods.*

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## We do not want this kind of problem to happen...

*http://wwwzenger.informatik.tu-muenchen.de/persons/huckle/bugse.html*

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Challenges in the Development of Scientific Codes

- Productivity
- Time to the first solution (prototype)
- Time to solution (production)
- Other requirements
- Complexity
- Increasingly sophisticated models
- Model coupling
- Interdisciplinarity
- Performance
- Increasingly complex algorithms
- Increasingly complex architectures
- Increasingly demanding applications

- Libraries written in different languages.
- Discussions about standardizing interfaces are often sidetracked into implementation issues.
- Difficulties managing multiple libraries developed by third-parties.
- Need to use more than one language in one application.
- The code is long-lived and different pieces evolve at different rates
- Swapping competing implementations of the same idea and testing without modifying the code
- Need to compose an application with some other(s) that were not originally designed to be combined

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Software Engineering Based in Components

***http://www.cca-forum.org***

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## The Reality...

- The development of complex simulation codes on parallel computers is not a trivial task.
- Usually, a significant percentage of the efforts focus on the development of the codes and their optimization.
- There is a need for a collaboration framework for ongoing development and deployment of computational tools.
- In 1999, the PITAC Report recommended the creation of a national library of certified domain-specific software in order to reduce the labor required for software development, testing and evolution.
- Research in computational sciences is fundamentally interdisciplinary and addresses, among many others, climate and environment modeling,  DNA sequencing, flows in geological structures, etc.

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## What is the ACTS Collection?

_**A**_dvanced _**C**_ompu_**T**_ational _**S**_oftware Collection
- Tools for developing parallel applications
- ACTS started as an “umbrella” project

***http://acts.nersc.gov***

**Goals**
*Extended support for experimental software *
*Make ACTS tools available on DOE computers *
*Provide technical support (acts-support@nersc.gov)*
*Maintain ACTS information center (http://acts.nersc.gov)*
*Coordinate efforts with other supercomputing centers*
*Enable large scale scientific applications*
*Educate and train *

***High***
- Intermediate level
- Tool expertise
- Conduct tutorials

***Intermediate***
- Basic level
- Higher level of support to users of the tool

***Basic***
- Help with installation
- Basic knowledge of the tools
- Compilation of user’s reports

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## ACTS Tools Functionalities 

| Category | Tool | Functionalities |
| --- | --- | --- |
| Numerical  | Aztec | Algorithms for the iterative solution of large sparse linear systems. |
|  | Hypre | Algorithms for the iterative solution of large sparse linear systems, intuitive grid-centric interfaces, and dynamic configuration of parameters. |
|  | PETSc | Tools for the solution of PDEs that require solving large-scale, sparse linear and nonlinear systems of equations. |
|  | OPT++ | Object-oriented nonlinear optimization package. |
|  | SUNDIALS | Solvers for the solution of systems of ordinary differential equations, nonlinear algebraic equations, and differential-algebraic equations. |
|  | ScaLAPACK | Library of high performance dense linear algebra routines for distributed-memory message-passing. |
|  | SuperLU | General-purpose library for the direct solution of large, sparse, nonsymmetric systems of linear equations. |
|  | TAO | Large-scale optimization software, including nonlinear least squares, unconstrained minimization, bound constrained optimization, and general nonlinear optimization. |
| Code Development
 | Global Arrays | Library for writing parallel programs that use large arrays distributed across processing nodes and that offers a shared-memory view of distributed arrays. |
|  | Overture | Object-Oriented tools for solving computational fluid dynamics and combustion problems in complex geometries. |
| Code Execution
 | CUMULVS | Framework that enables programmers to incorporate fault-tolerance, interactive visualization and computational steering into existing parallel programs |
|  | Globus | Services for the creation of computational Grids and tools with which applications can be developed to access the Grid. |
|  | PAWS | Framework for coupling parallel applications within a component-like model. |
|  | SILOON | Tools and run-time support for building easy-to-use external interfaces to existing numerical codes. |
|  | TAU | Set of tools for analyzing the performance of C, C++, Fortran and Java programs. |
| Library Development | ATLAS and PHiPAC | Tools for the automatic generation of optimized numerical software for modern computer architectures and compilers. |
|  | PETE | Extensible implementation of the expression template technique (C++ technique for passing expressions as function arguments). |

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Use of ACTS Tools

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Use of ACTS Tools

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Cosmic Microwave Background (CMB) Analysis

- The statistics of the tiny variations in the CMB (the faint echo of the Big Bang) allows the determination of the fundamental parameters of cosmology to the percent level or better.
- MADCAP (Microwave Anisotropy Dataset Computational Analysis Package)
- Makes maps from observations of the CMB and then calculates their angular power spectra. (See *http://crd.lbl.gov/~borrill*).
- Calculations are dominated by the solution of linear systems of the form *M=A*-1*B  *for dense *n*x*n* matrices *A* and *B* scaling as O(*n*3) in flops. MADCAP uses ScaLAPACK for those calculations.
- On the NERSC Cray T3E (original code):
- Cholesky factorization and triangular solve.
- Typically reached 70-80% peak performance.
- Solution of systems with *n* ~ 104 using tens of processors.
- The results demonstrated that the Universe is spatially flat, comprising 70% dark energy, 25% dark matter, and only 5% ordinary matter.
- On the NERSC IBM SP:
- Porting was trivial but tests showed only 20-30% peak performance.
- Code rewritten to use triangular matrix inversion and triangular matrix multiplication   one-day work
- Performance increased to 50-60% peak.
- Solution of previously intractable systems with *n* ~ 105 using hundreds of processors.

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Use of ACTS Tools

*EVH1 (Enhanced Virginia Hydrodynamics #1) benchmark*
*MPI code developed from VH1, based on the piece-wise parabolic method (PPM) of Colella and Woodward*
*PPM is a technique used for compressible, non-turbulent hydrodynamics. It has been used in a variety of astrophysical contexts, in addition to some ideal gas computations and studies of convection*
***TAU**** **has been used to profile EVH1*

*JRACY, time spent in each process.*

*JRACY, exclusive and inclusive level 1 data cache misses for all routines (except PARABOLA), mean over 16 processors. *

*Visualizing TAU traces with Vampir, a commercial trace visualization tool from Pallas, GmbH.*

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Matrix of Applications and Tools

- ... More Applications ...

***http://acts.nersc.gov/AppMat***

***Enabling sciences***
***and discoveries...***
***with***
***high performance and scalability...***

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Why do we need these tools?

- High Performance Tools
- portable
- library calls
- robust algorithms
- help code optimization
- More code development in less time!
- More simulation in less computer time!

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Related Activities

- Software Repositories:
- Netlib:*  http://www.netlib.org*
- HPC-Netlib:*  http://www.nhse.org/hpc-netlib*
- National HPCC Software Exchange NHSE:*  http://www.nhse.org*
- Guide to Available Mathematical Software:*  http://gams.nist.gov*
- MGNet:*  http://www.mgnet.org *
- NEOS:*  http://www-fp.mcs.anl.gov/otc/Guide *
- OO Numerics:  *http://oonumerics.org/oon*
- Portable timing routines, tools for debugging, compiler technologies:
- Ptools:*  http://www.ptools.org*
- Center for Programming Models for Scalable Parallel Computing:* http://www.pmodels.org *
- Education:
- Computational Science Educational Project:  *http://csep1.phy.ornl.gov** *
- UCB’s Applications of Parallel Computers:*  **http://www.cs.berkeley.edu/~demmel/cs267_Spr99*
- MIT’s Applied Parallel Computing: * **http://www.mit.edu/~cly/18.337*
- Dictionary of algorithms, data structures and related definitions: *http://www.nist.gov/dads*

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Lessons Learned

*There is still a gap between tool developers and application developers which leads to duplication of efforts*.
*The tools currently included in the ACTS Collection should be seen as dynamical configurable toolkits and should be grouped into toolkits upon user/application demand*.
*Users demand long-term support of the tools*.
*Applications and users play an important role in making the tools mature*.
*Tools evolve or are superseded by other tools*.
*There is a demand for tool interoperability and more uniformity in the documentation and user interfaces*.
*There is a need for an intelligent and dynamic catalog/repository of high performance tools.*

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Why is ACTS unique?

- Provides pointers and documentation about software tools.
- Accumulates the expertise and user feedback on the use of the software tools and scientific applications that used them:
- independent software evaluations
- participation in the developer user groups e-mail list
- leverage between tool developers and tool users
- presentation of a gallery of applications
- workshops and tutorials
- tool classification
- support

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Scientific Computing – Third Pillar of Science

***http://www.science.doe.gov/scidac***

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## Addressing the Performance Gap through Software

- Peak performance is skyrocketing
- In 1990s, peak performance increased 100x; in 2000s, it will increase 1000x
- But
- Efficiency for many science applications declined from 40-50% on the vector supercomputers of 1990s to as little as 5-10% on parallel supercomputers of today
- Need research on
- Mathematical methods and algorithms that achieve high performance on a single processor and scale to thousands of processors
- More efficient programming models for massively parallel supercomputers

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## High Performance Tools

***http://acts.nersc.gov***

**Tool descriptions, installation details, examples, etc**

**Agenda, accomplishments, conferences, releases, etc**

**Goals and other relevant information**

**Points of **
**contact**

**Search engine**

- portable
- library calls
- robust algorithms
- help code optimization
- Scientific Computing Centers like NERSC:
- Reduce user’s code development time that sums up in more production runs and faster and effective scientific research results
- Overall better system utilization
- Facilitate the accumulation and distribution of high performance computing expertise
- Provide better scientific parameters for procurement and characterization of specific user needs

**See also:***** http://acts.nersc.gov/documents***

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003

## References

**An expanded Framework for the Advanced Computational Testing and Simulation Toolkit,**** *****http://acts.nersc.gov/documents/Proposal.pdf***_*[ ](http://acts.nersc.gov/documents/ToolCertification.pdf)*_
**The Advanced Computational Testing and Simulation (ACTS) Toolkit. *****Technical Report LBNL-50414*****. **
**A First Prototype of PyACTS. *****Technical Report LBNL-53849*****. **
**ACTS - A collection of High Performing Tools for Scientific Computing. *****Technical Report LBNL-53897*****.**
**The ACTS Collection: Robust and high-performance tools for scientific computing. Guidelines for tool inclusion and retirement. *****Technical Report LBNL/PUB-3175*****. **
**An Infrastructure for the creation of High End Scientific and Engineering Software Tools and Applications. *****Technical Report LBNL/PUB-3176*****.**

**References**

*To appear: ACM TOMS featuring ACTS Tools.*

- The DOE ACTS Collection (http://acts.nersc.gov)

- 11/19/2003