# Presentation: 295187

## Scaling to New Heights

- Retrospective
- IEEE/ACM SC2002 Conference
- Baltimore, MD
- [Trimmed & Distilled for SOS7 by M. Levine
- 4-March-2003, Durango]

## Contacts and References

- David O’Neal  *oneal@ncsa.uiuc.edu*
- John Urbanic  *urbanic@psc.edu*
- Sergiu Sanielevici  *sergiu@psc.edu*
- 	Workshop materials:
- 	*www.psc.edu/training/scaling/workshop.html*

## Introduction

- More than 80 researchers from universities, research centers, and corporations around the country attended the first "Scaling to New Heights" workshop, May 20 and 21, 2002, at the PSC, Pittsburgh.
- Sponsored by the NSF leading-edge centers (NCSA, PSC, SDSC) together with the Center for Computational Sciences (ORNL) and NERSC, the workshop included a poster session, invited and contributed talks, and a panel.
- _Participants examined issues involved in adapting and developing research software to effectively exploit systems comprised of thousands of processors. [Fred/Neil’s __**Q1**__.]_
- 	The following slides represent a collection of ideas from the workshop

## Basic Concepts

- All application components must scale
- Control granularity; Virtualize
- Incorporate latency tolerance
- Reduce dependency on synchronization
- Maintain per-process load; Facilitate balance
- 	Only new aspect, at larger scale, is the degree to which these  things matter

## Poor Scalability?(Keep your eye on the ball)

## Good Scalability? (Keep your eye on the ball)

## Performance is the Goal! (Keep your eye on the ball)

_**Performance**_** is the Goal!**** ****(Keep your eye on the ball)**

## Issues and Remedies

- Granularity 		[**Q2a**]
- Latencies 		[**Q2b**]
- Synchronization
- Load Balancing 	[**Q2c**]
- Heterogeneous Considerations

## Granularity

- 	Define problem in terms of a large number of small objects independent of the process count  [Q2a]
- Object design considerations
- Caching and other local effects
- Communication-to-computation ratio
- Control granularity through virtualization
- Maintain per-process load level
- Manage comms within virtual blocks, e.g. Converse
- Facilitate dynamic load balancing

## Latencies

- Network
- Latency reduction lags improvement in flop rates; Much easier to grow bandwidth
- Overlap communications and computations; Pipeline larger messages
- Don’t wait – Speculate! [Q2b]
- Software Overheads
- Can be more significant than network delays
- NUMA architectures
- 	Scalable designs must accommodate latencies

## Synchronization

- Cost increases with the process count
- Synchronization doesn’t scale well
- Latencies come into play here too
- Distributed resource exacerbates problems
- Heterogeneity another significant obstacle
- Regular communication patterns are often characterized by many synchronizations
- Best suited to homogeneous co-located clusters
- 	Transition to asynchronous models?

## Load Balancing

- Static load balancing
- Reduces to granularity problem
- Differences between processors and network segments are determined *a priori*
- Dynamic process management requiring distributed monitoring capabilities [Q2c]
- Must be scalable
- System maps objects to processes

## Heterogeneous Considerations

- Similar but different processors or network components configured within a single cluster
- Different clock rates, NICs, etc.
- Distinct processors, networking segments, and operating systems operating at a distance
- Grid resources
- 	Elevates significance of dynamic load balancing; Data-driven objects immediately adaptable

## Tools [Q2d?]

- Automated algorithm selection and performance tuning by empirical means, e.g. ATLAS
- Generate space of algorithms and search for fastest implementations by running them
- Scalability prediction, e.g. PMaC Lab
- Develop performance models (machine profiles; application signatures) and trending patterns
- 	Identify/fix bottlenecks; choose new methods?

## Topics for Discussion

- How should large, scalable computational science problems be posed?
- Should existing algorithms and codes be modified or should new ones be developed?
- Should agencies explicitly fund collaborations to develop industrial-strength, efficient, scalable codes?
- What should cyber-infrastructure builders and operators do to help scientists develop and run good applications?

## Summary Comments (MJL)

- Substantial progress, with scientific payoff, is being made.
- It is hard work without magic bullets.
**>>>** Dynamic load balancing **<<<**
- Big payoff, homogeneous and heterogeneous
- Requires considerable people work to implement
- Runtime overhead very small.

## Case Study:NAMD Scalable Molecular Dynamics

- Three-dimensional object-oriented code
- Message-driven execution capability
- Fixed problem sizes determined by biomolecular structures
- Embedded PME electrostatics processor
- Asynchronous communications

## Slide 18

## Slide 19

## Slide 20

## Slide 21

## Case Study:Summary

- As more processes are used to solve the given fixed-size problems, benchmark times decrease to a few milliseconds
- PME communication times and operating system loads are significant in this range
- Scaling to many thousands of processes is almost certainly achievable now given a large enough problem
- 700 atoms/process x 3,000 processes = 2.1M atoms