# Presentation: 010679

## Ultrascale Distributed Data Analysis Architecture

- John Shalf
- LBNL/NERSC

## The Earth Simulator

- Compute Engine
- 40TF peak
- Memory 10TB
- Disk 700TB
- 70:1 Disk/RAM, 17.5:1 Disk/TF (*has proved insufficient*)
- Had to suspend new jobs recently due to inability to handle data production rate
*We don’t want that to happen to us!*

## Ultrascale Simulation Architecture

- 3 machines at 3 locations
- ORNL:  Cray X1  (~100TF)
- NERSC:  IBM SP / Blue Planet (~160 TF)
- 128-256TB of RAM
- 2.5PB of directly attached storage
- 19.5:1 Disk/RAM, 15:6:1 Disk/TF (*this should worry us*)
- ANL: IBM Blue Gene/L (~180TF)
- Dark Fiber UltraGrid Network
- Compute & Cache (C&C):  Data Analysis Centers with compute + high-performance storage co-located
- Must push data out these C&C sites (mirroring)
- Storage Resource Management must be an integral part of this grid!

## Data Production Rates(from Bill Johnston:  “Big Four Application Summaries”)

- Climate
- Now: 20-40TB per simulated year
- 5 yrs:  100TB/yr 5-10PB/yr
- Fusion
- Now: 100Mbytes/15min
- 5 yrs: 1000Mbytes/2 min with realtime comparison with running experiment, 500Mbits/sec guaranteed (QoS)
- High Energy Physics
- Now: 1-10PB data stored, Gigabit net.
- 5 yrs: 100PB data, 100Gbits/sec net
- Chemistry (Combustion and Nanostructures)
- Now: 10-30TB data
- 5 yrs: 30-100TB data, 10Gbits/sec multicast
- Astrophysics (JS. Opinion)
- Now and 5 yrs:  Can soak up anything you build!

## Interim UltraGrid Network (1qCY03

- _Phase I: __ORNL-LBL-ANL 10Gb/s test backplane __($4M) __, 40Gb/s testnet __($2M)_
- _Phase 2a: __60 Gb/s backplane __($12M/yr, 5 yrs)_
- _Phase 2b: __Optical extensions to add’l labs, bandwidth as required (up to backbone speed)_
- _Phase 3: __160-240 Gb/s backplane__ _

*To*
*ORNL*

*To*
*LBL*

- _Scalable Testbed_
- Leverages I-WIRE: **$2M**
- Live test of architecture & solutions
- Vendor proving ground

*Source:*
*Charlie Catlett*

## Source:

- C

- S

- C

- C

- S

- I

- C

- S

- C

- C

- S

- I

- C

- S

- C

- C

- S

- I

- C&C

- C&C

- C&C

- C&C

- C&C

- C&C

- C

- S

- C

- C

- S

- I

- C&C

- C&C

- C&C

- C&C

- C&C

- C&C

**bandwidth**

**Bandwidth**
**QoS**

**Bandwidth**
**QoS**
**Network cache and compute**

**Bandwidth**
**QoS**
**Network cache and compute**
**Robust bandwidth**

*Source:*
*Bill Johnston*

## Canonical Visualization Pipeline

- Read Data

- Filter
- Extract Geom

- Render

- Presentation
- GUI

- Remote Vis applications
- Repartition the pipeline
- Collapse stages of the pipeline
- Parallelization: SIMD and pipelined
- Improve throughput between stages
- Data reduction / Progressive Transmission (info proc & encoding)
- Protocol/transfer acceleration (hardware, drivers & protocols)
- Each method optimal for a narrow set of conditions

## Where are we now?

- Despite years of effort and demonstrations of remote vis technology, users predominantly use serial desktop tools
- Download data to workstation and use locally
- Use serial tools over remote X11 connections (just to avoid moving the data to a local workstation... that’s desperation!)
- Fractured component technology and remote vis efforts
- Open Source Frameworks (Parallel VTK, OpenDX)
- Commercial tools/frameworks (CEI Ensight, AVS Express, ...
- Standalone tools (Visapult, Terascale Browser)
- Lack of generality
- Do any of these tools offer a comprehensive solution that works on the emerging Ultrascale Architecture?
- No?
- Will they ever interoperate?
- Not likely without common architecture to write to...

## We Need a DiVA!A “Distributed Visualization Architecture”

- We will not be able to use this infrastructure effectively without distributed/parallel remote visualization systems!
- Remote visualization has repeatedly demonstrated advantages
- We won’t be able to do remote/distributed visualization effectively without a common framework that enables us to share/combine our work!
- There has been no common delivery platform to enable pervasive adoption by users
- Frameworks/Architectures are
- Rigid formalisms encoding (*enforcing*) best practices
- A way to encode for well-understood (*menial*) tasks so developers can focus on high level concepts
- A way to encode things we understand and have already thought out (*familiar/commonly used techniques are what we consider “menial”*)
- A method that does not readily accommodate new concepts *(but what does?)  *So we should expect to primarily encode current practices.

## What Should We Expect of a“Visualization Architecture” (DiVA)

- Modular component framework supporting community contributions
- Supports distributed/parallel components
- Supports remote vis (*eg. Latency tolerance, desktop interactivity*)
- Supports streaming/out-of-core/progressive execution model
- Decouple BackEnd distributed components from presentation/GUI
- Permits reuse of same compute-intensive components for different presentation methods and interfaces contexts
- Means we need a standard way to talk to back end components
- OGSA for visualization tools? (*grid speak for service abstraction...)*
- Robust internal data model(s)
- Essential feature of other community frameworks like OpenDX, AVS, and VTK
- Encode basic vis & science data structures *(FEM, Geom, Block)*
- Domain Decomp, hierarchical rep, progressive encoding, information indices (*commonly neglected in current frameworks!)*
- Perhaps limit scope to “big four” applications requirements

## What to Expect of a DiVA (cont...)

- Effortless selection and placement of components on distributed computers
- Requires a mature Grid (eg. Grid Application Toolkits)
- Requires common data model (or collection thereof)
- Requires robust performance model and runtime instrumentation
- Basic Data Transport between network-connected components
- Stream/discretized  : reliable/unreliable)
- Can leverage heavily on data model for higher level info representation
- Support of both batch (data intensive) and interactive (compute intensive) data analysis
- Data Mining, feature extraction, data summarization (batch)
- Interactive Visualization and Analysis (interactive)
- Data Preprocessing, reorg. and indexing, for interactive vis. (batch)
- Integration with Storage Resource Management (include QoS)
- Essential for efficient use of scarce network resources
- Includes data staging, cataloging, scheduling of preprocessing tasks,

## Conclusions

- Data Analysis Infrastructure will make or break the Ultrascale Computing Platform
- The way we currently do business is insufficient
- If we have to stop computing due to inability to analyze the data, then we waste considerable $$$$ and render purpose of ultrascale computing moot.
- Must have framework to support remote/distributed visualization
- We must have a way to share our work on remote vis pipelines
- Does not preclude any particular GUI/Presentation method
- Introduces substantial technical challenges despite comparatively conservative goal
- Please look at the rest of the slides for additional supporting information!!!!

## Interfaces and Displays (Presentation)DiVA must support all

- Web (supports collaboratory/portal efforts)
- Domain-specific interfaces (req. toolkits approach)
- Use DiVA objects underneath
- Thick, Thin, and Slender client interfaces
- Dataflow GUIs
- For vis experts
- Interpreted Data Analysis Languages (must drive Parallel back-end)
- Language for data processing that leads to visual representations
- Data extraction, reorganization, processing (eg. *spectral analysis*)
- Collaborative Environments
- Audio/Video integration: Access Grid, VRVS
- Shared Event and Data Spaces: CavernSoft
- CAVEs, Walls, and Immersive Displays
- Use immersion or ultra-high resolution for complex visual representations
- not the same as “big data” (danger of overselling this technology)

## Data Preprocessing for Vis

- Data reorganization is expensive and it is mandatory!
- Optimal data output methods for a simulation is rarely optimal for interactive visualization
- Efficient data representations for remote visualization include
- Multiresolution representations (Also applicable to AMR)
- Non-stream and progressive encodings
- Also applicable to loss tolerance for unreliable transport
- Indexing Hierarchies (isosurface accelerators)
- Wavelets (not just for image encoding)
- Multidimensional Splines
- Space filling Curves
*All require expensive data intensive pre-processing*

## Data Preprocessing Requires SRM Integration!

- Data reorganization requires offline pre-processing to re-encode/re-organize the data
- Must be tightly integrated part of Storage Resource Management effort
- Efficient scheduling of data replication & staging
- Efficient scheduling of data reorganization/preprocessing to support interactive visualization
- Compute & Cache (C&C) is insufficient
- It implies data mirroring with limited intelligence
- But these data repositories with attached computation are a critical component of a Data Analysis Architecture and require considerable sophistication to be effective!
- We must have Data Analysis Centers (DAC’s) as first order components of the Ultrascale Simulation Architecture

## Application Performance on the Grid

**Level 1**: Base (*map of the pipeline onto the virtual machine is explicit*)
- Uniform Security, I/O, data model compatability (basic Grid services)
- Ability to explicitly launch apps on a static map of machines.
**Level 2**: Static Maps (*optimal* *mapping of application to virtual machine*)
- Get a static mapping of resources that provides best overall performance
- Requires predictive performance models (heuristic, parameterized/algorithmic, statistical/history-based)
**Level 3**: Dynamic Maps (*runtime optimization*)
- Requires continuous instrumentation feedback to the parameterized models of performance.
- Must support multiple parallel pipelines dynamically refactored depending on response profile (which map can respond most rapidly)
- Requires commensurability between different methods that produce the same image

## Pipeline Performance Parameters

- Dynamic Response Constraints and Parameters
- Responds dynamically to runtime/user-defined constraints
- Display Framerate
- Datasets/sec Throughput (eg. Shuttling through datasets)
- Recompute on param change (eg. Change isosurface level)
- Respond to runtime resource constraints
- Contract violation
- hardware/network failure (fault tolerance)
- Respond to runtime dynamic data requirements
- Different data payloads or algorithm performance based on algorithm parameter choices
- Different data payloads or algorithm performance due to changing data characteristics

## A Simplified Example of Vis Pipeline Responsiveness

- A simple (*cooked*) performance model
- 50M triangles/sec (24-byte tri-strips) Graphics HW (1/8 for 8 PEs)
- 1 Second to compute isosurface with one processor (1/8 for 8PEs)
- 1 Gigabit Network with perfect performance
- Perfect Speedup for parallel algorithms
- The real world will offer a more complex performance model (just an example)

- Read
- Data

- Desktop
- Isosurface

- Desktop
- Render

- Desktop
- Display

- 8PE cluster
- Isosurface

- 8PE cluster
- Render

- Yellow arrows indicate
- choices in distributed
- application data flow

## Vis Pipeline Responsiveness

## Vis Pipeline Responsiveness

- Best Throughput

## Vis Pipeline Responsiveness

- Best Throughput

## Conclusion on Pipeline Example

- Just simple change in isolevel completely changes optimal pipeline selection!
- No single remote vis methodology is best in all circumstances (even at runtime)!
- Must have commensurable visual output from many different methods
- Simply scheduling resources for these overlapping pipelines will be hard, muchless auto-selecting between them!
- Must have a common framework to deliver a dynamic multi-pipeline visualization capability.
- so we can focus our effort on the “hard stuff”!

## Vis Pipeline Responsiveness(Original Data: only works on MacOS-X)