# Presentation: 388226

## Visualization Frameworks

## Where are we now?

- Despite years of effort and demonstrations of remote vis technology, users predominantly use serial desktop tools
- Download data to workstation and use locally
- Use serial tools over remote X11 connections (just to avoid moving the data to a local workstation... that’s desperation!)
- Fractured component technology and remote vis efforts
- Open Source Frameworks (Parallel VTK, OpenDX)
- Commercial tools/frameworks (CEI Ensight, AVS Express, ...
- Standalone tools (Visapult, Terascale Browser)
- Lack of generality
- Do any of these tools offer a comprehensive solution that works on the emerging Supercomputer Architectures?
- No?
- Will they ever interoperate?
- Not likely without common architecture to write to...

## A “Distributed Visualization Architecture”

- We will not be able to use Grid/Network infrastructure effectively without distributed/parallel remote visualization systems!
- Remote visualization has repeatedly demonstrated advantages
- We won’t be able to do remote/distributed visualization effectively without a common framework that enables us to share/combine our work!
- There has been no common delivery platform to enable pervasive adoption by developers and users alike

## About Frameworks/Architectures

- Frameworks/Architectures are
- Rigid formalisms encoding (*enforcing*) best practices
- A way to encode for well-understood (*menial*) tasks so developers can focus on high level concepts
- A way to encode things we understand and have already thought out (*familiar/commonly used techniques are what we consider “menial”*)
- A method that does not readily accommodate new concepts *(but what does?)  *
- We should expect to use a framework primarily encode current practices.  Anything that **some** domain expert cannot answer “the implementation is obvious” should **not** be encoded in a framework.

## What Should We Expect of a“Visualization Architecture” (DiVA)

- Modular component framework supporting community contributions
- Supports distributed/parallel components
- Supports remote vis (*eg. Latency tolerance, desktop interactivity*)
- Supports streaming/out-of-core/progressive execution model
- Decouple BackEnd distributed components from presentation/GUI
- Permits reuse of same compute-intensive components for different presentation methods and interfaces contexts
- Means we need a standard way to talk to back end components
- OGSA for visualization tools? (*grid speak for service abstraction...)*
- Robust internal data model(s)
- Essential feature of other community frameworks like OpenDX, AVS, and VTK
- Encode basic vis & science data structures *(FEM, Geom, Block)*
- Domain Decomp, hierarchical rep, progressive encoding, information indices (*commonly neglected in current frameworks!)*
- Perhaps limit scope to “big four” applications requirements

## What to Expect of a DiVA (cont...)

- Effortless selection and placement of components on distributed computers
- Requires a mature Grid (eg. Grid Application Toolkits)
- Requires common data model (or collection thereof)
- Requires robust performance model and runtime instrumentation
- Basic Data Transport between network-connected components
- Stream/discretized  : reliable/unreliable)
- Can leverage heavily on data model for higher level info representation
- Support of both batch (data intensive) and interactive (compute intensive) data analysis
- Data Mining, feature extraction, data summarization (batch)
- Interactive Visualization and Analysis (interactive)
- Data Preprocessing, reorg. and indexing, for interactive vis. (batch)
- Integration with Storage Resource Management (include QoS)
- Essential for efficient use of scarce network resources
- Includes data staging, cataloging, scheduling of preprocessing tasks,

## Conclusions

- Data Analysis Infrastructure will make or break the future computing platforms
- The way we currently do business is insufficient
- If we have to stop computing due to inability to analyze the data, then we waste considerable $$$$ and render purpose of large scale computing moot.
- Must have framework to support remote/distributed visualization
- We must have a way to share our work on remote vis pipelines
- Does not preclude any particular GUI/Presentation method
- Introduces substantial technical challenges despite comparatively conservative goal
- Please look at the rest of the slides for additional supporting information!!!!

## Interfaces and Displays (Presentation)DiVA must support all

- Web (supports collaboratory/portal efforts)
- Domain-specific interfaces (req. toolkits approach)
- Use DiVA objects underneath
- Thick, Thin, and Slender client interfaces
- Dataflow GUIs
- For vis experts
- Interpreted Data Analysis Languages (must drive Parallel back-end)
- Language for data processing that leads to visual representations
- Data extraction, reorganization, processing (eg. *spectral analysis*)
- Collaborative Environments
- Audio/Video integration: Access Grid, VRVS
- Shared Event and Data Spaces: CavernSoft
- CAVEs, Walls, and Immersive Displays
- Use immersion or ultra-high resolution for complex visual representations
- not the same as “big data” (danger of overselling this technology)

## Data Preprocessing for Vis

- Data reorganization is expensive and it is mandatory!
- Optimal data output methods for a simulation is rarely optimal for interactive visualization
- Efficient data representations for remote visualization include
- Multiresolution representations (Also applicable to AMR)
- Non-stream and progressive encodings
- Also applicable to loss tolerance for unreliable transport
- Indexing Hierarchies (isosurface accelerators)
- Wavelets (not just for image encoding)
- Multidimensional Splines
- Space filling Curves
*All require expensive data intensive pre-processing*

## Data Preprocessing Requires SRM Integration!

- Data reorganization requires offline pre-processing to re-encode/re-organize the data
- Must be tightly integrated part of Storage Resource Management effort
- Efficient scheduling of data replication & staging
- Efficient scheduling of data reorganization/preprocessing to support interactive visualization
- Compute & Cache (C&C) is insufficient
- It implies data mirroring with limited intelligence
- But these data repositories with attached computation are a critical component of a Data Analysis Architecture and require considerable sophistication to be effective!
- We must have Data Analysis Centers (DAC’s) as first order components of the Ultrascale Simulation Architecture

## Application Performance on the Grid

**Level 1**: Base (*map of the pipeline onto the virtual machine is explicit*)
- Uniform Security, I/O, data model compatability (basic Grid services)
- Ability to explicitly launch apps on a static map of machines.
**Level 2**: Static Maps (*optimal* *mapping of application to virtual machine*)
- Get a static mapping of resources that provides best overall performance
- Requires predictive performance models (heuristic, parameterized/algorithmic, statistical/history-based)
**Level 3**: Dynamic Maps (*runtime optimization*)
- Requires continuous instrumentation feedback to the parameterized models of performance.
- Must support multiple parallel pipelines dynamically refactored depending on response profile (which map can respond most rapidly)
- Requires commensurability between different methods that produce the same image

## Pipeline Performance Parameters

- Dynamic Response Constraints and Parameters
- Responds dynamically to runtime/user-defined constraints
- Display Framerate
- Datasets/sec Throughput (eg. Shuttling through datasets)
- Recompute on param change (eg. Change isosurface level)
- Respond to runtime resource constraints
- Contract violation
- hardware/network failure (fault tolerance)
- Respond to runtime dynamic data requirements
- Different data payloads or algorithm performance based on algorithm parameter choices
- Different data payloads or algorithm performance due to changing data characteristics

## Conclusion on Pipeline Example

- Just simple change in isolevel completely changes optimal pipeline selection!
- No single remote vis methodology is best in all circumstances (even at runtime)!
- Must have commensurable visual output from many different methods
- Simply scheduling resources for these overlapping pipelines will be hard, muchless auto-selecting between them!
- Must have a common framework to deliver a dynamic multi-pipeline visualization capability.
- so we can focus our effort on the “hard stuff”!