# Presentation: 685251

## Simulation and Modeling at the Exascale for Energy, Ecological Sustainability and Global SecurityAnalytics and Visualization (part of Goal 8) Position Statements 

**Town Hall Meeting ****Berkeley Lab ****April 17-18**

## Visualization/analytics architectures, engineering and deployment 

_**Lead position statements:  **_
_** **__**Hank Childs (LLNL)**_
_** **__**Jim Ahrens (LANL)**_

**Design and implementation of highly parallel visual data analysis; **
**Component architectures;  **
**applications -- barriers, opportunities; **
**Software and release engineering; **
**where are opportunities to leverage related efforts in research and industry? **
**What are deployment targets?  **
**How to best deliver/deploy research or production quality software?  **
**Leveraging emerging hardware and computational platforms.**

## Session 5: Visualization/analytics architectures, engineering and deployment

**Hank Childs**
**4/18/07**

**Breakout session 8**

## What’s a production postprocessing tool all about?

**WARNING**
**Use cases: data exploration, presentations, visual debugging, comparative analysis, quantitative analysis**
**Many other groups will deploy their technologies here**
**Used by code customers, code developers, postprocessing experts**
**Customers have an “emotional relationship” with the tool**
**Use the tool for hours a day**
**Care about:**
**Interface ...  how many buttons do I have to click?**
**Robustness / bugginess**
**Interactivity ******** exascale!!!!**
**Functionality ******** exascale!**

- Exascale is the obvious challenge ... but we can’t lose sight of the other concerns ... we are doomed to failure if we do

## Development model

**Agile!**
**Deliver to customers early and often.**
**It won’t be perfect**
**Lots of developers must be able to participate ... including universities**
**Pushes towards a module / component based design**
**Must have good SQE (regression tests, repo, etc.)**
**Sit with the customer.  Embed with the customer.  Become the customer.**

## Exascale random thoughts

**RTI visualization of 27B elements ... data parallel ... this won’t last**
**What are the requirements?**
**Interactive ******** co-processing?**
**Non-interactive ******** streaming?**
**LOD? (Is this a vis only solution?)**
**We’re getting lots of compute power (multi-core) ... but is it what we need?**
**How I/O bound are we?  (I think very...)**
**Co-processing: today’s SC is for the sim codes (high compute / good interconnect) ... what we want is different (high I/O / high mem)**

## Visualization/Analytics Architectures, Engineering and Deployment

**James****	**** Ahrens**
**Los Alamos National Laboratory**
**Lead Position Statement**

## What is today’s state-of-the art?

**Architectures, Engineering and Deployment**
**Requirements**
**Exascale**
**Applications**
**Climate/Chem, Energy, Microbal Life/Bioinformatics, Network Security, Innovative Computing, Social Sim.**

**State-of- the-art examples**
**Some general purpose tools**
**Application-specific tools**
** **
**Integration a challenge or impossible**
**Visualization**
**Ensight, Paraview, Visit**
**Ferret, CDM**
**Analytics**
**ITK**
**Boost**
**Workflow**
**Kepler, VisTrails**

## What are trying to accomplish? What is feasible to accomplish in 5-10 years?

**Architectures, Engineering and Deployment**
**Dynamic custom interfaces for application communities**
**Sharing/leveraging of technology that solve general complex infrastructure challenges**
**Minutes to design a custom analysis application that incorporates interface and systems functionality from multiple institutions**

**Layered Approach**
**Specific Apps.**
**Climate, Bio, Cyber**
**Applications**
**Scientific, Analytics, Observation/Experiment **
**User interfaces**
**Data Structures and Algorithms**
**Systems**
**Parallelism, Distributed, Streaming, Real-Time**
**Specific Systems**
**Many-core, custom design**

## How would we accelerate development?

- Interface

- Data Structures and Alg.

- Systems

- Custom systems

## How would we accelerate development?

**How would we accelerate development?**

- Biology

- Climate

- Network

- Distr.

- Interface

- Data Structures and Alg.

- Systems

- Custom systems

- Energy

## How would we accelerate development?

- Biology

- Climate

- Network

- Distr.

- Structured ...

- Parallelism

- Custom

- Interface

- Data Structures and Alg.

- Systems

- Custom systems

- Isosurface

- Energy

- Obs/Exp

## How would we accelerate development?

**How would we accelerate development?**

- Biology

- Climate

- Network

- Interface

- Data Structures and Alg.

- Systems

- Custom systems

- Energy

## How would we accelerate development?

- Biology

- Climate

- Distr.

- Parallelism

- Custom

- Interface

- Data Structures and Alg.

- Systems

- Custom systems

## What are the major challenges? What are the major risks?

**Challenges**
**Flexibly **
**Integration**
**Provide recognition**
**Questions**
**Who controls the product?**
**How are the interfaces/framework defined?**
**Balancing technology focus...**
**Components etc.**
**Focus on barriers driven by analysis (user interface, interrupt-driven execution, etc.)**

**Risks**
**Driven by interface - leads to monolithic systems**
**Redundancy**
**Not achieve community potential**

## What are the expected outcomes? Scale of investment?

**Expected outcomes**
**Successful deployment and sharing will lead to:**
**Improved science and understanding**

**Scale of investment**
**Double existing visualization program**
**Multiple experts developing independent pieces of software**
**Type of investment**
**Software with interfacing in mind**

## Visualization algorithms

_**Lead position statements:  **_
_** **__**Ken Joy (UC DAVIS)**_
_** **__**Valerio Pascucci (LLNL)**_

**Temporal analysis; **
**Comparative visualization and analysis; **
**Integrated information and scientific visualization; **
**Algorithms of extreme scale: **
**what paradigms/barriers will we encounter ****	****on the road to Exascale?**

## The Scalability Dimension ofRobust Quantitative Data Analysis 

**Valerio Pascucci****Center for Applied Scientific Computing, LLNL****Adjunct Prof. UC Davis****E3SGS, LBNL****April 19, 2007**

## Topological analysis of the channel structures in porous media

**We need quantitative data analysis for massive models in a variety of domains**

- Topological analysis of the channel structures in porous media

- Multiscale tracking of combustion particles

## Traditional data analysis tools are often ineffective for massive models

**Massive scientific models ****are challenging**
**Sheer volume of information **
**Complexity of the ****information represented**
**Tools do not scale with the ****data sizes**
**Difficult to capture multiple scales **
**Numerical methods unstable and sensitive to noise**
**Difficulty in providing error bounds**
**Lack of a mathematical language makes hard to reproduce results and map them to new definitions**
**Difficult to reproduce the data analysis process**

- Rayleigh–Taylor instability (Miranda)

**Notes:**
In addition to difficulties of bringing out important features, current visualization techniques have some limitations when applied to 3D data.  In large simulations like the RT instability simulation shown at right, the geometric complexity is so great that isosurfaces and even slice planes are not able to provide more than a qualitative view of the data.  In the image at right the top isosurface occludes itself.  Such occlusion can occur in small data sets given non-trivial geometries.  In addition, visualization tools treat time-varying data as a set of discrete snaphots.  Scientists that want to see the temporal features of their data must create a resonable visualization on one time step and create an animation.  This process is slow, and usually must be iterated to get the right set of parameters for the entire time sequence.  Our solution is to apply topological analysis to extract representations of the shape of data sets that are independent of the mesh, and thus do not need to be embedded in the 3D space of the simulation data.  These representations can also be used to represented an entire time sequence, giving a scientist to investigate the temporal properties of their data directly.

## Scalable tools require concurrent application of multiple techniques

**Scalable tools require concurrent application of multiple techniques**

- Functionality

- Scalability

## We achieve high performance withnew streaming data techniques

**We achieve high performance with****new streaming data techniques**

- RAWDATA

- isosurfaceextraction

- multiscaleMorse analysis

- geometric tracking

- quantitative analysis

- 4.5TB

- feedback for adjustment and revaluation

**Data analysis workflow**

## First time scientists can quantify robustly mixing rates by bubble count

## We provide the first quantification of known stages of the mixing process

**Mode-Normalized Bubble Count****(log scale) **

**Derivative**

**Bubble Count**

- -3

- -2.5

- -2

- -1.5

- -1

- -0.5

- 0

- 1.0 (9524 bubbles)

- 0.01

- 0.1

- 0.5

- slope 1.536

- slope 2.575

- slope 1.949

- t/**(log scale)**

**Derivative of Bubble Count**

- linear growth

- weak turbulence

- mixing
- transition

- strong
- turbulence

## Exa-challenges in analytical data exploration and understanding

**Methods:**
**Statistical methods**
**Image processing**
**Topology**
**Multi-scale models**
**Machine learning**
**App. specific **
**Compression (lossy?)**
**Techniques:**
**Robust computations**
**Multi-resolution**
**Stream processing**
**Parallelism**
**Tackle available platforms:**
**Multicore**
**FPGA’s, **
**Cell proc.**
**Paralle/Distributed**

**Challenges:**
**Theoretical integration**
**Characterization of the unexpected**
**Mathematical framework feeding back into science**
**Uncertainty characterization and presentation**
**Progressive methods for early user feedback**
**Unified management and processing of experimental and simulated data**
**Integrated data movement and processing**
**Opportunistic use of new technology**

## Organizational challenges

**Facilitate free basic research VS focus on applications and deployment**
**Exploration of many approaches and tools VS coordinated integration of multiple techniques**
**Incremental work VS disruptive technologies**
**Use ownership VS evangelization**
**Graduating tools from first use to large distribution (making ourselves useless)**

## Disruptive Technologies and Approaches

_**Lead position statements:  **_
_**Cecilia Aragon (LBNL)**_
_**Laura Monroe (LANL)**_

**These are topics not usually associated with mainstream visualization efforts in DOE that will likely figure more prominently in the future.  **
**Human-computer interfaces; **
**Usability; **
**Coupled visual and traditional analysis; **
**Analytics, visual analytics.**

## Disruptive Technologies

**Cecilia Aragon **
**LBNL**
**What is the rate-limiting step in the current state of data understanding?**

## Rate-limiting step in data understanding

**Amount of data****in the world**

- 5 exabytes

**Moore’s Law**

**Human cognitive capacity**

## Need for new research area: scientist-computer interfaces

**Science discovery depends on the scientist**
**The tools should not be the drivers**

## Why should DOE fund interface research?

**Visual interfaces for science will be different than commodity interfaces**
**Scientific approach to interfaces: develop metrics for visual interface evaluation**
**Commercial applications: 50-80% user interface code**

## Interface: A Fresh LookLaura Monroe (LANL)

**Questions, dichotomies **
**and a position or two**

## Back to the Beginning

**GOAL: Interfaces that speak to the scientific work at hand**
**Maybe one-size-fits-all is not the best thing**
**OPPORTUNITY: Use the power of the systems**
**OPPORTUNITY: Take this long planning time to reconsider**

## Dichotomy 1: Input vs. Output

**Input: from you to the computer **
**Punch cards**
**Keyboard**
**Keyboard and mouse (mid 80s)**
**Tracking, Voice, Tablet ), Collaborative distance interactions (ongoing work)**
**Output: from the computer to you**
**Printouts**
**Amber or green character-based monitor**
**B&W windowing system on a single monitor (mid 80s)**
**Color windowing on a single monitor (late 80s)**
**VR, Aural feedback, Haptics, Augmented reality ), Collaborative distance interactions (ongoing)**
**Our standard interface in both areas dates from the 80s.**** **

## Q1: What do we have now that we didn’t have in the mid-80s?

**High-resolution monitors**
**Much higher performance **
**Connectedness**
**Individual computers networked for one problem (SETI)**
**Internet**
**Wireless connections between individual computers**
**Access to information**
**OK search capabilities**
**Small scale and large scale computers are built out of the same building blocks**
**Are interface changes needed?**
**Are interface changes desirable?**

## Q1 again: What will we have in 10 years that we didn’t have in the mid-80s?

**Still flat panels? What size and resolution?**
**Ultra high-res?**
**Pocket-sized high-res monitors? (like your improved cell phone)**
**Much higher performance with many-core etc**
**Connectedness**
**Better search capabilities**
**Image-based search?**
**New input devices? (That are useful in at least one domain – i.e., not just a novelty)**
**More use of consumer products? (like cell phones etc)**
**Are interface changes needed?**
**Are interface changes desirable?**

## Q2: How do we learn about the world?

**Direct interaction**
**The world provides a huge amount of “data” to us at every moment**
**The ways we interact with the world are restricted, but not as much as our current computer interfaces**
**Input: Text-based or not too high-res**
**Output: Mostly visual, not too high-res, mostly planar displays**
**The ways we interact with the world are learned and refined through experience**
**Can we widen our computer interfaces to mimic this process more closely? Can we have learning interfaces?**

## Q3 again: How do we learn about the world?

**Extrapolate from what we already know**
**Apply symbols that capture something we already know to something we don’t know**
**Incremental move from the known**
**This involves a layer of domain-specific abstraction.**

## Dichotomy 2: Natural vs. Abstract

**Natural interface gives ease of use**
**Not always possible**
**But fairly universal**
**Examples: voice and language, sound, haptic**
**A new abstract technique might have a learning curve**
**But if it is powerful enough, the learning curve is worth it, and the technique will become natural**
**Abstractions are often domain-specific**

## Multi-modal approaches

**Disparate data**
**Combine viz, sound, text, graphs as appropriate**
**Once again the known abstractions are put to use on a new problem.**
**Can we use the parallel capability of many-core systems for this?**

## Some positions

**Domain-specific (NOT a killer interface for all possible applications)**
**Expert interface designers, along with the expert viz people already assisting scientists?**
**Interface toolkits, along with viz toolkits?**
**Use the abstractions already in use**
**Small scale consumer stuff + large scale**
**Multi-modal**

## Questions for the scientists

**What do you really want to do with your data that you can’t do right now? **
**Is there anything obviously missing in the interface for *****you***** and *****your problem*****? What drives you crazy when you are trying to get at your data?**
**How do you prefer to work with *****your kind of data*****?**

## Summary:Data Analysis and Visualization 1

**Vision: integrative development environment**
**There is a hidden costs in vis and analysis:**
** ****data exploration, prepare presentations, visual debugging,**** comparative analysis, quantitative analysis**
**Uses what/need robust, reliable systems that are simple to use**
**Need high IO/high memory machines **
**Shared disks - moving exabytes of data prohibitive**
**Vision: do data analysis and visualization on SuperComputer**
**Vision: tools framework for Exascale analysis and vis**
**Need to integrate vis, analysis, workflow tools**
**Vision: dynamic custom interfaces for app communities**
**Put together in “minutes” alternative interaface-tools connections**
**Share infrastructure between application domains**
**Layering of  interfaces – data structures – system**
**Standards are needed to enable reusability**

## Summary:Data Analysis and Visualization 2

** ****Vision: Develop tools providing insight for exascale data**
**Challenges: sheer volume, complexity of features**
** ****Current tools do not scale**
** ****Need to visualize and analyze both details and coarse scales **
** ****Need scaling for parallel analysis **
** ****Many algorithms need to be redesigned to scale**
** ****Formal/theoretical integration of several techniques:**
** ****statistical, topological, vision, mining, ...**
** ****Uncertainty characterization**
** ****Robust computations**
**Shorten gap between research and deployment**
**Reward dissemination and testing of new tools**
**Integration of formal mathematical models and computer science**

## Summary:Data Analysis and Visualization 3

**Vision: Interface research needed to improve interaction with users**
**Visual interface evaluation**
**Commercial applications: 50% of software in UI.**
**Need languages capturing high level concepts, e.g. parallelism**
**Vision: interfaces that speak to the scientific work at hand**
**You-to-computer**
** ****Now keyboard, maybe voice**
**Computer-to-you**
** ****Printout, monitors, ...**
**Now have**
**High resolution monitors, connectivity, **
**Can we take advantage of new hardware coming**
**High-res, many-core, multi-modal, ...**
**Interactivity important – web is limiting**
**Not having to be concerned with find data, reorganize, fit it in available space**