# Presentation: 812095

## Beyond Workflows - DOE Cloud Computing Paradigm and the SDMRole and Future

**Mladen A. Vouk, Nagiza Smatova, Paul Breimyer, Pierre Moualem, Mei Nagappan, and  the whole SPA team  (list available separately)**
**Scientific Data Management Center – Scientific Process Automation Group**
**NC State University, Raleigh, NC 27695**

## Overview

**Scientific Workflow technology – ****A success story from the past 7 years in the SDM center (a technology used in production or otherwise by application people) ****–  Developed components: **_**Workflows, Provenance, “Dashboard”, other**_
**DOE SDM “Cloud” -****Vision for the future of the SDM centre ****– Integration of components - **_**Intelligent Analytics and Social Networks, Component-based “cloud”, Integrated Services (service oriented architecture)**_
**Sustainable science - ****Long term approach for the survival of SDM center technology (Beyond SciDAC and longer) ****– Integration of **_**Research, Engineering, Transfer-of-Technology, Partnerships, Results (ROI, TOC)**_

## Scientific Process Automation

**A key differentiating element of a successful information technology (IT) is its ability to become a true, valuable, and economical contributor to cyberinfrastructure.**
**An IT-assisted workflow represents a series of structured activities and computations that arise in information assisted problem solving.**
_**Scientific process automation principles, as well as production level pilots, is SDM’s Key Contribution **_**over last 7 years – Smokey Mountains retreat.**
**From NC State: numerous publications, 3 graduated PhD and 4 MS with thesis students, several in progress, several generations of software.**

## Environment

**Environment**

- Computations

- Orchestration
**(Kepler)**

- _Data, DataBases_**Provenance...**_Storage_

- Analytics

- Control Panels
- (**Dashboard**)
- & Display

- Networking
- Local/Remote
- ... “Cloud” Services

- Analytics

- Analytics

- Computations

- Orchestration
**(Kepler)**

- _Data, DataBases_**Provenance...**_Storage_

## Workflow Framework

**Workflow Framework**

- Provenance,
- Tracking &
- Meta-Data
- (DBs and Portals)

- Control Plane
- (light data flows)

- Execution
- Plane
- (“Heavy
- Lifting”
- Computations
- and flows)

- Synchronous or Asynchronous

- Kepler

## Actor/Process in a Broader Sense

**Out**

**Network/”Cloud”**

**Bsub < code_run**
**------------ where code_run is a script --------------**
**code_run**
**#! /bin/csh **
**source /usr/local/lsf/conf/cshrc.lsf **
**#BSUB -W 5 **
**#BSUB -n 100 **
**mpiexec ./code**
**#BSUB -o /share/vouk/WFLOW/code.out.%J **
**#BSUB -e /share/vouk/WFLOW/code.err.%J **
**#BSUB -J codevouk**
**-------------------------**** **

**In**

## Modular Framework

**Modular Framework**

**Supercomputers**
**+**
**Analytics Nodes**

**Kepler**

**Dash**

**Storage**

- Meta-Data about:
**Processes,**
**Data,**
**Workflows,**
**System, Apps & Environment**

**Orchestration**

**Data**
**Store**

**Rec**
**API**

**Disp**
**API**

**Management  API**

- Access

- Trust

## Read More ...

**Singh M.P. and M.A. Vouk, "Network Computing," in John G. Webster (editor), *****Encyclopedia of Electrical and Electronics Engineering*****, John Wiley & Sons, New York, Vol. 14, pp. 114-132, 1999 **
**S Klasky, M Beck, V Bhat, E Feibush, B Ludäscher, M Parashar, A Shoshani, D Silver and M Vouk, "Data management on the fusion computational pipeline," SciDAC 2005, Journal of Physics: Conference Series 16 (2005), 510-520, doi:10.1088/1742-6596/16/1/070 **
**Ilkay Altintas, Oscar Barney, Zhengang Cheng, Terence Critchlow, Bertram Ludaescher, Steve Parker, Arie Shoshani and Mladen Vouk, "Accelerating the scientific exploration process with scientific workflows," sciDAC 2006, Journal of Physics: Conference Series 46 (2006), 468-478, doi:10.1088/1742-6596/46/1/065 **
**M. A. Vouk, I. Altintas R. Barreto, J. Blondin, Z.Cheng, T. Critchlow, A. Khan, S. Klasky, J. Ligon, B. Ludaescher, P. A. Mouallem, S. Parker, N. Podhorszki, A. Shoshani, C. Silva, " Automation of Network-Based Scientific Workflows," Proc. of the IFIP WoCo 9 on Grid-based Problem Solving Environemnts: Implications for Development and Deployment of Numerical Software, IFIP WG 2.5 on Numerical Software, Prescott, AZ, 2006, printed in IFIP, Vol 239, "Grid-Based Problem Solving Environments, eds. Gaffney PW and Pool JCT (Boston: Springer), pp. 35-61, 2007**
**Klasky, S.; Barreto, R.; Kahn, A.; Parashar, M.; Podhorszki, N.; Parker, S.; Silver, D.; Vouk, M.A. "Collaborative visualization spaces for petascale simulations," Proceedings of the CTS 2008 - International Symposium on Collaborative Technologies and Systems, pp 203-211, Digital Object Identifier 10.1109/CTS.2008.4543933,10-23 May 2008**
**More... http://sdm.ncsu.edu**

## DOE Cloud

**“****Cloud” computing – ****builds on decades of research in virtualization, distributed computing, utility computing,  grids, and more recently networking, web and software services****.**
**It implies a **_**seamless**_** service oriented and component-based architecture - ****delivery of an integrated and orchestrated suite of on-demand functions to an end-user through composition of both loosely and tightly coupled functions, or services - often network-based****, reduced information technology overhead for the end-user, service orchestration, virtualization of resources, great flexibility, reduced total cost of ownership, different “flavors”.**
_**Intelligent Analytics and Knowledge-Creating Social Networks, Component-based “Clouds”, Seamless/Integrated Services**_
_**Necessary in the context of Peta- and Exa- sciences, data, etc.**_

## “Analytics Cloud"

**“****Analytics Cloud"**

- Knowledge creation
- & Integration,
- Social Networking,
- Provenance,
- Tracking &
- Meta-Data
- (DBs and Portals)

**Execution**
**Plane - “Heavy duty”  in-cloud**
**Computations, Flows Services**

- W/F
- Engine

- Concept-driven Analytics

- W/F Generation
- Wizard

- Run-time Manager and Scheduler

- Synchronous & Asynchronous Services

**Workflow control plane**

- Analytics Enabled Resources

- Supercomputers

- Clusters

- Supercomputers

- Active Storage

- Other “cloud” devices

## Components

***Reusability (elements can  *****be re-used in other workflows)**
***Substitutability *****(alternative implementations are easy to insert, very precisely specified interfaces are available, run-time component replacement mechanisms exist, there is ability to verify and validate substitutions, etc), *****extensibility and scalability *****(ability to readily extend system component pool and to scale it, increase capabilities of individual components, have an extensible and scalable architecture that can automatically discover new functionalities and resources, etc),**
***Customizability (ability to customize generic *****features to the needs of a particular scientific domain and problem),**
***Composability (easy *****construction of more complex functional solutions using basic components, reasoning about such compositions, etc.). There are other characteristics that also are very important.**
***Reliability and availability *****of the components and services, **
***Cost*****  - the cost of the services, total cost of ownership, economy of scale**
**Security and privacyand so on.**

## Example: Meta-Data Framework

**Example: Meta-Data Framework**

**Supercomputers**
**+**
**Analytics**

**Kepler?**

**Dash**

**Storage**

**Orchestration**

**Custom**
**Web**

**Other.**
**..**

## Fault-Tolerance – Clouds of Clouds

- Master DB
- (replicated)

## User Categories

**Developers (10)**
**Service Authors (100 to 1,000)**
**Service Integrators (100– 10,000)**
**End-users (1000 - ?)**

## Read More ...

**Sam Averitt, Michael Bugaev, Aaron Peeler, Henry Shaffer, Eric Sills, Sarah Stein, Josh Thompson, Mladen Vouk “Virtual Computing Laboratory (VCL),” In the proceedings of the International Conference on Virtual Computing Initiative, May 7-8, 2007, IBM Corp., Research ****Triangle Park, NC, pp. 1-16.**
**Mladen Vouk, Sam Averitt, Michael Bugaev, Andy Kurth, Aaron Peeler, Andy Rindos*, Henry Shaffer, Eric Sills, Sarah Stein, Josh Thompson ,  “Powered by VCL” - Using Virtual Computing Laboratory (VCL) Technology to Power Cloud Computing, Published in the Prelim. Proceedings of the 2nd International Conference on Virtual Computing Initiative, 15-16 May 2008, RTP, NC, pp. 1-10, final version to be available through the ACM Digital Library**
**Mladen A. Vouk, “Cloud Computing – Issues, Research and Implementations,” ITI08, to appear in IEEE Digital Library**
**Google for “cloud computing” ...**
**Other ..**

## Sustainable Science

**A Long term approach for the survival of SDM center technology (Beyond SciDAC and longer)**
_**Research**_
_**Engineering**_
_**Transfer-of-Technology, **_
_**Partnerships with scientists**_
_**Operational open-source tools**_
_**Visible results (agreed upon ROI, and an accounting of TOC)**_