# Presentation: 530453

## Implementing Scientific Process Automation- from Art to Commodity 

- Implementing Scientific Process Automation*- from Art to Commodity*
- Mladen A. Vouk and Terence Critchlow

**Notes:**
Good morning ladies and gentlemen. It is an pleasure and a privilege to have the opportunity to present to you today. IBM has been a top corporate and research friend of this university for a long time, and it is a special honor for me to have you here today. 

What I will talk about is the fact the world is going flat  - business, science and interactions are going global – and universities, and their implementations and offerings in information technology need to recognize this. Specifically, I will touch upon the research and development activities of NC State University as they relate to virtualization of information technology resource, and how these services relate to our mission (education, research, outreach) and to our interactions with IBM.

## Overview	

- From Art to Commodity
- Component-based System Engineering
- Kepler vs. CCA vs ???
- Domain Specific
- Virtualization and Service-based System Engineering

## Team (the artists?)

- Ilkay Altinas
- Zhengang Cheng
- Terence Chritchlow
- Bertram Ludaesche
- Brent Marinello
- Pierre Moualem
- Steve Parker
- Elliot Peele
- Mladen A. Vouk
- Anthony Wilson
- Others ...(including the R&D of the Kepler and Ptolemy community, and W/F developers)

- John Blondin
- Doug Swesty
- Scott Klasky
- ....
- Numerous Kepler and Ptolemy users and apps

- Development and Support (Active)

- End-Users

## Cyberinfrastructure

*“**Cyberinfrastructure makes *_***applications dramatically easier to [use,] develop and deploy***_*, thus expanding the feasible scope of applications possible within budget and organizational constraints, and *_***shifting the [educator’s,] scientist’s and engineer’s effort away from information technology (development) and concentrating it on [knowledge transfer, and] scientific and engineering research***_*. Cyberinfrastructure also increases efficiency, quality, and reliability by capturing commonalities among application needs, and facilitates the efficient sharing of equipment and services.”*

- (from the Appendix of the Report of the National Science Foundation Blue-Ribbon Advisory Panel on Cyberinfrastructure, Jan 2003)

**Notes:**
What I am showing here is an excerpt from a 2003 NSF Blue Ribbon report on the future of Cyberinfrastructure report. It basically says that IT needs to be implemented in a fashion (utility like) that reduces the overhead/burden on the end-user to minimum so that scientists, educators, engineers, etc., can go about the business of doing their business rather than handle and fight technology. I think we all agree that that is the way to go.

## Rising Expectations

- Delivery of Service with *Minimization of Information Technology (IT) Overhead*
- Move away from the specific resources (e.g., h/w, s/w, net, storage, and app), to the ***ability to achieve one’s basic mission*** (learning, teaching, research, outreach, administration, etc.)
***Utility-like (appliance-like) on-demand access*** to needed IT. Use of IT-based solutions moves from a fixed location (such as an specific lab) and fixed resources (e.g., particular operating system) to a _(mobile) personal access_ device a scientist (e.g., laptop or a PDA or a cell phone) and _**service-based**__ delivery_
**Business model** behind IT virtualization & services needs to conform to the **mission** of the institution, as well as realistic resource and/or personnel constraints.

**Notes:**
There are rising expectation about minimization of the overhead, and focus on delivery of functionalities and services rather than platforms, operating systems, and applications.

Of course, the notions of virtualization, services, on-demand, and autonomic are very familiar to IBM and our visitors. After all, IBM is one of the originators of many of the ideas behind the terms, has been for 30+ years. More recently, Dr. Horn and IBM research, as well as the RTP Blade computing group and Tivoli, have been spearheading the autonomic computing and on-demand computing notions, as well as web-based and other network-based services in various forms.

## Overview	

- From Art to Commodity
- Component-based System Engineering
- Kepler vs. CCA vs ???
- Domain Specific
- Virtualization and Service-based System Engineering

## Component-Based System Engineering*

- Composition of systems (e.g., practical workflows) from (existing) components
- Systems as **assemblies** of components
- Development of components as **reusable** units
- Facilitation of the **maintenance** and **evolution** of systems by customizing and replacing their components
**Methods and tools** for support of different aspects of component-based approach
**Open** – source and process issues, organizational and management issues, coupling, domain, technologies (e.g., component models), component composition issues, tools...

**Building Reliable Component-Based Software Systems, **_**[Ivica](http://www.idt.mdh.se/%7Eicc/)**__**[ ](http://www.idt.mdh.se/%7Eicc/)**__**[Crnkovic](http://www.idt.mdh.se/%7Eicc/)**_
- and _[Magnus Larsson](http://www.idt.mdh.se/%7Emlo/)_ (editors), _[Artech](http://hhp.bvdep.com/artechhouse/default.asp?)__[ House Publishers](http://hhp.bvdep.com/artechhouse/default.asp?)_, ISBN 1-58053-327-2,
- http://www.idt.mdh.se/cbse-book/

## Advantages

- _Business:_ Shorter time-to-market, lower development and maintenance costs
- _Technical:_ Increased understandability of (complex) systems; Increased usability,  interoperability, flexibility, adaptability, dependability...
- _Strategic:_ Increasing (software) market share
- _Scope:_ Web- and internet-based applications, Desktop and office applications, Mathematical and other libraries, Graphical tools, GUI-based applications, etc.
- _Practical:_ De-facto standards, e.g., MS COM, .NET, Sun EJB, J2SEE, CORBA Component Model...
- Focus on **functionality**

## Some Issues

- Standards
- Non-functional requirements
- Performance including timing,
- Resource management
- Dependability, fault-tolerance
- Domain specific requirements?
- Skill level needed and IT overhead
- Changing expectations and scalability ...
- Coupling and synchronization (synchronous vs. asynchronous processing, loose vs. tight coupling, parallelization, data movement, bottlenecks ...)
- Provisioning / Security challenges
- Marketing hype/over-expectations
- Customer confusion / skepticism
- Service quality/control issues
- Demonstrating benefits/ROI
- Software Licensing
- Vendor Factor
- Other ...

## What is it? (*)

- The basis is the **Component**
- Components can be **assembled** according to the rules specified by the **component model**
- Components are assembled through their **interfaces**
- A **Component Composition** is the process of assembling components to form an assembly, a larger component or an application
- Component are performing in the context of a **component framework**
- All parts conform to the **component model**
- A **component technology** is a concrete implementation of a component model

- (*) From the talk entitled “**Component-based development**  challenges in building reliable systems”, by Crnkovic  at IIS 2005, Sept 2005.

## Component Framework

**Component ****Framework**

**Platform**

**Components**

**Repository**

**Supporting Tool**

- (*) From the talk entitled “**Component-based development**  challenges in building reliable systems”, by Crnkovic  at IIS 2005, Sept 2005.

## Component

- A unit of composition
- Contractually specified interfaces
- Explicit context dependencies (only?).
- Can be deployed _independently_
- Subject to composition by third party.
- Confirms a component model which defines interaction and composition standards
- Composed without modification according to a composition standard.
- Specified as a) black-box (signal/response), b) gray-box (access to internal states), c) white-box (access to all internals), or d) display-box (see internals, but cannot touch internals).

## Principles

**Reusability** (docs, re-use process, architecture, framework, V&V, ...)
**Substitutability** (alternative implementations, functional equivalence, equivalence on other issues, very precise interfaces and specs, run-time replacement mechanism, V&V, ...)
**Extensibility** (extending system component pool, increasing capabilities of individual components – extensible architecture, resource and new functionality discovery, V&V, ...)
**Composability** (functional, extra-functional, reasoning about compositions, V&V, ...)

## Scientific Workflow Automation (e.g., Astrophysics, v-Desk)In conjunction with John Blondin, NC State UniversityAutomate data acquisition, transfer and visualization of a large-scale simulation at ORNL

- Scientific Workflow Automation (e.g., Astrophysics, v-Desk)**In conjunction with John Blondin, NC State University****Automate data acquisition, transfer and visualization of a large-scale simulation at ORNL**

## Scientific Workflow Automation (e.g., Astrophysics, v-Desk)In conjunction with John Blondin, NC State UniversityAutomate data acquisition, transfer and visualization of a large-scale simulation at ORNL

- Scientific Workflow Automation (e.g., Astrophysics, v-Desk)**In conjunction with John Blondin, NC State University****Automate data acquisition, transfer and visualization of a large-scale simulation at ORNL**

## Scientific Workflow Automation (e.g., Astrophysics, v-Desk)In conjunction with John Blondin, NC State UniversityAutomate data acquisition, transfer and visualization of a large-scale simulation at ORNL

- Scientific Workflow Automation (e.g., Astrophysics, v-Desk)**In conjunction with John Blondin, NC State University****Automate data acquisition, transfer and visualization of a large-scale simulation at ORNL**

## Overview	

- From Art to Commodity
- Component-based System Engineering
- Kepler vs. CCA vs ???
- Domain Specific
- Virtualization and Service-based System Engineering

## CCA and Kepler

**CCA** is probably more suitable for tightly coupled applications, especially when all the components are ready within a machine or a local cluster. CCA focuses on high performance parallel and distributed processing.
**Kepler** is probably more suitable for loosely coupled and diverse components. The component can reside on different and widely separated machines. It is great for service and data that resides with its owner, and are exposed as services. Kepler focuses on process orchestration and control. More conducive of IP protection.

## CCA Component Architecture Model

**CCA components** interact with each other with a specific **CCA framework** implementation through standard CCA interfaces. Each component defines its inputs and outputs in **Scientific IDL**; these definitions are deposited in, and can be retrieved from a repository by using the **CCA Repository API**. In addition, these definitions serve as input to a proxy generator which generates **component stubs**: the component-specific parts of GPorts (white box in the picture). The components can also use framework services directly through the **CCA Framework Services** Interface. The **CCA Configuration API** ensures that the the components can collaborate with different builders associated with different frameworks.

**Chasm** - an F90 interoperability library from Los Alamos.
**Babel/SIDL** - an object oriented language interoperabilty interface definition language.
**CCA Specification** - The Common Component Architecture Specification for high performance components.
**Ccaffeine** - a CCA framework compliant with the CCA specification.
**Ccaffeine GUI** - A Graphical User Interface that works with Ccaffeine.

**Kepler**

## Option A: CCA-Aware Actor

- Actor interacts with CCA components. Kepler director only needs to pass relevant parameters.
- If such actor is not available, each CCA component may require a customized stub.
- Modification only in Kepler space.
- Maintains tight coupling among CCA components, e.g., for performance.

## Option B: CCA Component Service

- CCA component exposes an interface (service) that can be directly orchestrated by Kepler.
- Requires extra work on all CCA components. This might not be possible on all machines.

- Kepler Director

- Service
- actor

- CCA
-  Com.
- Service

- CCA
-  Com.
- Service

- CCA
-  Com.
- Service

## Option C: CCA-Aware Service Proxy

- Kepler and CCA bridged through a Service Proxy. It translates service requests into interactions.
- Advantage:
- Decouples CCA and Kepler
- Service Proxy can be very flexible
- No Special CCA execution module is required on the client.
- Disadvantage:
- Single point vulnerability as all Kepler users are depends on the Proxy. Extra overhead?

- Kepler Director

- Service
- actor

- CCA
-  Com.

- CCA
-  Com.

- CCA
-  Com.

- CCA Aware
- Service Proxy

## Q&A

- Q: I understand that I need to wrap CCA components into actors in order to use Kepler. But what new capabilities will I obtain by doing that compared to using CCA components and CCaffeine framework? In fact, what I like about CCA, is that one can use components written in different languages. How do I do this using Kepler?
- A: By wrapping CCA components, Kepler will able to use them. This allows CCA components and diverse Kepler services to work together, thus broaden application area.
- 	Kepler is in Java. It is used primarily for service orchestration, not execution. Each service runs in its specific environment. Programs written in other language can be enacted through the actor of that specific language. For example Perl actor to run Perl scripts.

## Q&A (2)

- Q: How do I distinguish more loosely coupled applications from more tightly coupled (to choose between CCA and Kepler)? Both use ports (interfaces), so how can I quantify an interface (loosely vs tightly coupled)?
- A: Tightly coupled components usually means synchronous and intense communication and control. For example, frequent amount of timing sensitive data exchanged among the components. CCA is more suitable in these case.
- 	Loosely coupled services are usually more focused on control and distributed services and data. May not be suitable for intense data-flows unless they are virtualized. Services from different service providers. Kepler is suitable for this area.

## Overview	

- From Art to Commodity
- Component-based System Engineering
- Kepler vs. CCA vs ???
- Domain Specific
- Virtualization and Service-based System Engineering

## Domain Specific

- Klasky FSP W/F
- Swesty TSI W/F
- Coleman W/F
- ChemInfo W/F
- SciRun
- Blondin TSI W/F
- Other ...

## Fusion Simulation Project Workflow Pilot In conjunction with Scott Klasky, CESP, PPPL, ORNL Automation of simulation, transfer and analytics of FSP

## Slide 29

## Slide 30

## Slide 32

## Scientific Workflow Automation (e.g., Astrophysics, v-Desk)In conjunction with John Blondin, NC State UniversityAutomate data acquisition, transfer and visualization of a large-scale simulation at ORNL

- Scientific Workflow Automation (e.g., Astrophysics, v-Desk)**In conjunction with John Blondin, NC State University****Automate data acquisition, transfer and visualization of a large-scale simulation at ORNL**

## Workflow - Abstraction

**Model**

**SendData**

**Merge &**
**Backup**

**To VizWall**

**Parallel Computation**

**RecvData**

**Parallel Visualization**

**Data Mover Channel**
**(e.g. LORS, BCC, SABUL, FC over SONET**

**Split & Viz**

**Web or Client GUI**

**Web**
**Services**

**Head Node**
**Services**

**Head Node**
**Services**

**Mass Storage**

**Fiber C. or Local NFS**

**Model**
**Merge**
**Backup**
**Move**
**Split**
**Viz**

**Construct**
**Orchestrate**
**Monitor/Steer**
**Change**
**Stop/Start**

**Control**

## Astrophysics Workflow (using Ptolemy II framework)

## Blondin Workflow V2

- Submit Job

- Merge

- Transfer

- Slicing
- /Dicing

- Viz via
- Ensight

- Browser
- Display

**Supercomputer Cray (Phoenix @ ORNL)**

**Local Cluster (Orbitty @ NCSU)**

**User Laptop**

- Web DB

## Basic

- A number of processing steps
- Range of data transfer rates
- Parallelism
- Speedup?
- Implementation (e.g., Scripts? App?)
- Ease of use (e.g., LORS)
- Tracking (e.g., DB, Web, provenance)
- Fault-Tolerance
- Distributed (for most part)

## Runtime Data Collection

- Each run of the workflow is associated with a runid. The info is organized around it.

## Workflow Screenshot (Swesty)

**Notes:**
The workflow we created is based on the Ptolemy II framework from UC Berkeley.  A screenshot is shown on this slide.  The meat of the action takes place in the upper half of this screenshot, so I’ve blown that up a bit more and show it on the next slide.

## Screenshot: Log

**Notes:**
An example of output from the logging actor is shown on this screen as displayed by the Mozilla web browser.  The left-hand frame is a table of contents of the individual files that can be shown.  Clicking on one of those links will cause that log file to be displayed in the bottom-right frame.  The top right frame shows all the logging messages received by the actor.

## Screenshot: Running

**Notes:**
The same combination of all the status messages is also shown in the workflow when it gets run as shown in this screenshot.  This is an easy way to get an idea about what the workflow is doing if you happen to be sitting in front of the same computer the workflow is run on.

## Generic Actors (Swesty W/F)

- This workflow uses a small number of actors repeatedly to deliver a complex behavior
- 160 instances an actors
- 18 different types of actors
-  ~100 Expression actor instances
- 13 boolean switch actor instances
- 13 array manipulation actor instances
- 8 ssh actor instances
- < 30 instances of other actors (ex: sleep, file I/O, etc)

**Notes:**
Another nice property to note about the workflow is that it uses mostly generic actors.  It turns out that there are around 160 instances of actors, and 18 different types of actors.  Of the 160 instances, about 100 are Expression actors, and there are around 13 BooleanSwitch actors, 13 array manipulation actors, and 8 SSH remote execution actors.  There are fewer that 30 actors of other types, such as the sleep actor and some file I/O actors.

## Work in Progress

- Validation of input data
- llsubmit script and config files need to be consistent
- Distributed infrastructure
- Start this from a web page
- Monitor and control the workflow from a remote site
- Incorporation of data analysis
- A small workflow driven by the specific simulation results and how Doug wants to visualize the data

**Notes:**
I’ll now describe a few things we’re working on now to improve the workflow.

First, we would like the workflow to perform some validation of input data.  When a simulation is submitted, there are configuration files as well as the llsubmit script that are currently manually created.  These configuration files contain some variables that must be mutually consistent.  These are currently manually validated.  If they are not consistent, then the simulation may not run even after waiting for days in the queue.  So we’d like to have the workflow automatically perform some of this validation.

Another item we are working on is to change the architecture of the workflow to be more distributed.  Currently, the workflow generates log files that can be served from a web server, so monitoring of the job progress can be done remotely.  However, we still need to start the workflow from the machine on which it will be run.  So we are working on a mechanism that allows the workflow to be initiated remotely.

Finally, we’re working on incorporating the data analysis portion of the workflow.

## Notes

- A complex workflow that is typical of many scientific workflows
- Follows the run simulation, move data, and analyze data paradigm
- Much can be done with a set of generic actors:
- Expression actor
- Ssh2Exec actor
- BooleanSwitch & Array actors
- Configuration parameters allow simple adaptation to different environments

**Notes:**
In summary, I’ve described today a workflow the SPA team created with Doug’s group.  This workflow is typical of many scientific workflows in that it follows the same paradigm of running the simulation, and transferring the data.

From our experience, we’ve learned that a small set of generic actors can be used to perform complex and useful jobs.  Also, the workflow framework supports configuration parameters that allow workflows to be easily configured to run in different environments.

Our website is at the bottom of the slide if you’d like to download the software and have a try at creating your own workflows.

This is the end of my presentation, thank you for your attention, and I will now take questions.

## Notes (Blondin)

- W/F support system needs to be
- Flexible - everything changes often!  If such tools are to be used by application scientists they need to be easy to reconfigure.
- Detachable - for one reason or another, one component may not work (network not usable, disks full on local end), so one would like to fire up individual parts of the workflow as needed.
- Fault-tolerant - ideally, the software itself can recognize some faults and  correct them (eg, re-attempt file upload).

## Key Issue

- Very important to distinguish between a custom-made workflow solution and a more cannonical set of operations, methods, and solutions that can be composed into a scientific workflow.
- Complexity, skill level needed to implement, usability, maintainability, “standardization”
- e.g., sort, uniq, grep, ftp, ssh on unix boxes
- SAS (that can do sorting), home-made sort,
- LORS, SABUL, bbcp (free, but not standard), etc.

## Overview	

- From Art to Commodity
- Component-based System Engineering
- Kepler vs. CCA vs ???
- Domain Specific
- Virtualization and Service-based System Engineering

## Make everything into a Service

- Network-based and On-demand
- Complements an access/communication unit of choice
- Utility-like: ubiquitous, reliable, available, maintainable
- Nearly Device-independent
- May be application level or smaller granularity (e.g., functions, web-service, grid-service)

## Virtualization

- Services

- Middleware

- Hardware

- Applications

- Provisioning

- Operating Systems

- To effectively deliver on-demand computing services that are
- maintainable, scalable, and customizable it is essential that
- the tiers of virtualization are separated but can be coupled

## Scientific Workflow Automation (e.g., Astrophysics, v-RP)In conjunction with John Blondin, NC State UniversityAutomate data acquisition, transfer and visualization of a large-scale simulation at ORNL

- Input
- Data

- Highly
- Parallel
- Computer

- Output files
- ~500x500

**Aggregate** to ~500 files (< 10+GB each)

- HPSS
- archive

- Data Depot

- Logistic Network
- L-Bone

- Local Mass
- Storage 14+TB)

**Aggregate** to one file (~1 TB each)

- Viz
- Wall

- Viz Client

- Local 44 Proc. Data Cluster
- - data sits on local nodes for weeks

- Viz Software

- Scientific Workflow Automation (e.g., Astrophysics, v-RP)**In conjunction with John Blondin, NC State University****Automate data acquisition, transfer and visualization of a large-scale simulation at ORNL**

**Notes:**
Another example is the workflow shown here. It is part of a large Department of Energy research project on orchestration, management and “virtualization” and component-based engineering of scientific workflows.  The workflow shown is a production workflow, and it stretches between Oak Ridge National Laboratory (and their Cray computers) and NC State. At ORNL it uses high-performance parallel computing engines to perform supernova computations, while at the NCSU the scientist performs data analyses. 

While the scientist does coding for the purposes of simulation of the astrophysics phenomena, there is no interest in tending to IT issues (e.g., long-haul large volume data transfers – terabytes of data, workflow recovery and management). However, there is a considerable interest in automation of the workflow, data provenance and efficient resource utilization. We are investigating use of VCL facilities for provision of some of the anchoring and workflow services.

## A Resource “Utility Wall”

- V-Desks
- V-Flow
- (resource
- collections)
- 1-1-1 to
- N-M-K
- ...

**Notes:**
What about something closer to home? Indeed, to pave the way for broader use of campus analytics (both at the universities and in K-12 environments), we are about to deliver business intelligence services to a number of customers in the form of v-desks. In cooperation with FI (and running on SUR provided blades) we plan to research both the technical challenges and the pedagogical challenges of delivering on-demand services to our K-12 environments.

For example, a blade, on-demand loaded with either a v-class application (e.g., SAS software) or a business intelligence image (v-desk) offers on-demand or scheduled secure services to either local or remote users either per seat basis within a particular time-slot, or on a semester-long or project-long basis. V-desks are typically dedicated one-user environments, v-classes are servers that provide the same application to a group of users who may need it simultaneously (synchronously – a special challenge) or asynchronously. Services are offered at the granularity the user needs, from operating system, to platform, to application, to function, and in a end-user platform independent way.

## vcl.ncsu.edu

**vcl.ncsu.edu**

- Ongoing:
- Tipping pt.
- Usability
- Availability
- Pedagogy
- ...
- Currently
- scaling to
- 8000+ users
- - N-M-K

## Advantages (2)

**Easy to use remote access** from one's own desktop or mobile computer in homes,  offices, or the local coffee house, bringing the "lab" to you
- Full access to a **dedicated computing resource **(some scheduling choices  include monitored root or administrator access). This access is the same or more  than what is possible in physical computing laboratory.
**Vendor-standard** remote access protocols and client software. Eliminates the need for specialized customization of one's own computer and eases updates and maintenance
**Platform agnostic **(Macs, Win, Linux, ...)
**Extensible** to any remotely-accessible desktop systems in specialized campus labs.  Departments can bring *their* lab to *their* students.
**Protection of Intellectual Property**
**Data Provenance and tracking**
**Fault-Tolerance**
**Higher Security**

## Issues (1)

- Communication Coupling (loose, tight, v. tight, code-level) and Granularity (fine, medium?, coarse)
- Communication Methods (e.g., ssh tunnels, xmprpc, snmp, web/grid services,etc.) – e.g., apparently poor support for Cray
- Storage issues (e.g., p-netcdf support, bandwidth)
- Direct and Indirect Data Flows (functionality, throughput, delays, other QoS parameters)
- End-to-end performance
- Level of abstraction
- Workflow description language(s) and exchange issues – interoperability
- “Standard” scientific computing “W/F functions”

## Issues (2)

- Problem is currently similar to old-time punched-card job submissions (long turn-around time, can be expensive due to front end computational resource I/O bottleneck)  - need up front verification and validation – things will change
- Back-end bottleneck due to hierarchical storage issues (e.g., retrieval from HPSS)
- Long term workflow state preservation - needed
- Recovery (transfers, other failures) – more needed
- Tracking data and files, provenances
- Who maintains equipment, storage, data, scripts, workflow elements? Elegant solutions my not be good solutions from the perspective of autonomy.
- EXTREMELY IMPORTANT!!! – We are trying to get out of the business of totally custom-made solutions.

## Slide 56