# Presentation: 258114

## Workflow Topics for the Next-Generation SDM-Center

- Ilkay Altintas
- altintas@**SDSC**.edu
- Bertram Ludäscher
- ludaesch@UC**DAVIS**.edu

- SciDAC SDM AHM
- Oct 5-6, 2005, NCSU
- Raleigh, NC

**Sir Walter Raleigh**

## Overview

**Kepler/SPA:**
- What we have (The GOOD)
- What we don’t (yet) have (The BAD)
- What we really need?? (The UGLY)
-  Things we might do; prioritization

## Macro Definitions ... 

**#define  KEPLER   KEPLER/SPA **
**#define  KEPLER   KEPLER*SPA **
**By the end: **
**#define  SPA   KEPLER****HPC**** **

## What we have – The GOOD

**Big Heritage from Ptolemy II**
- Vergil GUI for design and (some) execution monitoring
**Actor-Oriented Modeling & Design**
*Director* / *Actor* Separation
- Models of Computation: *PN*, *SDF*, *DE*, ..
- Nested Workflows & Hierarchical Modeling
- Research Results on Modeling Complex Systems
- modal models, mobile models, reconfig’able models, model lifecycle management, higher-order actors, ...
- head-start for **CCA Extensions**, *e.g.*
- SciRUN-2 Extensions (Steve P. *et al*.)
- Self-Managing, Dynamically-Adaptive, Autonomous Components (Manish *et al*.)

## What we have – The GOOD

**Kepler Extensions (to Ptolemy II)**
- Mostly: **loosely coupled, ***e.g.* WS (web service) workflows
- Many **generic actors**
- ssh, scp, cmd-line,SRB, Globus, ...
- new R expression actor
- Many **custom actors**
*e.g.* in PIW, TSI-1, TSI-2, GEON, SEEK, Resurgence, ...
- Several **ad-hoc extensions **& (initial) **research**, *e.g.*
- External job scheduling (*e.g.* NIMROD, ...)
- Director extensions (fault tolerance via WS “retry”)
- WF-Templates (structured combination of dataflow & control-flow: fault-tolerance, reusability)
- Higher-order functions (map/3, iterate-over-array, ... : simpler control-flow, optimization potential, ...)

## Some KEPLER Actors (out of 160+ ... and counting...)

**Some KEPLER Actors ****(out of 160+ ... and counting...)**

## What we have – The GOOD

**Kepler Extensions (Cont’d)**
- Some **generic extensions**
- Metadata-based (EML/ADN) Dataset Search
- Concept-based Actor Search (OWL)
- Documentation Framework
- Authentication & Authorization Framework (GAMA from GEON)
- Improved component/WF archival & plug-in (KAR,...)
- Provenance Recorder (“*Listener*”)
**PS        ... a growing open-source developers community ...**
**... ****and some scientific users ...  (TSI-1/2, PIW, GEON, SEEK, ... )**

## Concept-based Actor Search
Implemented as proof-of-concept
Additional operations slated for next Kepler Release (data search, port-based actor search, etc.)

Biggest Challenges
Building/searching a repository ...
Making changes to MoML (see KAR)
GUI changes
Ontology management

- Implemented as proof-of-concept
- Additional operations slated for next Kepler Release (data search, port-based actor search, etc.)
**Biggest Challenges**
- Building/searching a repository ...
- Making changes to MoML (see KAR)
- GUI changes
- Ontology management

**Concept-based Actor Search**

- Workflow
- Components
- (MoML/KAR)

- Ontologies
- (OWL)
*Default + Other*

- Semantic
- Annotations

*urn ids*

*instance*
*expressions*

## Slide 9

## The GOOD: Kepler Archives

**Purpose**: Encapsulate WF data and actors in an archive file
- ... inlined or by reference
- ... version control
-  More robust workflow exchange
-  Easy management of semantic annotations
-  Plug-in architecture (Drop in and use)
-  Easy documentation updates
- A jar-like archive file (.kar) including a manifest
- All entities have unique ids (LSID)
- Custom object manager and class loader
- UI and API to create, define, search and load .kar files

## KAR File Example

-  <entity name="Multiply or Divide" class="ptolemy.kernel.ComponentEntity">
- <property name="entityId"  value="urn:lsid:localhost:actor:80:1" class="org.kepler.moml.NamedObjId"/>
- <property name="documentation" **class="org.kepler.moml.DocumentationAttribute**"></property>
- <property name="class" **value="ptolemy.actor.lib.MultiplyDivide"** class="ptolemy.kernel.util.StringAttribute">
-   <property name="id" **value="urn:lsid:localhost:class:955:1**" class="ptolemy.kernel.util.StringAttribute"/></property>
- <property name="multiply" class="org.kepler.moml.PortAttribute">
-   <property name="direction" value="input" class="ptolemy.kernel.util.StringAttribute"/>
-   <property name="dataType" value="unknown" class="ptolemy.kernel.util.StringAttribute"/>
-   <property name="isMultiport" value="true" class="ptolemy.kernel.util.StringAttribute"/></property>
- <property name="divide" class="org.kepler.moml.PortAttribute">
-   <property name="direction" value="input" class="ptolemy.kernel.util.StringAttribute"/>
-   <property name="dataType" value="unknown" class="ptolemy.kernel.util.StringAttribute"/>
-   <property name="isMultiport" value="true" class="ptolemy.kernel.util.StringAttribute"/>
- </property>
- <property name="output" class="org.kepler.moml.PortAttribute">
-   <property name="direction" value="output" class="ptolemy.kernel.util.StringAttribute"/>
-   <property name="dataType" value="unknown" class="ptolemy.kernel.util.StringAttribute"/>
-   <property name="isMultiport" value="false" class="ptolemy.kernel.util.StringAttribute"/></property>
- <property **name="semanticType00" value="http://seek.ecoinformatics.org/ontology#ArithmeticMathOperationActor"**     class="org.kepler.sms.SemanticType"/>
- </entity>

## Kepler Object Manager

**Designed to access local and distributed objects**
**Objects: data, metadata, annotations, actor classes, supporting libraries, native libraries, etc. archived in kar files**
**Advantages:**
- Reduce the size of Kepler distribution
- Only ship the core set of generic actors and domains
- Easy exchange of full or partial workflows for collaborations
- Publish full workflows with their bound data
- Becomes a provenance system for derived data objects
- => Separate SPA workflow repository and distribution

## Provenance Framework 

**Provenance**
- Track origin and derivation information about scientific workflows, their runs and derived information (datasets, metadata...)
**Need for Provenance**
- Association of process and results
- reproduce results
- “explain & debug” results (via lineage tracing, parameter settings, ...)
- optimize: “Smart Re-Runs”
**Types of Provenance Information**:
- Data provenance
- Intermediate and end results including files and db references
- Process (=workflow instance) provenance
- Keep the wf definition with data and parameters used in the run
- Error and execution logs
- Workflow design provenance (quite different)
- WF design is a (little supported) process (art, magic, ...)
- for free via cvs: edit history
- need more “structure” (e.g. templates) for individual & collaborative workflow design

## Kepler Provenance Recording Utility

**Parametric and customizable **
- Different report formats
- Variable levels of detail
- Verbose-all, verbose-some, medium, on error
- Multiple cache destinations
**Saves information on**
- User name, Date, Run, etc...

**Joint work with Oscar Barney**

## Provenance: Next Steps

- .kar file generation, registration and search for provenance information
- Possible data/metadata formats
- Automatic report generation from accumulated data
- A relational schema for the provenance info in addition to the existing XML
- Smart re-runs

## The Future

**From GOOD via BAD to UGLY**
**The good news (about ‘bad’ and ‘ugly’)**
- Lots of interesting challenges!
- ... so ‘ugly’ is actually good!

## What we don’t (yet) have ... THE BAD

**Much is still to do (or still ongoing)**
- Detached execution
- many options; depend on requirements
- Kepler WF repository w/ dynamic actor plug-in
- Smart Reruns
***avoid*** doing (old) work twice
- Smarter Reruns (too smart?)
***reuse*** previous results for speed-up of (new) work
- NIMROD Director, CONDOR Director ...
- Task manager / monitor
- Support for WF design & reuse
- Semantic extensions
- “Design Patterns”, Templates

## What we don’t have ... THE BAD cont’d

**Vertical SDM Integration**
- Workflow layer could be used to embed other SDM components and glue them together
- Scope & Architecture unclear
- Data Mining tools  new WF actors
- Parallel-R  new WF actors !?
- SEA, Bitmap tools  new !?
- MPI-IO  alternative to current Kepler data access!?
- ...
- Not only a technical problem
- e.g. need for driving use-cases that require combination of several SDM layers together

## Challenges

**Easier said ...**
- “We’re not going to reinvent the wheel ...”
- “We just use XYZ ...”
- XYZ in {CCA, HDF5, PnetCDF, Ccaffeine, Condor, MPI-IO, parallel-R, ...}
**... ****than done ...**
- Incompatible, isolated solutions and frameworks
- Can’t use workflow/actor/director A with B
**Coming up with a coherent, overall architecture is hard!**

## HTC Example (using: NIMROD)

** ****need to make Kepler NIMROD/Condor/... “aware”**
** ****similar need for HPC support **

## Another Distribution Approach

**Another Distribution Approach**

**Client**

**Servers**

**Computer Network**

**Service Locator****(Peer Discovery)**

**Simulation is orchestrated in a centralized manner**

** ****Source: Daniel Lázaro Cuadrado, Aalborg University **

**Notes:**
The implementation of the distributed sdf domain and the distributed platform required for the distributed execution follows a client server approach.
Given a computer network, we can have a client that runs the distributed sdf model.
The server computers have to run a server application in order to become a part of the distributed platform.
In between we need a peer discovery protocol that will allow the client to find the servers over the network.
For our implementation we have chosen Jini for this purpose.
A service locator service has to be running somewhere for the servers to register and for the client to locate the servers.
So the idea is to send the different actors over the network and interconnect them over the network the same way as the topology of the model shows.
Once this is done the client can orchestrate the simulation in a centralized manner sending commands over the network.

## What we don’t have ... THE UGLY

**Workflow Design & (Re-)Usability**
- Difficult Marriage of Dataflow and Control-flow
- e.g. PIW, TSI-1/2, GEON-A-type-WF, ...
- WF development, deployment, maintenance, use
- from (Mess...) to Art to **Commodity**** **
- ( next presentation)
- support for WF whole life-cycle
**Fault Tolerance**
- current embedding of control-flow into dataflow yields to **non-maintainable workflows!**
**Close Coupling of Components for HPC **
- CCA-style
- MPI-style
- Memory-to-Memory (on single nodes)
- large, efficient data transfer
- ...

## WF-Design: Adapters for Semantic & Structural Incompatibility

**Adapters may:**
- be abstract (no impl.)
- be concrete
- bridge a semantic gap
- fix a structural mismatch
- be generated automatically (e.g., Taverna’s “list mismatch”)
- be reused components(based on signatures)

- C1

- C1

- D1

- C1

- C2

- C

- D

- C

- C

- D

- D

- D

- D

- C2

- C2

- D2

*f**2*

*f**1*

- [S]

- S

- T

- [S]

- [S]

*f**1*

- [T]

*f**2*

*map*

*f**2*

*f**1*

- [[S]]

- S

- T

- [[S]]

- [[S]]

*f**1*

- [[T]]

*f**2*

*map*

*map*

## Additional Design Primitives for Semantic Types

*Extended Transformations*

*Starting Workflow*

*Resulting Workflow*

*t*9: Actor Semantic Type Refinement (*T*     *T*)

*T*

*t*12: I/O Constraint
- Strengthening
- (*** * *** *)

*t*10: Port Semantic TypeRefinement
- (C     C, D     D)

- C

*t*14: Adapter Insertion

*T***

*t*11: AnnotationConstraint Refinement
- (**  **)

- s

- C

***1*

**

*t*15: Actor Replacement

*f*

*f***

*t*16: Workflow Combination
- (Map)

*t*13: Data Connection
- Refinement

**

*Resulting Workflow*

- D

- C

- D

- C

- D

- t

- D

***2*

***1*

- t

- D

***2*

- s

- C

***1*

- t

- D

***2*

- s

- C

## Workflow Design Primitives

**Workflow Design Primitives**

**End-to-End Workflow Design and Implementation**
- Viewed as a series of primitive “transformations”
- Each takes a WF and produces a new WF
- Can be combined to form design “strategies”

*W*0

*t*

*W*1

*W*2

*W*m

*W*n

- ...

*t*

*t*

*Workflow*
*Design*

*Workflow*
*Implementation*

*Top-Down*

*Bottom-Up*

*Input Driven*

*Output Driven*

*Structure Driven*

*Semantic Driven*

*Task Driven*

*Data Driven*

## Fault Tolerance & Maintenance Challenges

## Workflow Templates and Patterns

*** ******New Ingredients ***

*** ******Proposed Layered Architecture  ***

***work w/ Anne Ngu, Shawn Bowers, Terence Critchlow ***

## Use Ideas from Fault Tolerant Shell

** ****Source: Douglas Thain, Miron Livny **
***The Ethernet Approach to Grid Computing ***

***Good ideas in ftsh; ***
***some might be (semi-)low hanging fruits for Kepler ... ***

## Kepler Coupling Components & Codes 

**Types of Coupling ...**
**Loosely coupled** (“1st Phase”)
- Web Services (SPA, GEON, SEEK, ...),
- ssh actors, ..
- + reusability (behavorial polymorphism)
- + scalability (# components)
- – efficiency
**Tight(er) coupling** (“2nd Phase”)
- Via **CCA** (SciRUN-2, Ccaffeine, ...)            (Cipres uses CORBA)
**HPC needs: code-coupling as efficient & flexible as possible **(e.g. Scott’s challenges...)
- memory-to-memory (single node or shared memory),
- MPI (multiple-nodes)
- optimizations for transfer of data & control (streaming, socket-based connections)

## Accord-CCA: Ccaffeine w/ Self-Managed Behavior 

- Source: Hua Liu and Manish Parashar

- cf. w/ mobile models, reconfiguration in Ptolemy II
* *
*... **begging for a Kepler design and implementation ...*

## Different “Directors” for Different Concerns

**Example: **
- Ptolemy Directors – “factoring out” the concern of workflow “orchestration” (MoC)
- common aspects of overall execution **not** left to the actors
**Similarly: **
- “Black Box” (“flight recorder”)
- a kind of “recording central” to avoid wiring 100’s of components to recording-actor(s)
- “Red Box” (error handling, fault tolerance)
-  use ftsh ideas; tempaltes
- “Yellow Box” (type checking)
- for workflow design
- “Blue Box” (shipping-and-handling)
- central handling of data transport (by value, by reference, by scp, SRB, GridFTP, ...)
- “CCA++ Boxes”
- Change behavior (e.g. algorithm) of a component
- Change behavior (i.e., wiring) of a workflow in-flight

-  SDF/PN/DE/...

- Provenance
-  Recorder

- SHA @

-  Static Analysis

-  On Error

**Component Mgr**

**  ****Composition Mgr  **

## Summary

**The GOOD: **
- lots to build upon
**The BAD: **
- no common / integrated architecture
-  use Kepler/SPA as a glue
-  this might be harder than it sounds
-  needs a mix of end-to-end application-drive and serious design effort for the integration architecture
**The UGLY: **
- HPC challenges: close coupling, fault tolerance, ...
- The good news: there’s work to be done!

## Use of Semantics in SWF... 

**“****Smart” Search**
- Concept-based, e.g., “find all datasets containing biomass measurements”
**Improved Linking, Merging, Integration**
- Establishing links between data ***through*** semantic annotations & ontologies
- Combining heterogeneous sources based on annotations
- Concatenate, Union (merge), Join, etc.
**Transforming**
- Construct mappings from schema S1 to S2 based on annotations
**Semantic Propagation**
- “Pushing” semantic annotations through transformations/queries

## Helping with “shims” / adapters

- Services can be ***semantically compatible***, but ***structurally incompatible***

*Source*
*Actor*

*Target*
*Actor*

*P*s

*P*t

**Semantic****Type *****P******s***

**Semantic****Type *****P******t***

- StructuralType *P**t*

- StructuralType *P**s*

**Desired Connection**

**Incompatible**

**Compatible**

- (⋠)

- (**⊑**)

- (*P**s*)

- 

*Ontologies (OWL)*

**Source: [Bowers-Ludaescher, DILS’04]**