# Presentation: 928238

## Experience with Fusion Workflows

**Norbert Podhorszki, **Bertram Ludäscher
- Department of Computer Science
- University of California, Davis

## New Challenges

**The CPES project brought new challenges for Kepler and workflow automation people**
**Remote computations, services and tools **
**Long running simulations, large amounts of data**
**One-time-passwords**
**Workflow = “Glue” **
**Scientists only need to connect individual components together**
**Automate tedious processes (logins, copies of data, control, start-stop)**
**Do it reliably**
**Show what is going on**

**Notes:**
...basically we face here that many things already exist, they are used, but it becomes tedious to orchestrate those many things to perform the additional little tasks around a large simulation. We are expected to give support for that...

## Workflows

**Real-time Monitoring of Simulation:**
**Transfer current data set to a secondary resource**
**Execute short analysis/visualization routines**
**Display result**
**Archival and post-processing**
**Transfer, pack and archive data ****sets on the fly**

**Notes:**
.. .this is what the scientist want to have...
Some reasons besides the manually cumbersome procedure:
- good to see what the simulation is doing. A bad simulation costs the same as a good one, so stopping it early pays off.
- archival is always painful, but how painful it is, when you generate so much data, that it cannot be stored alltogether on the local system till the end of simulation (quota, filesys problems).

## Kepler actors for CPES

**Job submission to various resource managers**
**Permanent SSH connection to perform tasks on a remote machine**
**Generalized actors (workflows themselves) for specified tasks:**
**Watch a remote directory for simulation timesteps**
**Execute an external command on a remote machine**
**Tar and archive data in large junks to HPSS**
**Transfer a remote image file and display on screen**
**Control a running SCIRun server remotely**
**Above actors do logging/checkpointing**
**the final workflow can be stopped / restarted**

**Notes:**
...Enumeration of our beautiful accomplishments...

## Archival Workflow

*Monitor*

**Archival Workflow**

**Plasma physics simulation on 2048 processors on Seaborg@NERSC (LBL)**
**Gyrokinetic Toroidal Code (GTC) to study energy transport in fusion devices (plasma microturbulence)**
**Generating 800GB of data (3000 files, 6000 timesteps, 267MB/timestep), 30+ hour simulation run**
**Under workflow control:**
**Monitor (watch) simulation progress (via remote scripts)**
**Transfer from NERSC to ORNL concurrently with the simulation run**
**Convert each file to HDF5 file**
**Archive files to 4GB chunks into HPSS**

## Monitoring Workflow

**Monitoring Workflow**

**Notes:**
The actors developed for the fusion project are mature enough for their application in other disciplines. At the SC06 in Florida, twenty people created the monitoring workflow, which transferred simulation data timestep-by-timestep, converted, archived and created images, transferred and displayed the images to the windows desktop.

## Future Plans

**Currently we have specialized actors that should be generalized for other disciplines and systems**
**“****watching for” simulation output**
**safe and robust transfer, recovery from failure**
**archiving to different MSS, with different security policies, robust to failures and maintenance periods**
**Next workflow is cyclic, not just streaming**
**couple two simulations on two resources, transfer data and control between them**
**use local job manager for code execution**
**What about provenance management?**
**main reason to use scientific workflow system e.g. in bioinformatics workflows – needed for debugging runs, interpreting results, etc. **

**Notes:**
The Kepler actors are specifically implemented for the system and circumstances on the fusion project, like OTP passwords, bbcp for transfer, HPSS mass storage with direct hsi access, and the specification of the simulation code, like large number of timesteps, write-once files, nicely named with timestep numbers. 

For other systems or simulations, requirements surely differ just a bit, resulting in a desperate workflow developer's life: what if the security model is different (e.g. credentials), other mass storage system is available, data transfer fails regularly, permanent remote connections are restricted, the simulation overwrites its data files in several steps, naming is not consistent, there is not enough space on one of the resources, etc. We want to generalize the existing actors to be more flexible so that the workflow developer can tailor them to the actual circumstances. 

The fusion project aims to couple two different simulations to make the computation more precise and robust. We need to transfer data forth and back between the two simulations running on two different systems, and also control them. The automated workflow thus, will be a cyclic workflow, where timing and robustness are even more important than in the previous workflow.

## There is more, e.g., how to get from messy to neat & reusable designs?

## The Answer (YMMV)

**The Answer (YMMV)**

**Collection-Oriented Modeling & Design (COMAD)**
**embrace an **assembly line metaphor
**data = **tagged** **nested** **collections
**e.g. represented as flattened, **pipelined token streams:

**Notes:**
YMMV = Your Mileage May Vary ... (ie what works for us might not work for you..)