# Presentation: 177030

## Tutorial: Introduction to the Virtual Data Language

- Summer Grid 2004
- UT Brownsville South Padre Island Center
- 24 June 2004
- Presented by: Mike Wilde
- Argonne National Laboratory
- Mathematics and Computer Science Division
- Developed by: Jens Voeckler
- The University of Chicago
- Department of Computer Science

## Outline

- Install the Virtual Data System
- Simple workflows:
- Get (some) user apps to run with *shplan*
- Euryale – yet another concrete planner
- Pegasus = Planning and Execution in grids

**Notes:**
Install= run test-version script.

## Components of GVDS

- The Chimera system for managing virtual data products
- Virtual data: materialize data on-demand
- Virtual data language, catalog and interpreter
- The Pegasus system for planning and execution in grids
- Pegasus is a configurable system that can map and execute complex workflows on grid resources
- Contributed packages:
- Visualizers, shell planner, Euryale

## Prepare GVDS Installation

- Log onto your Linux system.
- Ensure Java version 1.4.*
- Type "$JAVA_HOME/bin/java –version"
- If necessary, download from _[http://java.sun.com/j2se/1.4/](http://java.sun.com/j2se/1.4/)_
- Set JAVA_HOME environment variable
- Points to Java installation directory
- Example: setenv JAVA_HOME /usr/lib/java

**Notes:**
Some developers are able to use cygwin runtime environment. There are issues with the path setup that are not easily portable between Unix, cygwin and Windows. You have been warned.

## Installation

- Download the latest tar-ball_[http://www.griphyn.org/workspace/VDS/snapshots.php](http://www.griphyn.org/workspace/VDS/snapshots.php)_
- The binary release is smaller
- The source release allows patching
- Let's try the source release for now...
- Unpack
*gtar xvzf vds-source-1.2.5.tar.gz*
* **cd vds-1.2.5*
- Use the latest version.

**Notes:**
Also comes as part of VDT 1.1.14.

## Starting

- Source necessary shell script (all releases)
- Explicit setting of *VDS_HOME=`/bin/pwd`* before sourcing **required**
- Bourne: *. set-user-env.sh*
- C-Shell: *source set-user-env.csh*
- Run *test-version* program to check
- Script is a small sanity test
- If it fails, Chimera will not run!
- Developer mode will fail to find gvds.jar – no worry.

**Notes:**
Developers should instead source the appropriate set-devel-env.sh or set-devel-env.csh scripts. 
If you are changing between user mode and developer mode, unset the CLASSPATH variable before sourcing the correct shell script.
The difference between user mode and developer mode is that the user mode includes chimera.jar into the CLASSPATH variable, whereas in developer mode, the CLASSPATH points to the start of the source hierarchy instead.  All scripts append the PATH with $VDS_HOME/bin and MANPATH. 
In order to run Chimera in developer mode, one must compile all classes successfully, e.g. use "ant jar" without error.

## Sample test-version Script Output

- # checking for JAVA_HOME
- # checking for CLASSPATH
- # checking for VDS_HOME
- # starting java
- # Using recommended version of Java, OK
- # looking for Xerces
- # found /home/voeckler/vds/lib/xercesImpl.jar, OK for now
- # found /home/voeckler/vds/lib/xmlParserAPIs.jar, OK for now
- # looking for antlr
- # found /home/voeckler/vds/lib/antlrall.jar, OK for now
- # looking for GNU getopt
- # found /home/voeckler/vds/lib/java-getopt-1.0.9.jar, OK for now
- # looking for JDBC driver
- # found /home/voeckler/vds/lib/pg73jdbc3.jar, OK for now
- # found /home/voeckler/vds/lib/mysql-connector-java-...
- # looking for my replica catalogs
- # found /home/voeckler/vds/lib/rls.jar, OK for now
- # looking for myself
- # in developer mode, ignore not finding gvds.jar
- # found /home/voeckler/vds/lib/gvds.jar, OK for now

**Notes:**
In developer mode, the last step may fail, as there is no chimera.jar in the CLASSPATH. That is o.k.

## For Support and Problems

- Mailing lists
- _[chimera-](mailto:chimera-announce@griphyn.org)__[announce@griphyn.org](mailto:chimera-announce@griphyn.org)_
- Few emails; announcements only  [closed list]
- _[chimera-support@griphyn.org](mailto:chimera-support@griphyn.org)_
- Errors reports [open list]
- _[chimera-discuss@griphyn.org](mailto:chimera-discuss@griphyn.org)_
- Features, futures, discussion, thoughts [open list]
- Bugzilla website
- _[http://](http://bugzilla.globus.org/chimera/)__[bugzilla.globus.org](http://bugzilla.globus.org/chimera/)__[/chimera/](http://bugzilla.globus.org/chimera/)_

**Notes:**
It's not always Chimera's fault!

## Working with Virtual Data

- VDLt

- vdlt2vdlx

- VDLx

- vdlx2vdlt

- ins/upd

- VDLx

- VDC

- gendax

- DAX

- Only the abstract planning process is shown

- search

**Notes:**
VDLt: textual version of the virtual data language
VDLx: XML version of the virtual data language
DAX: abstract directed acyclic graph description
VDLt is slightly less expressive than VDLx, thus the initial back-and-forth conversion between the formats might lose some non-essential information. Any further round will yield equivalent results.

## The Virtual Data Languages

- VDLt is textual
- Concise
- For human consumption
- Usually for (TR) **transformation**
- Process by converting to VDLx

- VDLx uses XML
- Uses XML-Schema
- For generation from scripts
- Usually for (DV)**derivations**
- Storage representation of current VDDB

**Notes:**
The database format will improve over the current textual file version.

## VDL: Virtual Data LanguageDescribes Data Transformations

- Transformation
- Abstract template of program invocation
- Similar to "function definition"
- Derivation
- “Function call” to a Transformation
- Store past and future:
- A record of how data products were generated
- A recipe of how data products can be generated
- Invocation
- Record of a Derivation execution

- TR

- DV

- PTR

**Notes:**
Derivations are "named calls". Future versions may use more unnamed calls, and have naming an option, not a mandate.

## Example Transformation

- TR tut::t1( out a2, in a1, none pa = "500", none env = "100000" ) {
-   argument = "-p "${pa};
-   argument = "-f "${a1};
-   argument = "-x –y";
-   argument stdout = ${a2};
-   profile env.MAXMEM = ${env};
- }

- $a1

- $a2

- tut::t1

## Example Derivations

- DV tut::d1->tut::t1 (env="20000", pa="600",a2=@{out:run1.exp15.T1932.summary},a1=@{in:run1.exp15.T1932.raw},
- );
- DV tut::d2->tut::t1 (a1=@{in:run1.exp16.T1918.raw},a2=@{out.run1.exp16.T1918.summary}
- );

## Hello World

- Exercise 1:
- Wrap *echo 'Hello world!'* into VDL
- Start with the transformation
- TR tut::hw( output file ){argument = "Hello world!"; 	argument stdout = ${file}; }
- Then "call" the transformation
- DV tut::hello->tut::hw(file=@{output:"out.txt"} );
- Save into a file *hw.vdl*

**Notes:**
"world" is the name of the transformation. This is a logical name. 
"hello" is the name of the derivation (left of arrow), which calls "hello" (right of arrow).
Parameter matching is done on name (like Python), but expected to take position into account in future versions.
Every argument may be named. Only the special names "stdin", "stdout" and "stderr" refer to the streams to capture from or into a file.
All file names are logical.
Where now does the echo application come into the game?
And what about the logical file out.txt ?

## What to do with it?

- All VDLt must be converted into VDLx before it becomes "usable" to the VDS.
*vdlt2vdlx hw.vdl hw.xml*
- All VDLx must be stored into your VDDB, before the system can work with it
*insertvdc hw.xml*
- or
*updatevdc hw.xml*

- But where did it go?

**Notes:**
Insert will add to the database, but does not overwrite existing definitions
Update will add to the database, adding new items if necessary, or updating existing items. Updates work by removing possibly existing definitions, and adding them again. Thus, insertion is faster, and should be preferred, if possible. 
Tip: Build VDL-generators to generate VDLx to save the conversion step.

## Properties I

- The GVDS is configured by properties
- Prefix "vds."
- Best documentation in $VDS_HOME/etc/sample.properties
- Global properties in$VDS_HOME/etc/properties
- User-local properties (overwrites) in$HOME/.chimerarc
- Command-line properties highest prio-Dvds.some.key=value

**Notes:**
Settings in the local user properties can overwrite setting in the global properties. Command-line properties take precedence over both. 
If command-line properties are specified, they must be the first argument(s). 

## VDC storage options

- Support for flat files...
- For simple, out-of-the-box test use
- Default, suggested maximum 1000
- vds.db.**vdc**.schema = SingleFileSchema
- vds.db.vdc.schema.file.store = <flatfile xml>
- #vds.db.vdc.schema.xml.url = <VDLx xsd>

**Notes:**
Could/should be turned into a web service.

## VDC storage options II

- ... and relational database systems
- For production systems
- PostGreSQL 7.3.* (and 7.4.*)
- vds.db.*.driver = Postgres
- MySQL 4.1.*
- vds.db.*.driver = MySQL
- Permits spreading of unrelated catalogs across DBs
- vds.db.<schema1>.driver = Postres
- vds.db.<schema2>.driver = MySQL

**Notes:**
Could/should be turned into a web service.
Toying with Oracle support.
The * is a stand-in to mean "all schemas" that are not handled explicitely.

## VDC storage options III

- rDBMS requires schema setup
- $VDS_HOME/sql contains schema files.
- $VDS_HOME/doc user guide for setup.
- and properties per rDBMS
- The usual suspects include
- vds.db.<schema>.driver.url = jdbc:...
- vds.db.<schema>.driver.user = ${user.name}
- vds.db.<schema>.driver.password = ${user.name}

**Notes:**
Could/should be turned into a web service.

## VDC schema flavors

- ChunkSchema
- Simple, fast
- Current default

- AnnotationSchema
- Based on chunk schema
- Permits annotations
- Future default

- InvocationSchema
- Currently optional
- Future required
- Tracks provenance

- either/or

**Notes:**
You must chose either chunk or anno schema. Both will work with the InvocationSchema. 

## VDC schema storage options

- Schemas also require properties
- Simple chunk schema
- vds.db.**vdc**.schema = ChunkSchema
- Annotated schema
- vds.db.**vdc**.schema = Annotationchema
- Provenance Tracking Catalog (PTC)
- vds.db.**ptc**.schema = InvocationSchema

**Notes:**
Could/should be turned into a web service.

## So, Where did it go?

- You can check with your VDDB
*searchvdc –n tut *
*2004.05.14 ... [app] searching the database*
*tut::hw*
*tut::hello->tut::hw*
- Output and search options
- List view, VDLt view, VDLx results
- Search by namespace, definition type, LFN
- Etc.

**Notes:**
Try "searchvdc –h" (future: --help) for quick help on search options. Or refer to the manpage.

## From VDC to DAX

- Deriving the provenance of a logical file or derivation is the *abstract planning* process.
- Abstract DAG in XML = DAX
- All files and transformations are logical!
- The complete provenance will be unrolled
- No external catalogs (TC,RC) are queried

**Notes:**
The label "-l" is used later to identify the master instance (DAG file, master shell script, etc.) to invoke later. You should always use a label. Otherwise, it defaults to "test". 

## From VDC to DAX

- Ask the catalog for the produced file
*gendax –l hw –o hw1.dax –f out.txt*
- Or ask for the derivation
*gendax –l hw –o hw2.dax –n tut –i hello **or*
*gendax –l hw –o hw3.dax –D tut::hello*

**Notes:**
The label "-l" is used later to identify the master instance (DAG file, master shell script, etc.) to invoke later. You should always use a label. Otherwise, it defaults to "test". 

## The DAX file

- Result of the Chimera **abstract planner**.
- Richer than Condor DAGMan format.
- Expressed in terms of logical entities.
- Contains complete (build) lineage.
- Consists of three major parts
- All logical files necessary for the DAG.
- All jobs necessary to produce all files.
- The dependencies between the jobs.

**Notes:**
DAX format is projected to change. The sections order may change next.

## Output hw1.dax

- <?xml version="1.0" encoding="UTF-8"?>
- <!-- generated: 2004-05-14T15:21:38-05:00 -->
- <!-- generated by: voeckler [??] -->
- <adag xmlns="http://www.griphyn.org/chimera/DAX" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.griphyn.org/chimera/DAX http://www.griphyn.org/chimera/dax-1.8.xsd" version="1.8" count="1" index="0" name="hw">
- <!-- part 1: list of all files used (may be empty) -->
-   <filename file="out.txt" link="output"/>
- <!-- part 2: definition of all jobs (at least one) -->
-   <job id="ID000001" namespace="tut" name="hw" level="1" dv-namespace="tut" dv-name="hello">
-     <argument>Hello World! </argument>
-     <stdout file="out.txt" link="output" varname="file"/>
-     <uses file="out.txt" link="output" dontRegister="false" dontTransfer="false"/>
-   </job>
- <!-- part 3: list of control-flow dependencies (empty for single jobs) -->
- </adag>

## Making a DAX Runnable

- Map logical entities to physical entities.
- Require external services for mapping:
- Replica Catalog (RC)
- Maps LFN to PFN
- and thus TFN, SFN
- Resource (pool) Catalog (PC)
- Contains site-specific configurations
- Transformation Catalog (TC)
- Maps transformation to application

## The Transformation Catalog

- Translates logical transformation into an application specific for a certain pool
- Default: Simple, text based file
- 3+ columns
- Blank lines and comments are ignored
- Standard location
- $VDS_HOME/var/tc.data
- adjustable with property vds.tc.file
- New: Pegasus will use an rDBMS
- available soon

**Notes:**
$VDS_HOME/etc/properties contains all user properties for VDS. Please refer to the user guide for details on available properties and their effects. 

## Transformation Catalog Columns

- 1st column is the pool handle
- Shell planner only uses the "local" handle
- Other handles for concrete planners
- 2nd column is the logical transformation
- Format: <ns>_ _ <name>_<version>
- Name-only names are just <name>
- 3rd column is the path to the executable
- 4th column for environment variables
- Use "null" if unused
- Format: key=value;key=value

**Notes:**
Quirk: If you use ns or vs, you must specify both – for Pegasus.
Support for the fully-qualified transformation name may be lacking in some concrete planners.
Examples are strewn over many directories.

## The Shplan Replica Catalog

- A replica catalog translates a logical filename into a set of physical filenames
- This RC is for the shell planner only
- Production-strength RC is RLS-based.
- Simple textual file
- Three columns
- Blank lines and comments are ignored
- Standard location
- $VDS_HOME/var/rc.data
- Adjustable with property vds.db.rc

**Notes:**
$VDS_HOME/etc/properties contains all user properties for VDS. Please refer to the user guide for details on available properties and their effects. 

## Shell planner RC Columns

- 1st column is the pool handle
- Shell planner only uses the "local" handle
- Other handles for concrete planner
- 2nd column is the logical filename
- A LFN may contain slash etc.
- 3rd column is the path to the file
- Are multiples allowed?
- No, only the last (first?) match is taken

## TC and RC Short-cuts

- Application locations can come from VDL
- hints.pfnHint takes precedence over TC
- pfnHint is a **deprecated** feature!
- Shell planner may run w/o any TC
- Files need not be registered with RC
- Existence checks are done in file system
- RC updates can be optional
- Shell planner may run w/o any RC

-  After all, we run locally with the shell planner!

**Notes:**
Using application locations in VDL is not recommended, as the specific location of an application may change from pool to pool. 

## Preparing to Run The Shell Planner

- Make sure that your TC contains a translation for logical transformation "hw"
- local	tut__hw	/bin/echo	null
- Make sure that there is a RC **without** weird content:
*cp /dev/null $VDS_HOME/var/rc.data*

**Notes:**
The distributed rc.data may contain some weird content, sorry!

## Running The Shell Planner

- Run the shell planner
*shplanner –o hw hw1.dax*
- Check directory "hw"
- Master script: <DAX-label>.sh
- Job scripts: <TR>_<DAX-JobID>.sh
- Helper files: <TR>_<DAX-JobID>.lst

**Notes:**
Ignore the slightly verbose output for now

## Running The Shell-Plan

- Shell planner generates a master plan
- This is a tool-kit  no auto-run feature
- Run the master plan
*( cd hw && ./hw.sh )*
- Can check log file for status etc.
- Standard name: <DAX-label>.log
- Master script will exit with 0 on success
- Exit 1, if any sub-script fails.

## Convert A Unix Pipeline

- Example 2: Full name from a username
*grep ^user /etc/passwd | awk –F: '{ print $5 }'*
- Create abstract transformations
- Create concrete derivations to call them
- Convert into VDLx
- Add to VDDB
- Run abstract planner
- Run shell planner
- Run master script

## Workflow from File Dependencies

- TR tut::tr1(in a1, out a2) {
-     argument stdin = ${a1};
-     argument stdout = ${a2}; }
- TR tut::tr2(in a1, out a2) {
-     argument stdin = ${a1};
-     argument stdout = ${a2}; }
- DV tut::x1->tut::tr1(a1=@{in:file1}, a2=@{out:file2});
- DV tut::x2->tut::tr2(a1=@{in:file2}, a2=@{out:file3});

- file1

- file2

- file3

- tut::x1

- tut::x2

**Notes:**
Explicit dependencies.

## Abstract Transformations

- Grep for an arbitrary user name
- TR tut::grep( none name, output of ){ argument = "^"${name}" /etc/passwd";argument stdout = ${of}; }
- Extract a full name from a line
- TR tut::awk( input line, output full ){ argument stdin = ${line};argument stdout = ${full};argument = "-F: '{ print $5 }' "; }

**Notes:**
DO NOT call your files any of the reserved words none, in, input, inout, io, out, or output. Currently, the standard vdlt2vdlx also does not allow these reserved words as prefix to any name, but we intend to fix that.
Some tricks here. The /etc/passwd, albeit a file, is handled like a constant. Thus, it will be untracked. For most Unix systems, this is ok. 
The script to awk is presented on the commandline to awk. Thus, its quotes must be quoted for correct usage. Please note that the first example might run into troubles, too, if certain things are used w/o quoting.

## Concrete Derivations

- Run with a concrete username...
- DV tut::d2->tut::grep( name="voeckler", of=@{output:"grepout.txt"} );
- ... and post-process the results
- DV tut:: d3-> tut:: awk( line=@{in:"grepout.txt"}, full=@{out:"awkout.txt"} );
- The two derivations are linked by a LFN
- "grepout.txt" is produced by "d2"
- "grepout.txt" is consumed by "d3"

## Convert, Insert and Plan

- Convert into VDLx
*vdlt2vdlx ex2.vdl ex2.xml*
- Insert into your VDDB
*insertvdc ex2.xml*
- Run abstract planner
*gendax –l ex2 –o ex2.dax –f awkout.txt*

## Run Shell Planner

- Prepare transformation catalog
- local	tut__grep	/usr/bin/egrep	null
- local	tut__awk	/opt/gnu/bin/gawk	null
- Run shell planner
*shplanner –o ex2 ex2.dax*
- Run master plan
*( cd ex2 && ./ex2.sh )*

## Kanonical Executable for Grids

- Simple application
- Copies all input with indentation to all output files
- Adds additional data about itself
- Allows tracking
- What ran where, when, which architecture, and how long
- To be used as stand-in for real applications
- Allows to check the DAG stages

**Notes:**
Run "keg –o /dev/fd/1"

## The "black diamond" WF

- Complex structure
- Fan-in
- Fan-out
- "left" and "right" can run in parallel
- Uses input file
- Register with RC
- Complex file dependencies
- Glues workflow

**findrange**

**findrange**

**analyze**

**preprocess**

**Notes:**
We deal with "regular diamonds" (as in the big dag), and "black diamonds". Regular diamonds use a generating top node, and don't have cross-dependencies. 
The processes are neither "findrange" nor "right". "findrange" is the TR name, "right" the DV name. The application (and thus process) is most likely "keg".

## Workflow step "preprocess"

- TR preprocess turns f.a into f.b1 and f.b2
- TR tut::preprocess( output b[], input a ) {argument = "-a top";argument = " –i "${input:a};argument = " –o " ${output:b};
- }
- Makes use of the "list" feature of VDL
- Generates 0..N output files.
- Number file files depend on the caller.

## Workflow step "findrange"

- Turns two inputs into one output
- TR tut::findrange( output b, input a1, input a2, none name="findrange", none p="0.0" ) {argument = "-a "${name};argument = " –i " ${a1} " " ${a2};argument = " –o " ${b};argument = " –p " ${p};
- }
- Uses the default argument feature

## Can also use list[] parameters

- TR tut::findrange( output b, input a[],none name="findrange", none p="0.0" ) {argument = "-a "${name};argument = " –i " ${" "|a};argument = " –o " ${b};argument = " –p " ${p};
- }

## Workflow step "analyze"

- Combines intermediary results
- TR tut::analyze( output b, input a[] ) {argument = "-a bottom";argument = " –i " ${a};argument = " –o " ${b};
- }

## Complete VDL workflow

- Generate appropriate derivations
**DV** tut::top**->**tut::preprocess**(** b=[ @{out:"f.b1"}, @{out:"f.b2"} ], a=@{in:"f.a"} **);**
**DV** tut::left**->**tut::findrange**(** b=@{out:"f.c1"}, a2=@{in:"f.b2"}, a1=@{in:"f.b1"}, name="left", p="0.5" **);**
**DV** tut::right**->**tut::findrange**(** b=@{out:"f.c2"}, a2=@{in:"f.b2"}, a1=@{in:"f.b1"}, name="right" **);**
**DV** tut::bottom**->**tut::analyze**(** b=@{out:"f.d"}, a=[ @{in:"f.c1"}, @{in:"f.c2"} **);**

**Notes:**
Filenames provide the glue.
findrange is called multiple times.

## The Diamond DAG II

- The "black diamond" takes one input file
*echo –e "local\tf.a\t${HOME}/f.a" **» $VDS_HOME/var/rc.data*
*date > ${HOME}/f.a*
- It uses three different transformations
*echo –e "local\ttut__preprocess\t$VDS_HOME/bin/keg\tnull" » $VDS_HOME/var/tc.data*
*echo –e "local\ttut__findrange\t$VDS_HOME/bin/keg\tnull" **» $VDS_HOME/var/tc.data*
*echo –e "local\ttut__analyze\t$VDS_HOME/bin/keg\tnull" **» $VDS_HOME/var/tc.data*

## Interlude: LFN

- Historically in two flavors:
- Staged and tracked (regular) files
**@{***dir***:"***lfn***"}**
- Unstaged and untracked (temporary) files
**@{***dir***:"***lfn***":"***tmp***"}**
- Now with option flags for finer control:
**@{***dir***:"***lfn***"** [**:"***tmp***"**] [**|***flags*]**}**
-  **@{***dir***:"***lfn***"|rt}**
-  **@{***dir***:"***lfn***":"***tmp***"|}**

## Immediate Future LFNs

- Optional file transfers
- Don't fail a transfer is a file is missing.
- Prototypical new transfer tool working.
- VDLt: Since 1.22: @{lfn|T} flag
- Optional files
- Don't fail abstract job, if a file is missing.
- VDLt: Since 1.23: @{lfn|o} flag

## A Big Example

- Overall tree structure
- 70 "regular" diamond DAGs on top
- 10 collecting nodes
- 1 final collector
- 291 jobs to run
- All files are generated, no RC preloading
- Add "multi" TR to last example
*echo –e "local\ttut__multi\t$VDS_HOME/bin/keg\tnull" » $VDS_HOME/var/tc.data*

## Compound Transformations

- Using compound TR
- Composition of complex TRs from basic ones
- Calls are independent
- unless linked through LFN
- A Call is effectively an anonymous derivation
- Late instantiation at workflow generation time
- Permits bundling of repetitive workflows
- Model: Function calls nested within a function definition
- Permits temporary filenames

## Compound Transformations

- TR diamond1 bundles black-diamonds
**TR** tut::diamond1**(** out fd, io fc1, io fc2, io fb1, io fb2, in fa, p1, p2 **) {**
-  **call** tut::preprocess**(** a=${fa}, b=[ ${out:fb1}, ${out:fb2} ] **);**
-  **call** tut::findrange**(** a1=${in:fb1}, a2=${in:fb2}, name="LEFT", p=${p1}, b=${out:fc1} **);**
-  **call** tut::findrange**(** a1=${in:fb1}, a2=${in:fb2}, name="RIGHT", p=${p2}, b=${out:fc2} **);**
-  **call** tut::analyze**(** a=[ ${in:fc1}, ${in:fc2} ], b=${fd} **);**
**}**

**Notes:**
The declaration uses abbreviations, nothing for "none", "in" for "input" etc.
Until transient filenames are implemented, it is required to declare the glue filenames as IO formal argument.
IO arguments must declare their direction either as in or out when being used (referenced).

## Compound Transformations (cont)

- Multiple DVs allow easy generator scripts:
**DV** tut::d1**->**tut::diamond1**(** fd=@{out:"f.00005"}, fc1=@{io:"f.00004"|}, fc2=@{io:"f.00003"|}, fb1=@{io:"f.00002"|}, fb2=@{io:"f.00001"|}, fa=@{io:"f.00000"}, p2="100", p1="0" **);**
**DV** tut::d2**->**tut::diamond1**(** fd=@{out:"f.0000B"}, fc1=@{io:"f.0000A"|}, fc2=@{io:"f.00009"|}, fb1=@{io:"f.00008"|}, fb2=@{io:"f.00007|"}, fa=@{io:"f.00006"}, p2="141.42135623731", p1="0" **);**
- ...
**DV** tut::d70**->**tut::diamond1**( **fd=@{out:"f.001A3"}, fc1=@{io:"f.001A2"|}, fc2=@{io:"f.001A1"|}, fb1=@{io:"f.001A0"|}, fb2=@{io:"f.0019F"|}, fa=@{io:"f.0019E"}, p2="800", p1="18" **);**

**Notes:**
Very verbose to link workflows together with inout files. All these files are tracked, transferred and registered, even though they may not serve any purpose beyond gluing a workflow together. 

## New Compound Transformations

- TR diamond2 bundles black-diamonds better:
**TR** tut::diamond2**(** out fd, in fa, p1, p2 **) {**
** **inout fb1 = @{inout:"f.b1":"f.b1.XXXXXX"};
** **inout fb2 = @{inout:"f.b2":"f.b2.XXXXXX"};
** **inout fc1 = @{inout:"f.c1":"f.c1.XXXXXX"};
** **inout fc2 = @{inout:"f.c2":"f.c2.XXXXXX"};
-  **call** tut::preprocess**(** a=${fa}, b=[ ${out:fb1}, ${out:fb2} ] **);**
-  **call** tut::findrange**(** a1=${in:fb1}, a2=${in:fb2}, name="LEFT", p=${p1}, b=${out:fc1} **);**
-  **call** tut::findrange**(** a1=${in:fb1}, a2=${in:fb2}, name="RIGHT", p=${p2}, b=${out:fc2} **);**
-  **call** tut::analyze**(** a=[ ${in:fc1}, ${in:fc2} ], b=${fd} **);**
**}**

**Notes:**
The formal argument declaration uses abbreviations, nothing for "none", "in" for "input" etc.
The temporary filenames which glue the graph are declared first. A new set of LFNs is instantiated for each instantiation of the compound.
If the glue filenames provide valuable information, render them trackable: Declare those glue filenames as IO formal argument.
IO filenames must declare their direction either as in or out when being used (referenced).
Using a temporary pattern implies dontRegister (no "r") and dontTransfer (no "t") option for LFNs.

## New Compound Transformations

- Easier generator scripts, less clutter:
**DV** tut::d1**->**tut::diamond**(** fd=@{out:"f.00002"}, fa=@{io:"f.00001"}, p2="100", p1="0" **);**
**DV** tut::d2**->**tut::diamond**(** fd=@{out:"f.00004"}, fa=@{io:"f.00003"}, p2="141.42135623731", p1="0" **);**
- ...
**DV** tut::d70**->**tut::diamond**( **fd=@{out:"f.00140"}, fa=@{io:"f.00139"}, p2="800", p1="18" **);**

**Notes:**
Less files to transfer or clutter RLS.