# Presentation: 828616

## Chimera Tutorial

- PDQ Meeting
- December 10th, 2002

## Outline

- Install Chimera
- Run Chimera
- Hello world
- Convert simple shell pipeline
- Some diamond, etc.
- Get (some) user apps to run with *shplan*
- Introduce The Grid
- As time permits

**Notes:**
Install= run test-version script.

## Prepare Chimera Installation

- Log onto your Linux system.
- Ensure Java version 1.3.*
- Type "$JAVA_HOME/bin/java –version"
- If necessary, download from _[http://java.sun.com/j2se/1.3/](http://java.sun.com/j2se/1.3/)_
- Using Java 1.4.* is permissible
- Check JAVA_HOME environment variable
- Points to Java installation directory
- Example: setenv JAVA_HOME /usr/lib/java

**Notes:**
Some developers are able to use cygwin runtime environment. There are issues with the path setup that are not easily portable between Unix, cygwin and Windows. You have been warned.
Java 1.4 needs more testing to see if anything is wrong.

## Install Chimera

- Download the latest tar-ball_[http://www.griphyn.org/workspace/VDS/snapshots.php](http://www.griphyn.org/workspace/VDS/snapshots.php)_
- The binary release is smaller
- The source release allows patching
- Let's try the source release for now...
- Unpack
*gtar xvzf vds-source-YYYYMMDD.VV.tar.gz*
* **cd vds-1.0b4*
- Current version is beta4

## Starting Chimera

- Source necessary shell script (all releases)
- Some environments require explicit setting of *VDS_HOME=`/bin/pwd`* before sourcing
- Bourne: *. setup-user-env.sh*
- C-Shell: *source setup-user-env.csh*
- Ignore warnings (but not errors) about VDS_HOME
- Run *test-version* program to check
- Script is a minimum sanity test
- If it fails, Chimera will not run!

**Notes:**
Developers should instead chdir into the src directory, and source the appropriate set-devel-env.sh or set-devel-env.csh scripts. 
If you are changing between user mode and developer mode, unset the CLASSPATH variable before sourcing the correct shell script.
The difference between user mode and developer mode is that the user mode includes chimera.jar into the CLASSPATH variable, whereas in developer mode, the CLASSPATH points to the start of the source hierarchy instead.  Additionally, the user mode script appends the PATH with $VDS_HOME/bin. 
In order to run Chimera in developer mode, one must compile all classes, e.g. use the make-class script.

## Sample test-version Script Output

- # checking for JAVA_HOME
- # checking for CLASSPATH
- # checking for VDS_HOME
- # script JAVA_HOME=/usr/lib/java
- # script CLASSPATH=.../lib/java-getopt-1.0.9.jar:.../lib/xercesImpl.jar:.../lib/chimera.jar:.../lib/antlrall.jar:.../lib/xmlParserAPIs.jar
- # starting java
- # Using recommended version of Java, OK
- # looking for Xerces
- # found xercesImpl\.jar, OK for now
- # looking for antlr
- # found antlr(all)?.jar, OK for now
- # looking for GNU getopt
- # found java-getopt-.\..\..\.jar, OK for now
- # looking for myself
- # in developer mode, ignore not finding chimera.jar
- # found chimera\.jar, OK for now

**Notes:**
In developer mode, the last step may fail, as there is no chimera.jar in the CLASSPATH. That is o.k.

## Working with Chimera files

- VDLt

- vdlt2vdlx

- VDLx

- vdlx2vdlt

- ins/upd

- VDLx

- VDLx

- gendax

- DAX

- Only the abstract planning process is shown

**Notes:**
VDLt: textual version of the virtual data language
VDLx: XML version of the virtual data language
DAX: abstract directed acyclic graph description
VDLt is slightly less expressive than VDLx, thus the initial back-and-forth conversion between the formats might lose some non-essential information. Any further round will yield equivalent results.

## The Virtual Data Languages

- VDLt is textual
- Concise
- For human consumption
**Usually for (TR) transformation!**
- Process by converting to VDLx

- VDLx uses XML
- Uses XML-Schema
- For generation from scripts
**Usually for (DV)****derivations!**
- Storage representation of default VDDB

## Virtual Data Language

- A "function call" paradigm
- Transformations are like function definitions.
- Derivations are like function calls.
- Many calls to one definition
- Many (zero to N) calls of the same transformation.

## The Virtual Data Database

- 5 different backends supported
- 3 backends employ SQL
- Some experimental code included

| file (default) | up to 1000 |
| --- | --- |
| dir (experimental) | up to 10000 |
| postgres | more than 1000 |
| mysql | more than 1000 |
| chunk (Pg-based) | more than 1000 |

## Hello World

- Exercise 1:
- Wrap *echo 'Hello world!'* into VDL
- Start by defining the transformation
- TR hw( output file ){argument = "Hello world!"; 	argument stdout = ${file}; }
- Then "call" the transformation
- DV d1->hw( file=@{output:"out.txt"} );
- Save into a file *hw.vdl*

**Notes:**
"world" is the name of the transformation. This is a logical name. 
"hello" is the name of the derivation (left of arrow), which calls "hello" (right of arrow).
Parameter matching is done on name (like Python), but expected to take position into account in future versions.
Every argument may be named. Only the special names "stdin", "stdout" and "stderr" refer to the streams to capture from or into a file.
All file names are logical.
Where now does the echo application come into the game?
And what about the logical file out.txt ?

## What to do with it?

- All VDLt must be converted into VDLx before it becomes "usable" to the VDS.
*vdlt2vdlx hw.vdl hw.xml*
- All VDLx must be stored into your VDDB, before the system can work with it
*insertvdc –d my.db hw.xml*
- or
*updatevdc –d my.db hw.xml*

**Notes:**
Insert will add to the database, but does not overwrite existing definitions
Update will add to the database, adding new items if necessary, or updating existing items.

## vdlt2vdlx

- All VDLt must be converted into VDLx before it becomes "usable" to the VDS.
*vdlt2vdlx hw.vdl hw.xml*

## XML Output

**<?xml version="1.0" encoding="UTF-8"?>**
**<!-- ... -->**
**<definitions ...>**
**  ****<transformation name="hw">**
**    ****<declare name="file" container="scalar" link="output"/>**
**    ****<argument>**
**      ****<text>Hello world!</text>**
**    ****</argument>**
**    ****<argument name="stdout">**
**      ****<use name="file"/>**
**    ****</argument>**
**  ****</transformation>**
**  ****<derivation name="d1" uses="hw">**
**    ****<pass bind="file">**
**        ****<scalar>**
**          ****<lfn file="out.txt" link="output"/>**
**        ****</scalar>**
**    ****</pass>**
**  ****</derivation>**
**</definitions>**

**Notes:**
Auto-generated comments were omitted
Attributes to the root element were omitted

## Creating the VDDB

- All VDLx must be stored into the VDDB, before the system can work with it
*insertvdc hw.xml*
- or
*updatevdc hw.xml*
- Default VDDB in $VDS_HOME/var/vds.db
- Unless overwritten by properties

**Notes:**
Insert will add to the database, but does not overwrite existing definitions
Update will add to the database, adding new items if necessary, or updating existing items.

## Result of Insertion

**sheveled:~ $ ****insertvdc -d my.db hw.xml**
**2002.12.09 10:32:56.895 CST: [app] Connecting the database backend**
**2002.12.09 10:32:57.276 CST: [app] will use /homes/voeckler/vds-1.0b4/etc/vdl-1.20.xsd**
**2002.12.09 10:32:57.276 CST: [app] parsing "hw.xml"**
**2002.12.09 10:32:58.115 CST: [app] Adding hw**
**2002.12.09 10:32:58.124 CST: [app] Adding d1**
**2002.12.09 10:32:58.151 CST: [app] modified 2 definitions**
**2002.12.09 10:32:58.151 CST: [app] rejected 0 definitions**

## From VDC to DAX

- Deriving the provenance of a logical file or derivation is the *abstract planning* process.
- All files and transformations are logical!
- The complete provenance will be unrolled
- No external catalogs (TC, RC) are queried
- Ask the catalog for the produced file
*gendax –l hw –o hw1.dax –f out.txt*
- Or ask for the derivation (to be fixed)
*gendax –l hw –o hw2.dax –i hello*

**Notes:**
More to be done for the red marked gendax statement: Currently, you will need to specify the fully qualified definition name, containing the namespace, name and version. Since we didn't give it a namespace nor version, this is a little tricky...
The label "-l" is used later to identify the master instance (DAG file, master shell script, etc.) to invoke later.

## From VDC to DAX

**sheveled:~ $ ****gendax –l hw –o hw1.dax –f out.txt**
**2002.12.09 11:08:25.031 CST: [app] Connecting the database backend**
**2002.12.09 11:08:25.419 CST: [app] will use /homes/voeckler/vds-1.0b4/etc/vdl-1.20.xsd**
**2002.12.09 11:08:26.291 CST: [app] 2 definitions loaded into main memory**
**2002.12.09 11:08:26.292 CST: [app] starting to route dbase**
**2002.12.09 11:08:26.315 CST: [app] requesting LFN "out.txt"**
**2002.12.09 11:08:26.353 CST: [app] saving output to hw1.dax**

**Notes:**
More to be done for the red marked gendax statement: Currently, you will need to specify the fully qualified definition name, containing the namespace, name and version. Since we didn't give it a namespace nor version, this is a little tricky...
The label "-l" is used later to identify the master instance (DAG file, master shell script, etc.) to invoke later.

## The DAX file

- The result of the Chimera abstract planner.
- Richer than Condor DAGMan format.
- Expressed in terms of logical entities.
- Contains complete (build) lineage.
- Consists of three major parts
- All logical files necessary for the DAG.
- All jobs necessary to produce all files.
- The dependencies between the jobs.

## The DAX file

**<?xml version="1.0" encoding="UTF-8"?>**
**<!-- generated: 2002-12-09T10:55:41-06:00 -->**
**<!-- generated by: voeckler [US] -->**
**<adag ... count="1" index="0" name="hw">**
**  ****<filename file="out.txt" link="output" isTemporary="false"/>**
**  ****<job name="hw" id="ID000001">**
**    ****<argument>Hello world! </argument>**
**    ****<stdout file="out.txt" link="output" isTemporary="false"/>**
**    ****<uses file="out.txt" link="output" isTemporary="false"/>**
**  ****</job>**
**</adag>**

## The Transformation Catalog

- Translates logical transformation into an application specific for a certain pool
- Similar for all concrete planners of Chimera
- Simple, text based file
- 3+ columns
- Blank lines and comments are ignored
- Standard location
- $VDS_HOME/var/tc.data
- adjustable with property vds.db.tc

**Notes:**
$VDS_HOME/etc/properties contains all user properties for VDS. Please refer to the user guide for details on available properties and their effects. 
Comments start with a octothorpe (#).

## Transformation Catalog Columns

- 1st column is the pool handle
- Shell planner only uses the "local" handle
- Other handles for grid planner
- 2nd column is the logical transformation
- Format: <ns>_ _ <name>_<version>
- Name-only names are just <name>
- 3rd column is the path to the executable
- 4th column for environment variables
- Use "null" if unused
- Format: key=value;key=value

**Notes:**
Support for the fully-qualified transformation name may be lacking in some concrete planners.
Examples are strewn over many directories, e.g. $VDS_HOME/test/test?/tc.data

## Example for a TC

- local world /homes/voeckler/bin/keg null
- uofc globus-url-copy /vdt/bin/globus-url-copy GLOBUS_LOCATION=/vdt; LD_LIBRARY_PATH=/vdt/lib

## The Shplan Replica Catalog

- A replica catalog translates a logical filename into a set of physical filenames
- This RC is for the shell planner only!
- Simple textual file
- Three columns
- Blank lines and comments are ignored
- Standard location
- $VDS_HOME/var/rc.data
- Adjustable with property vds.db.rc

**Notes:**
$VDS_HOME/etc/properties contains all user properties for VDS. Please refer to the user guide for details on available properties and their effects. 

## Shell planner RC Columns

- 1st column is the pool handle
- Shell planner only uses the "local" handle
- Other handles for concrete planner
- 2nd column is the logical filename
- A LFN may contain slash etc.
- 3rd column is the path to the file
- Are multiples allowed?
- No, only the last (first?) match is taken

## TC and RC Short-cuts

- Application locations can come from VDL
- hints.pfnHint takes precedence over TC
- Shell planner may run w/o any TC
- Files need not be registered with RC
- Existence checks are done in file system
- RC updates can be optional
- Shell planner may run w/o any RC

-  After all, we run locally with the shell planner!

**Notes:**
Using application locations in VDL is not recommended, as the specific location of an application may change from pool to pool. 
Thus pfnHint is not recommended.

## Preparing to Run The Shell Planner

- Make sure that your TC contains a translation for logical transformation "hw"
- local	hw	/bin/echo	null
- Make sure that there is a RC without weird content:
*cp /dev/null $VDS_HOME/var/rc.data*

**Notes:**
The distributed rc.data may contain some weird content, sorry!

## Running The Shell Planner

- Run the shell planner
*shplanner –o hw hw1.dax*
- Check directory "hw"
- Master script: <DAX-label>.sh
- Job scripts: <TR>_<DAX-JobID>.sh
- Helper files: <TR>_<DAX-JobID>.lst

**Notes:**
Ignore the slightly verbose output for now

## Running The Shell-Plan

- Shell planner generates a master plan
- This is a tool-kit  no auto-run feature
- Run the master plan
*( cd hw && ./hw.sh )*
- Can check log file for status etc.
- Standard name: <DAX-label>.log
- Master script will exit with 0 on success
- Exit 1, if any sub-script fails.

## Running The Shell-Plan

**sheveled:~ $ ****./hw.sh**
**Running hw**
**job finished**
**please look at hw.log for execution logs.**
**sheveled:~ $ ****cat out.txt**
**Hello world!**
**sheveled:~ $ ****cat hw.log**
**2002-12-09T11:04:02 ID000001 started hw**
**2002-12-09T11:04:02 ID000001 finished hw**

## Convert A Unix Pipeline

- Example 2: Full name from a username
*grep ^user /etc/passwd | awk –F: '{ print $5 }'*
- Create abstract transformations
- Create concrete derivations to call them
- Convert into VDLx
- Add to VDDB
- Run abstract planner
- Run shell planner
- Run master script

## Abstract Transformations

- Grep for an arbitrary user name
- TR grep( none name, output of ){ argument = "^"${name}" /etc/passwd";argument stdout = ${of}; }
- Extract a full name from a line
- TR awk( input line, output full ){ argument stdin = ${line};argument stdout = ${full};argument = "-F: '{ print $5 }' "; }

**Notes:**
DO NOT call your files any of the reserved words none, in, input, inout, io, out, or output. Currently, the standard vdlt2vdlx also does not allow these reserved words as prefix to any name, but we intend to fix that.
Some tricks here. The /etc/passwd, albeit a file, is handled like a constant. Thus, it will be untracked. For most Unix systems, this is ok. 
The script to awk is presented on the commandline to awk. Thus, its quotes must be quoted for correct usage. Please note that the first example might run into troubles, too, if certain things are used w/o quoting.

## Concrete Derivations

- Run with a concrete username...
- DV d2->grep( name="voeckler", of=@{output:"grepout.txt"} );
- ... and post-process the results
- DV d3->awk( line=@{in:"grepout.txt"}, full=@{out:"awkout.txt"} );
- The two derivations are linked by a LFN
- "grepout.txt" is produced by "d2"
- "grepout.txt" is consumed by "d3"

## Convert, Insert and Plan

- Convert into VDLx
*vdlt2vdlx ex2.vdl ex2.xml*
- Insert into your VDDB
*insertvdc ex2.xml*
- Run abstract planner
*gendax –l ex2 –o ex2.dax –f awkout.txt*

## Run Shell Planner

- Prepare transformation catalog
- local	grep	/usr/bin/egrep	null
- local	awk	/opt/gnu/bin/gawk	null
- Run shell planner
*shplanner –o ex2 ex2.dax*
- Run master plan
*( cd ex2 && ./ex2.sh )*

## Kanonical Executable for Grids

- Simple application
- Copies all input with indentation to all output files
- Adds additional data about itself
- Allows tracking
- What ran where, when, which architecture, and how long
- To be used as stand-in for real applications
- Allows to check the DAG stages

**Notes:**
Run "keg –o /dev/fd/1"

## Kanonical Executable for Grids

**sheveled:~ $ ****keg -o /dev/fd/1**
**Timestamp Today: 20021210T111605-06:00 (1039540565.690;0.003)**
**Applicationname: keg @ 140.221.10.73 (sheveled.mcs.anl.gov)**
**Current Workdir: /home/voeckler**
**Systemenvironm.: i686-Linux 2.4.17-w**
**Processor Info.: 2 x Pentium III (Coppermine) @ 996.635**
**Output Filename: /dev/fd/1**

**Notes:**
IPv4 is tried hard to look up, fqdn may fail (not guaranteed)

## The Black Diamond DAG

- Complex structure
- Fan-in
- Fan-out
- "left" and "right" can run in parallel
- Uses input file
- Register with RC
- Complex file dependencies

## The Black Diamond DAG II

- The "black diamond" takes one input file
*echo –e "local\tf.a\t${HOME}/f.a" » $VDS_HOME/var/rc.data*
*date > ${HOME}/f.a*
- It uses three different transformations
*echo –e "local\tpreprocess\t$VDS_HOME/bin/keg\tnull" » $VDS_HOME/var/tc.data*
*echo –e "local\tfindrange\t$VDS_HOME/bin/keg\tnull" » $VDS_HOME/var/tc.data*
*echo –e "local\tanalyze\t$VDS_HOME/bin/keg\tnull" » $VDS_HOME/var/tc.data*

## The Black Diamond DAG III

- Running on the grid is more complex
*Activate your grid proxy*
*Set up the replica catalog*
*Currently RLS (new) and G2RC (default) available*
*Register src, dst, and run pool with RC using a dummy*
*Register input file *f.a*, unregister all other files*
*Set up *pool.config* for src, dst, run and local*
*Set up *tc.data* file for src, dst, run and local*
*Run concrete planner with --donotrun*
*Run Condor DAGMan on output*

## The Black Diamond DAG IV

## A Big Example

- Overall tree structure
- 70 "regular" diamond DAGs on top
- 10 collecting nodes
- 1 final collector
- 291 jobs to run
- All files are generated, no RC preloading
- Add "multi" TR to last example
*echo –e "local\tmulti\t$VDS_HOME/bin/keg\tnull" » $VDS_HOME/var/tc.data*

## Submitting Chimera Work to the Grid

- Setting up a submit site: VDT Client
- Setting up a replica catalog
- Setting up grid sites: VDT Server
- Issues
- Reccommendations
- VDT based
- Installing apps in the Grid
- Setting up the pool and tc config files