# Presentation: 243206

## Lecture 3Grid Resources and Job Management

- Jaime Frey
- Condor Project,
- University of Wisconsin-Madison
- jfrey@cs.wisc.edu
-                  Grid Summer Workshop
-                        June 21-25, 2004

- Lecture2: Grid Job Management

- June 21-25, 2004

## Question For Today

- How do you manage jobs on a Grid?
- Recall globus-job-run yesterday
- No job tracking: what happens when you hit control-c?
- No way to run sets of jobs
- Clearly we need something better!
- Condor-G for reliable job management
- DAGMan for controlling sets of jobs
- First we’re going to tell you a little bit about Condor

- Lecture2: Grid Job Management

- June 21-25, 2004

## The Condor Project (Established ‘85)

- Distributed High Throughput Computing research performed by a team of ~35 faculty, full time staff and students.

- Lecture2: Grid Job Management

- June 21-25, 2004

## What is Condor?

- Condor converts collections of distributively owned workstations and dedicated clusters into a distributed high-throughput computing (HTC) facility.
- Condor manages both resources (machines) and resource requests (jobs)
- Condor has several unique mechanisms such as :
- ClassAd Matchmaking
- Process checkpoint/ restart / migration
- Remote System Calls
- Grid Awareness

- Lecture2: Grid Job Management

- June 21-25, 2004

## Condor can manage a large number of jobs

- Managing a large number of jobs
- You specify the jobs in a file and submit them to Condor, which runs them all and keeps you notified on their progress
- Mechanisms to help you manage huge numbers of jobs (1000’s), all the data, etc.
- Condor can handle inter-job dependencies (DAGMan)
- Condor users can set job priorities
- Condor administrators can set user priorities

- Lecture2: Grid Job Management

- June 21-25, 2004

## Condor can manage Dedicated Resources...

- Dedicated Resources
- Compute Clusters
- Manage
- Node monitoring, scheduling
-  Job launch, monitor & cleanup

- Lecture2: Grid Job Management

- June 21-25, 2004

## ...Condor can manage non-dedicated resources...

- Non-dedicated resources examples:
- Desktop workstations in offices
- Workstations in student labs
- Non-dedicated resources are often idle  --- ~70% of the time!
- Condor can effectively harness the otherwise wasted compute cycles from non-dedicated resources

- Lecture2: Grid Job Management

- June 21-25, 2004

## ... and Condor Can Manage Grid jobs

- Condor-G is a specialization of Condor. It is also known as the “Globus universe” or “Grid universe”.
- Condor-G can submit jobs to Globus resources, just like globus-job-run.
- Condor-G benefits from all the wonderful Condor features, like a real job queue.

- Lecture2: Grid Job Management

- June 21-25, 2004

## Some Grid Challenges

- Condor-G does whatever it takes to run your jobs, even if ...
- The gatekeeper is temporarily unavailable
- The job manager crashes
- The network goes down

- Lecture2: Grid Job Management

- June 21-25, 2004

## Remote Resource Access: Globus

- “globusrun
-         myjob ...”

- Globus GRAM Protocol

- Globus
- JobManager

- fork()

- Organization A

- Organization B

- Lecture2: Grid Job Management

- June 21-25, 2004

## Remote Resource Access: Globus

- Globus GRAM Protocol

- Globus
- JobManager

- fork()

- Organization A

- Organization B

- “globusrun
-         myjob ...”

- Lecture2: Grid Job Management

- June 21-25, 2004

## Remote Resource Access: Globus + Condor

- Globus GRAM Protocol

- Globus
- JobManager

**Submit to Condor**

- Condor Pool

- Organization A

- Organization B

- “globusrun
-         myjob ...”

- Lecture2: Grid Job Management

- June 21-25, 2004

## Remote Resource Access: Globus + Condor

- “globusrun ...”

- Globus GRAM Protocol

- Globus
- JobManager

**Submit to Condor**

- Condor Pool

- Organization A

- Organization B

- Lecture2: Grid Job Management

- June 21-25, 2004

## Remote Resource Access: Condor-G + Globus + Condor

- Globus GRAM Protocol

- Globus
- JobManager

**Submit to Condor**

- Condor Pool

- Organization A

- Organization B

**Condor-G**

- myjob1
- myjob2
- myjob3
- myjob4
- myjob5
- ...

- Lecture2: Grid Job Management

- June 21-25, 2004

## Just to be fair...

- The gatekeeper doesn’t have to submit to a Condor pool.
- It could be PBS, LSF, Sun Grid Engine...
- Condor-G will work fine whatever the remote batch system is.

- Lecture2: Grid Job Management

- June 21-25, 2004

## The Idea

- Computing power is everywhere, Condor tries to make it usable by anyone.

- Lecture2: Grid Job Management

- June 21-25, 2004

## First Condor, then Condor-G

- We’re going to learn the basics of Condor first...
- Almost everything you learn about Condor applies to Condor-G as well.

- Lecture2: Grid Job Management

- June 21-25, 2004

## Meet Frieda.

**She is a scientist.  But she has a big**
**problem.**

- Lecture2: Grid Job Management

- June 21-25, 2004

## Frieda’s Application ...

- Simulate the behavior of **F**(*x,y,z*) for 20 values of *x*, 10 values of *y* and 3 values of *z  *(20*10*3 = 600 combinations)
**F** takes on the average 3 hours to compute on a “typical” workstation (total = 1800 hours)
**F **requires a “moderate” (128MB) amount of memory
**F** performs “moderate” I/O - (*x,y,z*) is 5 MB and **F**(*x,y,z*) is 50 MB

- Lecture2: Grid Job Management

- June 21-25, 2004

## I have 600simulations to run.Where can I get help?

- Lecture2: Grid Job Management

- June 21-25, 2004

## Install a 

**Install a **
**Personal Condor!**

- Lecture2: Grid Job Management

- June 21-25, 2004

## Installing Condor

- Download Condor for your operating system
- Available as a free download from
- 		http://www.cs.wisc.edu/condor
- Stable –vs- Developer Releases
- Naming scheme similar to the Linux Kernel...
- Available for most Unix platforms and Windows NT
- It’s already installed on your computers

- Lecture2: Grid Job Management

- June 21-25, 2004

## So Frieda Installs Personal Condor on her machine...

- What do we mean by a “Personal” Condor?
- Condor on your own workstation, no root access required, no system administrator intervention needed
- So after installation, Frieda submits her jobs to her Personal Condor...

- Lecture2: Grid Job Management

- June 21-25, 2004

## Slide 24

- Lecture2: Grid Job Management

- June 21-25, 2004

## Personal Condor?!What’s the benefit of a Condor “Pool” with just one user and one machine?

- Lecture2: Grid Job Management

- June 21-25, 2004

## Your Personal Condor will ...

- ... keep an eye on your jobs and will keep you posted on their progress
- ... implement your policy on the execution order of the jobs
- ... keep a log of your job activities
- ... add fault tolerance to your jobs
- ... implement your policy on when the jobs can run on your workstation

- Lecture2: Grid Job Management

- June 21-25, 2004

## Getting Started: Submitting Jobs to Condor

- Choosing a “Universe” for your job
- Just use VANILLA for now
- This isn’t a grid job, but almost everything applies, without the complication of the grid
- Make your job “batch-ready”
- Creating a *submit description* file
- Run *condor_submit* on your submit description file

- Lecture2: Grid Job Management

- June 21-25, 2004

## Making your job ready

- Must be able to run in the background: no interactive input, windows, GUI, etc.
- Can still use STDIN, STDOUT, and STDERR (the keyboard and the screen), but files are used for these instead of the actual devices
- Organize data files

- Lecture2: Grid Job Management

- June 21-25, 2004

## Creating a Submit Description File

- A plain ASCII text file
- Tells Condor about your job:
- Which executable, universe, input, output and error files to use, command-line arguments, environment variables, any special requirements or preferences (more on this later)
- Can describe many jobs at once (a “cluster”) each with different input, arguments, output, etc.

- Lecture2: Grid Job Management

- June 21-25, 2004

## Simple Submit Description File

**# Simple condor_submit input file**
**# (Lines beginning with # are comments)**
**# NOTE: the words on the left side are not**
**#       case sensitive, but filenames are!**
**Universe   = vanilla**
**Executable = my_job**
**Queue**

- Lecture2: Grid Job Management

- June 21-25, 2004

## Running condor_submit

- You give *condor_submit* the name of the submit file you have created
*condor_submit *parses the file, checks for errors, and creates a “ClassAd” that describes your job(s)
- Sends your job’s ClassAd(s) and executable to the condor_schedd, which stores the job in its queue
- Atomic operation, two-phase commit
- View the queue with *condor_q*

- Lecture2: Grid Job Management

- June 21-25, 2004

## Running condor_submit

**% condor_submit my_job.submit-file**
**Submitting job(s).**
**1 job(s) submitted to cluster 1.**
**% condor_q**
**-- Submitter: perdita.cs.wisc.edu : <128.105.165.34:1027> : **
** ****ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               **
**  ****1.0    frieda           6/16 06:52   0+00:00:00 I  0   0.0  my_job**
**1 jobs; 1 idle, 0 running, 0 held**
**%**

- Lecture2: Grid Job Management

- June 21-25, 2004

## Another Submit Description File

**# Example condor_submit input file**
**# (Lines beginning with # are comments)**
**# NOTE: the words on the left side are not**
**#       case sensitive, but filenames are!**
**Universe   = vanilla**
**Executable = /home/wright/condor/my_job.condor**
**Input      = my_job.stdin**
**Output     = my_job.stdout**
**Error      = my_job.stderr**
**Arguments  = -arg1 -arg2**
**InitialDir = /home/wright/condor/run_1**
**Queue**

- Lecture2: Grid Job Management

- June 21-25, 2004

## “Clusters” and “Processes”

- If your submit file describes multiple jobs, we call this a “cluster”
- Each job within a cluster is called a “process” or “proc”
- If you only specify one job, you still get a cluster, but it has only one process
- A Condor “Job ID” is the cluster number, a period, and the process number (“23.5”)
- Process numbers always start at 0

- Lecture2: Grid Job Management

- June 21-25, 2004

## Example Submit Description File for a Cluster

- # Example condor_submit input file that defines
- # a cluster of two jobs with different iwd
- Universe   = vanilla
- Executable = my_job
- Arguments  = -arg1 -arg2
- InitialDir = run_0 **  **
- Queue     ****** Becomes job 2.0**
- InitialDir = run_1
- Queue     ****** Becomes job 2.1**

- Lecture2: Grid Job Management

- June 21-25, 2004

## % condor_submit my_job.submit-file

**Submitting job(s).**
**2 job(s) submitted to cluster 2.**
**% condor_q**
**-- Submitter: perdita.cs.wisc.edu : <128.105.165.34:1027> : **
** ****ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               **
**  ****1.0    frieda           6/16 06:52  0+00:02:11 R  0   0.0  my_job**
**  ****2.0    frieda           6/16 06:56  0+00:00:00 I  0   0.0  my_job**
**  ****2.1    frieda           6/16 06:56  0+00:00:00 I  0   0.0  my_job**
**3 jobs; 2 idle, 1 running, 0 held**
**%**

- Lecture2: Grid Job Management

- June 21-25, 2004

## Submit Description File for a BIG Cluster of Jobs

- The initial directory for each job is specified with the $(Process) macro, and instead of submitting a single job, we use “Queue 600” to submit 600 jobs at once
- $(Process) will be expanded to the process number for each job in the cluster (from 0 up to 599 in this case), so we’ll have “run_0”, “run_1”, ... “run_599” directories
- All the input/output files will be in different directories!

- Lecture2: Grid Job Management

- June 21-25, 2004

## Submit Description File for a BIG Cluster of Jobs

**# Example condor_submit input file that defines**
**# a cluster of 600 jobs with different iwd**
**Universe   = vanilla**
**Executable = my_job**
**Arguments  = -arg1 –arg2**
**InitialDir = run_$(Process)**
**Queue 600**

- Lecture2: Grid Job Management

- June 21-25, 2004

## Using condor_rm

- If you want to remove a job from the Condor queue, you use *condor_rm*
- You can only remove jobs that you own (you can’t run *condor_rm* on someone else’s jobs unless you are root)
- You can give specific job ID’s (cluster or cluster.proc), or you can remove all of your jobs with the “*-a*” option.

- Lecture2: Grid Job Management

- June 21-25, 2004

## Temporarily halt a Job

- Use *condor_hold *to place a job on hold
- Kills job if currently running
- Will not attempt to restart job until released
- Use *condor_release *to remove a hold and permit job to be scheduled again

- Lecture2: Grid Job Management

- June 21-25, 2004

## A Job’s life story: The “User Log” file

- A UserLog must be specified in your submit file:
- Log = filename
- You get a log entry for everything that happens to your job:
- When it was submitted, when it starts executing, preempted, restarted, completes, if there are any problems, etc.
- Very useful!  Highly recommended!

- Lecture2: Grid Job Management

- June 21-25, 2004

## Sample Condor User Log

**000 (8135.000.000) 05/25 19:10:03 Job submitted from host: <128.105.146.14:1816>**
**...**
**001 (8135.000.000) 05/25 19:12:17 Job executing on host: <128.105.165.131:1026>**
**...**
**005 (8135.000.000) 05/25 19:13:06 Job terminated.**
**	****(1) Normal termination (return value 0)**
**	****	****Usr 0 00:00:37, Sys 0 00:00:00  -  Run Remote Usage**
**	****	****Usr 0 00:00:00, Sys 0 00:00:05  -  Run Local Usage**
**	****	****Usr 0 00:00:37, Sys 0 00:00:00  -  Total Remote Usage**
**	****	****Usr 0 00:00:00, Sys 0 00:00:05  -  Total Local Usage**
**	****9624  -  Run Bytes Sent By Job**
**	****7146159  -  Run Bytes Received By Job**
**	****9624  -  Total Bytes Sent By Job**
**	****7146159  -  Total Bytes Received By Job**
**...**

- Lecture2: Grid Job Management

- June 21-25, 2004

## Uses for the User Log

- Easily read by human or machine
- C++ library and Perl Module  for parsing UserLogs is available
- log_xml=True – XML formatted
- Event triggers for schedulers
- DAGMan runs sets of jobs in a specified order.
- It watches the UserLog to learn when jobs finish
- Visualizations of job progress
- Condor JobMonitor Viewer

- Lecture2: Grid Job Management

- June 21-25, 2004

## What if each job needed to run for 20 days?Something might crash...

- Lecture2: Grid Job Management

- June 21-25, 2004

## Condor’s Standard Universe to the rescue!

- Wouldn’t it be nice if when your job crashed, you could *roll-back* to where you job was an hour ago, instead of completely restarting it?
- The *Standard universe* supplies checkpointing, to give you this functionality.

- Lecture2: Grid Job Management

- June 21-25, 2004

## Process Checkpointing

- Condor’s Process Checkpointing mechanism saves all the state of a process into a checkpoint file
- Memory, CPU, I/O, etc.
- The process can then be restarted *from right where it left off.*
- Typically no changes to your job’s source code needed – however, your job must be *relinked* with Condor’s Standard Universe support library
- We will say no more today, since we are concentrating on Grid jobs.

- Lecture2: Grid Job Management

- June 21-25, 2004

## Happy Day!  Frieda’s organization purchased a Beowulf Cluster!

- Frieda Installs Condor on all the dedicated Cluster nodes, and configures them with her machine as the central manager...
- Now her Condor Pool can run multiple jobs at once

- Lecture2: Grid Job Management

- June 21-25, 2004

## Slide 48

- Lecture2: Grid Job Management

- June 21-25, 2004

## Back to the Story...

**Frieda Needs Remote Resources...**

- Lecture2: Grid Job Management

- June 21-25, 2004

## Condor-G: Access non-Condor Grid resources

- Globus
- middleware deployed across entire Grid
- remote access to computational resources
- dependable, robust data transfer

- Condor
- job scheduling across multiple resources
- strong fault tolerance with checkpointing and migration
- layered over Globus as “personal batch system” for the Grid

- Lecture2: Grid Job Management

- June 21-25, 2004

## Condor-G 

**Condor-G**

- Job Description (Job ClassAd)

- Lecture2: Grid Job Management

- June 21-25, 2004

## Frieda Submits a Globus Universe Job

- In her submit description file, she specifies:
- Universe = Globus
- Which Globus Gatekeeper to use
- Optional: Location of file containing your Globus certificate
**universe        = globus**
**globusscheduler = beak.cs.wisc.edu/jobmanager**
**executable      = progname**
**queue**

- Lecture2: Grid Job Management

- June 21-25, 2004

## How It Works

- Schedd

- LSF

- Personal Condor

- Globus Resource

- Lecture2: Grid Job Management

- June 21-25, 2004

## How It Works

- Schedd

- LSF

- Personal Condor

- Globus Resource

- Lecture2: Grid Job Management

- June 21-25, 2004

## How It Works

- Schedd

- LSF

- Personal Condor

- Globus Resource

- GridManager

- Lecture2: Grid Job Management

- June 21-25, 2004

## How It Works

- Schedd

- JobManager

- LSF

- Personal Condor

- Globus Resource

- GridManager

- Lecture2: Grid Job Management

- June 21-25, 2004

## How It Works

- Schedd

- JobManager

- LSF

- User Job

- Personal Condor

- Globus Resource

- GridManager

- Lecture2: Grid Job Management

- June 21-25, 2004

## Globus Universe Concerns

- What about Fault Tolerance?
- Local Crashes
- What if the submit machine goes down?
- Network Outages
- What if the connection to the remote Globus jobmanager is lost?
- Remote Crashes
- What if the remote Globus jobmanager crashes?
- What if the remote machine goes down?

- Lecture2: Grid Job Management

- June 21-25, 2004

## Changes to the Globus JobManager	for Fault Tolerance

- Ability to restart a JobManager
- Enhanced two-phase commit submit protocol

- Lecture2: Grid Job Management

- June 21-25, 2004

## Globus Universe Fault-Tolerance: Submit-side Failures

- All relevant state for each submitted job is stored persistently in the Condor job queue.
- This persistent information allows the Condor GridManager upon restart to read the state information and reconnect to JobManagers that were running at the time of the crash.
-  If a JobManager fails to respond...

- Lecture2: Grid Job Management

- June 21-25, 2004

## Globus Universe Fault-Tolerance:Lost Contact with Remote Jobmanager

***Can we contact gatekeeper?***

***Yes – network was down***

***No – machine crashed or job completed***

***Yes - jobmanager crashed***

***No – retry until we can talk to gatekeeper again...***

***Can we reconnect to jobmanager?***

***Has job completed?***

***No – is job still running?***

***Yes – update queue***

***Restart jobmanager***

- Lecture2: Grid Job Management

- June 21-25, 2004

## Globus Universe Fault-Tolerance: Credential Management

- Authentication in Globus is done with limited-lifetime X509 proxies
- Proxy may expire before jobs finish executing
- Condor can put jobs on hold and email user to refresh proxy
- Todo: Interface with MyProxy...

- Lecture2: Grid Job Management

- June 21-25, 2004

## Can Condor-G decide where to run my jobs?

- Lecture2: Grid Job Management

- June 21-25, 2004

## Condor-G Matchmaking

- Use Condor-G matchmaking with globus universe jobs
- Allows Condor-G to dynamically assign computing jobs to grid sites
- An example of *lazy planning*

- Lecture2: Grid Job Management

- June 21-25, 2004

## Condor-G Matchmaking, cont.

- Normally a globus universe job must specify the site in the submit description file via the “globusscheduler” attribute like so:
**Executable = foo**
**Universe = globus**
**Globusscheduler = beak.cs.wisc.edu/jobmanager-pbs**
**queue**

- Lecture2: Grid Job Management

- June 21-25, 2004

## Condor-G Matchmaking, cont.

- With matchmaking, globus universe jobs can use requirements and rank:
**Executable = foo**
**Universe = globus**
**Globusscheduler = $$(GatekeeperUrl)**
**Requirements = arch == LINUX**
**Rank = NumberOfNodes**
**Queue**
- The $$(x) syntax inserts information from the target ClassAd when a match is made.

- Lecture2: Grid Job Management

- June 21-25, 2004

## Condor-G Matchmaking, cont.

- Where do these target ClassAds representing Globus gatekeepers come from?  Several options:
- Simple script on gatekeeper publishes an ad via *condor_advertise* command-line utility (method used by D0 JIM, USCMS)
- Program to query Globus MDS and convert information into ClassAd (method used by EDG)
- Run HawkEye with appropriate plugins on the gatekeeper
- For explanation of Condor-G matchmaking setup for USCMS, see http://www.cs.wisc.edu/condor/USCMS_matchmaking.html

- Lecture2: Grid Job Management

- June 21-25, 2004

## But Frieda Wants More...

- She wants to run standard universe jobs on Globus-managed resources
- For matchmaking and dynamic scheduling of jobs
- For job checkpointing and migration
- For remote system calls

- Lecture2: Grid Job Management

- June 21-25, 2004

## One Solution: Condor-G GlideIn

- Frieda can use the Globus Universe to run Condor daemons on Globus resources
- When the resources run these GlideIn jobs, they will temporarily join her Condor Pool
- She can then submit Standard, Vanilla, PVM, or MPI Universe jobs and they will be matched and run on the Globus resources

- Lecture2: Grid Job Management

- June 21-25, 2004

## Condor Pool

- Friendly Condor Pool

- Lecture2: Grid Job Management

- June 21-25, 2004

## How It Works

- Schedd

- LSF

- Collector

- Personal Condor

- Globus Resource

- Lecture2: Grid Job Management

- June 21-25, 2004

## How It Works

- Schedd

- LSF

- Collector

- Personal Condor

- Globus Resource

- Lecture2: Grid Job Management

- June 21-25, 2004

## How It Works

- Schedd

- LSF

- Collector

- Personal Condor

- Globus Resource

- GridManager

- Lecture2: Grid Job Management

- June 21-25, 2004

## How It Works

- Schedd

- JobManager

- LSF

- Collector

- Personal Condor

- Globus Resource

- GridManager

- Lecture2: Grid Job Management

- June 21-25, 2004

## How It Works

- Schedd

- JobManager

- LSF

- Startd

- Collector

- Personal Condor

- Globus Resource

- GridManager

- Lecture2: Grid Job Management

- June 21-25, 2004

## How It Works

- Schedd

- JobManager

- LSF

- Startd

- Collector

- Personal Condor

- Globus Resource

- GridManager

- Lecture2: Grid Job Management

- June 21-25, 2004

## How It Works

- Schedd

- JobManager

- LSF

- User Job

- Startd

- Collector

- Personal Condor

- Globus Resource

- GridManager

- Lecture2: Grid Job Management

- June 21-25, 2004

## GlideIn Concerns

- What if a Globus resource kills my GlideIn job?
- That resource will disappear from your pool and your jobs will be rescheduled on other machines
- Standard universe jobs will resume from their last checkpoint like usual
- What if all my jobs are completed before a GlideIn job runs?
- If a GlideIn Condor daemon is not matched with a job in 10 minutes, it terminates, freeing the resource

- Lecture2: Grid Job Management

- June 21-25, 2004

## DAGMan

_**D**_**irected **_**A**_**cyclic **_**G**_**raph **_**Man**_**ager**
- DAGMan allows you to specify the *dependencies* between your Condor jobs, so it can *manage* them automatically for you.
- By default, Condor may run your jobs in any order, or everything simultaneously, so we need DAGMan to enforce an ordering when necessary.
- (e.g., “Don’t run job “B” until job “A” has completed successfully.”)

- Lecture2: Grid Job Management

- June 21-25, 2004

**Notes:**
In the simplest case...

## What is a DAG?

- A DAG is the data structure used by DAGMan to represent these dependencies.
- Each job is a “node” in the DAG.
- Each node can have any number of “parent” or “children” nodes – as long as there are no loops!

- Lecture2: Grid Job Management

- June 21-25, 2004

**Notes:**
a DAG is the best data structure to represent a workflow of jobs with dependencies
children may not run until their parents have finished – this is why the graph is a directed graph ... there’s a direction to the flow of work
In this example, called a “diamond” dag, job A must run first; when it finishes, jobs B and C can run together; when they are both finished, D can run; when D is finished the DAG is finished
Loops, where two jobs are both descended from one another, are prohibited because they would lead to deadlock – in a loop, neither node could run until the other finished, and so neither would start – this restriction is what makes the graph acyclic

## A DAG is defined by a .dag file, listing each of its nodes and their dependencies:

# diamond.dag
Job A a.sub
Job B b.sub
Job C c.sub
Job D d.sub
Parent A Child B C
Parent B C Child D
each node will run the Condor job specified by its accompanying Condor submit file

**# diamond.dag**
**Job A a.sub**
**Job B b.sub**
**Job C c.sub**
**Job D d.sub**
**Parent A Child B C**
**Parent B C Child D**
- each node will run the Condor job specified by its accompanying Condor submit file

- Defining a DAG

- Lecture2: Grid Job Management

- June 21-25, 2004

**Notes:**
This is all it takes to specify the example “diamond” dag.
DAGMan will output the log file for each job to determine what to do next.

## Submitting a DAG

- To start your DAG, just run ***condor_submit_dag*** with your .dag file, and Condor will start a personal DAGMan daemon which to begin running your jobs:
- % condor_submit_dag diamond.dag
- condor_submit_dag  submits a job with DAGMan as the executable.
- This job happens to run on the submitting machine, not any other computer.
- Thus the DAGMan daemon itself runs as a Condor job, so you don’t have to baby-sit it.

- Lecture2: Grid Job Management

- June 21-25, 2004

**Notes:**
Just like any other Condor job, you get fault tolerance in case the machine crashes or reboots, or if there’s a network outage
And you’re notified when it’s done, and whether it succeeded or failed
% condor_q
-- Submitter: foo.bar.edu : <128.105.175.133:1027> : foo.bar.edu
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
2456.0   user            3/8  19:22   0+00:00:02 R  0   3.2  condor_dagman -f -

## Running a DAG

- DAGMan acts as a “meta-scheduler”, managing the submission of your jobs to Condor based on the DAG dependencies.

- Condor
- Job
- Queue

- C

- D

- A

- A

- B

- .dag
- File

- Lecture2: Grid Job Management

- June 21-25, 2004

**Notes:**
First, job A will be submitted alone...

## Running a DAG (cont’d)

- DAGMan holds & submits jobs to the Condor queue at the appropriate times.

- Condor
- Job
- Queue

- C

- D

- B

- C

- B

- A

- Lecture2: Grid Job Management

- June 21-25, 2004

**Notes:**
Once job A completes successfully, jobs B and C will be submitted at the same time...

## Running a DAG (cont’d)

- In case of a job failure, DAGMan continues until it can no longer make progress, and then creates a *“rescue” file* with the current state of the DAG.

- Condor
- Job
- Queue

- X

- D

- A

- B

- Rescue
- File

- Lecture2: Grid Job Management

- June 21-25, 2004

**Notes:**
If job C fails, DAGMan will wait until job B completes, and then will exit, creating a rescue file.  Job D will not run.
In its log, DAGMan will provide additional details of which node failed and why.

## Recovering a DAG

- Once the failed job is ready to be re-run, the rescue file can be used to restore the prior state of the DAG.

- Condor
- Job
- Queue

- C

- D

- A

- B

- Rescue
- File

- C

- Lecture2: Grid Job Management

- June 21-25, 2004

**Notes:**
Since jobs A and B have already completed, DAGMan will start by re-submitting job C

## Recovering a DAG (cont’d)

- Once that job completes, DAGMan will continue the DAG as if the failure never happened.

- Condor
- Job
- Queue

- C

- D

- A

- B

- D

- Lecture2: Grid Job Management

- June 21-25, 2004

## Finishing a DAG

- Once the DAG is complete, the DAGMan job itself is finished, and exits.

- Condor
- Job
- Queue

- C

- D

- A

- B

- Lecture2: Grid Job Management

- June 21-25, 2004

## Additional DAGMan Features

- Provides other handy features for job management...
- nodes can have PRE & POST scripts
- failed nodes can be automatically re-tried a configurable number of times
- job submission can be “throttled”

- Lecture2: Grid Job Management

- June 21-25, 2004

**Notes:**
PRE and post scripts are commonly used to stage data files before/after the job.
DAGMan takes note of the exit value of the scripts as well as the program. If the PRE scriptfails (exit value != 0), then neither the program nor the POST script runs, and the node is marked as failed.
If the PRE script succeeds, the program is submitted to Condor. If the program fails and there is no POST script, the DAG node is marked as failed. An exit value not equal to 0 indicates program failure. It is therefore important that the program returns the exit value 0 to indicate the program did not fail.
If the program fails and there is a POST script, node failure is determined by the exit value of the POST script. A failing value from the POST script marks the node as failed. A succeeding value from the POST script (even with a failed program) marks the node as successful.

## Another sample DAGMan submit file

- # Filename: diamond.dag
- Job A A.condor
- Job B B.condor
- Job C C.condor
- Job D D.condor
- Script PRE A top_pre.csh
- Script PRE B mid_pre.perl $JOB
- Script POST B mid_post.perl $JOB $RETURN
- Script PRE C mid_pre.perl $JOB
- Script POST C mid_post.perl $JOB $RETURN
- Script PRE D bot_pre.csh
- PARENT A CHILD B C
- PARENT B C CHILD D
- Retry C 3

- Lecture2: Grid Job Management

- June 21-25, 2004

## DAGMan, cont.

- DAGMan works with all kinds of Condor jobs.
- This means that it works fine with Grid jobs.
- In your exercises, you will submit Condor-G jobs by themselves, and using DAGMan.

- Lecture2: Grid Job Management

- June 21-25, 2004

**Notes:**
Why not just use make?
You could.  But DAGMan is easier---
 Make relies on evidence left behind on the filesystem.  What if a jobs side effect is to impact a third party database, thereby not leaving any “result” behind?
Also : Features like retries w/ exponential backoff, better info about where the dag failed (in the rescue file), throttling (helps w/ limited staging space, etc), and also where we are taking DAGMan: dynamic dags, where the workflow can change as the job evolves.   Hawk == Research DAGMan that incorporates data placement considerations in the workflow.
So Oracle Corp uses DAGMan + Condor for its nightly build and regression test of Oracle Server.