# Presentation: 458972

## GRAM: Globus Resource Allocation & Management

- Globus ToolkitTM Developer Tutorial
- The Globus ProjectTM
- Argonne National LaboratoryUSC Information Sciences Instutute
- http://www.globus.org/

## Resource Management Review

- Resource Specification Language (RSL) is used to communicate requirements
- The Globus Resource Allocation and Management (GRAM) API allows programs to be started on remote resources, despite local heterogeneity
- A layered architecture allows application-specific resource brokers and co-allocators (e.g. DUROC) to be defined in terms of GRAM services

- Globus ToolkitTM Developer Tutorial: GRAM

## GRAM Components

- Globus Security
- Infrastructure

- Job Manager

- GRAM client API calls to
- request resource allocation
- and process creation.

- MDS client API calls
- to locate resources

- Query current status
- of resource

- Create

- RSL Library

- Parse

- Request

- Allocate &
- create processes

- Process

- Process

- Process

- Monitor &
- control

- Site boundary

- Client

- MDS: Grid Index Info Server

- Gatekeeper

- MDS: Grid Resource Info Server

- Local Resource Manager

- MDS client API calls
- to get resource info

- GRAM client API state
- change callbacks

- Globus ToolkitTM Developer Tutorial: GRAM

## Resource Management APIs

- Globus Toolkit has APIs for RSL, GRAM, and DUROC:
- globus_rsl
- globus_gram_client
- globus_gram_myjob
- globus_duroc_control
- globus_duroc_runtime

- Globus ToolkitTM Developer Tutorial: GRAM

## Resource Specification Language

- Much of the power of GRAM is in the RSL
- Common language for specifying job requests
- GRAM service translates this common language into scheduler specific language
- GRAM service constrains RSL to a conjunction of (attribute=value) pairs
- E.g. &(executable=“/bin/ls”)(arguments=“-l”)
- GRAM service understands a well defined set of attributes

- Globus ToolkitTM Developer Tutorial: GRAM

## globus_rsl

- Module for manipulating RSL expressions
- Parse an RSL string into a data structure
- Functions to manipulate the data structure
- Unparse the data structure into a string
- Can be used to assist in writing brokers or filters which refine an RSL specification

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Attributes For GRAM

- (executable=string)
- Program to run
- A file path (absolute or relative) or URL
- (directory=string)
- Directory in which to run (default is $HOME)
- (arguments=arg1 arg2 arg3...)
- List of string arguments to program
- (environment=(E1 v1)(E2 v2))
- List of environment variable name/value pairs

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Attributes For GRAM

- (stdin=string)
- Stdin for program
- A file path (absolute or relative) or URL
- (stdout=string)
- Stdout for program
- A file path (absolute or relative) or URL
- (stderr=string)
- Stdout for program
- A file path (absolute or relative) or URL

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Attributes For GRAM

- (count=integer)
- Number of processes to run (default is 1)
- (hostCount=integer)
- On SMP multi-computers, number of nodes to distribute the “count” processes across
- (project=string)
- Project (account) against which to charge
- (queue=string)
- Queue into which to submit job

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Attributes For GRAM

- (maxTime=integer)
- Maximum wall clock or cpu runtime (schedulers’s choice) in minutes
- (maxWallTime=integer)
- Maximum wall clock runtime in minutes
- (maxCpuTime=integer)
- Maximum CPU runtime in minutes

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Attributes For GRAM

- (maxMemory=integer)
- Maximum amount of memory for each process in megabytes
- (minMemory=integer)
- Minimum amount of memory for each process in megabytes

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Attributes For GRAM

- (jobType=value)
- Value is one of “mpi”, “single”, “multiple”, or “condor”
- mpi: Run the program using “mpirun -np <count>”
- single: Only run a single instance of the program, and let the program start the other count-1 processes.
- multiple: Start <count> instances of the program using the appropriate scheduler mechanism
- condor: Start a <count> Condor processes running in “standard universe”

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Attributes for GRAM

- (gramMyjob=value)
- Value is one of “collective”, “independent”
- Defines how the globus_gram_myjob library will operate on the <count> processes
- collective: Treat all <count> processes as part of a single job
- independent: Treat each of the <count> processes as an independent uniprocessor job
- (dryRun=true)
- Do not actually run job

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Attributes for GRAM

- (save_state=yes)
- Causes the jobmanager to save job state/information to a persistent file on disk
- Recover from a jobmanager crash
- New in Globus Toolkit v2.0
- (two_phase=<int>)
- Implement a two-phase commit for job submission and completion
- <int>=seconds to wait before job times out
- New in Globus Toolkit v2.0

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Attributes for GRAM

- (restart=<old jm contact>)
- Start a new jobmanager but instead of submitting a new job, start watching over an existing job.
- New in Globus Toolkit v2.0
- (stdout_position=<int>)
- (stderr_position=<int>)
- specified as part of a job restart
- restart file streaming from this byte
- New in Globus Toolkit v2.0

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Substitutions

- RSL supports simple variable substitutions
- Substitutions are declared using a list of pairs
- (rslSubstitution=(SUB1 val1)(SUB2 val2))
- A substitution is invoked with $(SUB)
- Processing order:
- Within scope, processed left-to-right,
- Outer scope processed before inner scope
- Variable definition can reference previously defined variables

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Substitution Example

- This
- &(rslSubstitution=(URLBASE “ftp://host:1234”))
-   (rslSubstitution=(URLDIR $(URLBASE)/dir))
-   (executable=$(URLDIR)/myfile)
- is equivalent to this
- &(executable=ftp://host:1234/dir/myfile)

- Globus ToolkitTM Developer Tutorial: GRAM

## GRAM DefinedRSL Substitutions

- GRAM defines a set of RSL substitutions before processing the job request
- Machine Information
- GLOBUS_HOST_MANUFACTURER
- GLOBUS_HOST_CPUTYPE
- GLOBUS_HOST_OSNAME
- GLOBUS_HOST_OSVERSION

- Globus ToolkitTM Developer Tutorial: GRAM

## GRAM DefinedRSL Substitutions

- Paths to Globus
- GLOBUS_LOCATION
- Miscellaneous
- HOME
- LOGNAME
- GLOBUS_ID

- Globus ToolkitTM Developer Tutorial: GRAM

## GRAM Examples

- The globus-job-run client is a sample GRAM client that integrates GASS services for executable staging and standard I/O redirection, using command-line arguments rather than RSL.
- % globus-job-run pitcairn.mcs.anl.gov /bin/ls
- % globus-job-run pitcairn.mcs.anl.gov –s myprog
- % globus-job-run pitcairn.mcs.anl.gov \
-          –s myprog –stdin –s in.txt –stdout –s out.txt

- Globus ToolkitTM Developer Tutorial: GRAM

## GRAM Examples

- The globusrun client is a more involved tool that allows complicated RSL expressions.
- % globusrun –r pitcairn.mcs.anl.gov –f myjob.rsl
- % globusrun –r pitcairn.mcs.anl.gov \
-       ‘&(executable=myprog)’

- Globus ToolkitTM Developer Tutorial: GRAM

## globus_gram_client

- globus_gram_client_job_request()
- Submit a job to a remote resource
- Input:
- Resource manager contact string
- RSL specifying the job to be run
- Callback contact string, for notification
- Output:
- Job contact string

- Globus ToolkitTM Developer Tutorial: GRAM

## Finding The Gatekeeper

- globus_gram_client_job_request() requires a _resource manager contact_ string to find the gatekeeper
- 		hostname[:port][/service][:subject]
- hostname – host of gatekeeper
- required
- port – port on which gatekeeper is listening
- defaults to well known port = gsigatekeeper = 2119
- service – gatekeeper service to invoke
- defaults to “jobmanager”
- subject – security subject name of gatekeeper
- Defaults to standard host cert form: “.../cn=host/hostname”
- Applies fuzzy match to deal with interface names, etc.

- Globus ToolkitTM Developer Tutorial: GRAM

## Job Contact

- globus_gram_client_job_request() returns a _job contact_
- Opaque string
- Other globus_gram_client_*() functions use the job contact to find the right job manager to which requests are made
- Job contact string can be passed between processes, even on different machines

- Globus ToolkitTM Developer Tutorial: GRAM

## globus_gram_client

- globus_gram_client_job_status()
- Check the status of the job
- UNSUBMITTED, PENDING, ACTIVE, FAILED, DONE, SUSPENDED
- Can also get job status through callbacks
- globus_gram_client_callback_{allow,disallow,check}()
- globus_gram_client_job_cancel()
- Cancel/kill a pending or active job

- Globus ToolkitTM Developer Tutorial: GRAM

## globus_gram_client

- globus_gram_client_job_signal()
- Controls the jobmanager
- COMMIT_REQUEST*
- submit job
- COMMIT_END*
- Cleanup job
- COMMIT_EXTEND*
- Wait additional N seconds
- * when jobs have “(two_phased=yes)”

- Globus ToolkitTM Developer Tutorial: GRAM

## globus_gram_client

- globus_gram_client_job_signal(), continued
- STDIO_UPDATE
- Allows client to submit an RSL that changes some I/O attributes of the job
- stdout, stderr, stdout_position, stderr_position, remote_io_url
- STDIO_SIZE
- verify that streamed I/O has been completely received
- STOP_MANAGER
- Tells JM to exit, but leave the job running

- Globus ToolkitTM Developer Tutorial: GRAM

## State Change Callbacks

- GRAM managed job can be in the states:
- Unsubmitted, Pending, Active, Failed, Done, Suspended
- GRAM client can register for asynchronous state change callbacks
- Registration can be done during submission
- Globus_gram_client_job_request()
- Registration can be done later by any process, using the job contact
- globus_gram_client_job_callback_register()

- Globus ToolkitTM Developer Tutorial: GRAM

## globus_gram_client

- globus_gram_client_callback_allow()globus_gram_client_callback_disallow()globus_gram_client_callback_check()
- Create/destroy a client port to listen for asynchronous state change callbacks
- Callback to local function on state change
- globus_gram_client_job_callback_register()globus_gram_client_job_callback_unregister()
- Register with job manager to receive callbacks

- Globus ToolkitTM Developer Tutorial: GRAM

## globus_gram_myjob

- When a set of processes in a single job startup, they may need to self organize
- How many processes in the job?
- What is my rank within the job?
- Simple send/receive between job processes.
- This API is a minimal set of functions to allow this self organization
- This is a bootstrapping library.  It is NOT meant to be a general purpose message passing library for use by applications

- Globus ToolkitTM Developer Tutorial: GRAM

## DUROC Review

- Simultaneous allocation of a resource set
- Handled via optimistic co-allocation based on free nodes or queue prediction
- In the future, advance reservations will also be supported
- globusrun will co-allocate specific multi-requests
- Uses a Globus component called the Dynamically Updated Request OnlineCo-allocator (DUROC)

- Globus ToolkitTM Developer Tutorial: GRAM

## A Co-allocation Multirequest

- +( & (resourceManagerContact=
-       *** “flash.isi.edu:2119/jobmanager-lsf:/O=Grid/.../CN=host/flash.isi.edu”)
-         (count=1)
-         (label="subjob A")
-         (executable= my_app1)
-    )
-    ( & (resourceManagerContact=
-          ***“sp139.sdsc.edu:2119:/O=Grid/.../CN=host/sp097.sdsc.edu")
-         (count=2)
-         (label="subjob B")
-         (executable=my_app2)
-    )

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Attributes For DUROC

- (subjobStartType=value)
- Alters the startup barrier mechanism
- values are “strict-barrier”, “loose-barrier”, “no-barrier”
- (subjobCommsType=value)
- values are “blocking-join” and “independent”
- if value is set to “independent”, the subjob won’t be seen from the other subjobs when doing inter-subjob communication.

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL Attributes For DUROC

- (label=string)
- Identifier for this subjob
- (resourceManagerContact=string)(resourceManagerName=string)
- Resource manager to which to submit a subjob

- Globus ToolkitTM Developer Tutorial: GRAM

## globus_duroc_control

- Submit a multi-request
- Edit a pending request
- Add new nodes, edit out failed nodes
- Commit to configuration
- Delay to last possible minute
- Barrier synchronization
- Initialize computation
- Bootstrap library
- Monitor and control collection

- Globus ToolkitTM Developer Tutorial: GRAM

## globus_duroc_runtime

- globus_duroc_runtime_barrier()
- All processes in DUROC job must call this
- It will wait until the DUROC control module releases all processes from the barrier
- globus_duroc_runtime_inter_subjob_*()
- Bootstrap library between subjobs
- globus_duroc_runtime_intra_subjob_*()
- Bootstrap library within a subjob

- Globus ToolkitTM Developer Tutorial: GRAM

## Job Manager Files 

- Job Manager Files

- X509_USER_PROXY

- GASS_CACHE

- Gatekeeper

- stdout

- stderr

- Staged
- EXE

- Staged
- stdin

- Submission

- Scheduler
- Desc.
- Exe=x
- Args=y
- Env=z

- Job
- status

- monitoring

- UP

- UP

- Client

- GRIS

- JOB

- Jobmanager

- Globus ToolkitTM Developer Tutorial: GRAM

## GRAM exercises

- Note: GRAM has three APIs:
- client, myjob, job_manager
- Most users will never use job_manager API
- Go to the “gram” subdirectory
- Documentation
- http://www.globus.org/gram
- Follow instructions in the file README

- Globus ToolkitTM Developer Tutorial: GRAM

## DUROC exercises

- Note: DUROC has two APIs:
- control, runtime
- Go to the “duroc” subdirectory
- Documentation
- http://www.globus.org/duroc
- Follow instructions in the file README

- Globus ToolkitTM Developer Tutorial: GRAM

## RSL exercises

- Go to the “rsl” subdirectory
- Documentation
- http://www.globus.org/gram/rsl
- Follow instructions in the file README

- Globus ToolkitTM Developer Tutorial: GRAM

## Changes: 1.1.x  2.0

- One-and-only-once submission
- Through 2 phase commit signal
- Recoverability
- Job manager can be restarted
- Restart/redirect stdout/err
- Generalized signaling of job manager

- Globus ToolkitTM Developer Tutorial: GRAM

## Future “GRAM 1.6”

- Asynchronous client API
- New RSL attribute to pass through scheduler specific commands
- No more piggy-backing on the environment attributes
- File staging
- scratch dir, input, output
- Advanced output management
- Stream/store stdout and stderr to multiple destinations

- Globus ToolkitTM Developer Tutorial: GRAM

## Interesting Issues

- The Globus Toolkit does not include a resource broker or a metascheduler!
- We have helped many people to build these using GRAM and MDS services; many now exist.
- Condor-G, DRM, PUNCH, Nimrod/G, Cactus, AppLeS,

- Globus ToolkitTM Developer Tutorial: GRAM