# Presentation: 246360

## RunJob

- Greg Graham - FNAL CD/CMS
- PPDG Collaboration Meeting
- 15-Dec-2003

## What is RunJob?

- RunJob is a tool to build Grid or local jobs based on user specification of metadata with support for:
- multi-application workflows
- parallelization
- instrumentation for monitoring and control
- RunJob is script based so that a RunJob macro script contains provenance as well as procedural information on how to build jobs.
- XML standard for the RunJob macro scripts available soon
- Variants are being used at DZero, CMS, and CDF.
- A project, “RunJob,” is underway at Fermilab CD to guide common development where appropriate

## Modular Design

- Configurator: Each application step or external service or database lookup is represented internally as
- A set of metadata that describes the application or service invocation together
- dynamically registered functions that execute transformations on the metadata during job building
- Linker: creates the runtime bits that execute the job.
- Runs the framework which executes the Configurator functions and generates scriptObjects
- Collects scriptObjects for each target environment
- eg- VDL(Chimera), Shell scripts, MOP (Condor DAGs)
- Each target environment is also modeled as a Configurator

## RunJob Architecture

## Metadata Constraints

- Metadata which describes the job can be explicitly constrained
- Attributes can be set on key-value pairs which specify metadata references instead of straight  values.
- Example of metadata constraints:
- site dependent runtime conditions
- physics group favored calibration parameters.
- Configurators can have explicit dependencies which are then used to express either data flow or metadata flow.

## Delegation

- Delegation is a mechanism which allows multiple target runtime environments to be targeted simultaneously.
- Metadata transforming “framework calls”, normally handled locally by each Configurator, are handled by “Delegate” Configurators.
- Eg- VDLConfigurator, MOPDagGen, ImpalaLite(CMS), DZero runtime environment, etc.
- Multiple delegates allow targeting of multiple or composable environments.
- Eg- Including ImpalaLite and MOPDagGen simultaneouly generates DAGs that target the CMS runtime environment.

## Contexts

- A collection of rules specifying metadata definitions, constraints, Configurator dependencies, and delegations is called a Context.
- A context modifies “on the fly” an existing RunJob macro script to tailor it to specific environments.
- Contexts should be composable
- Eg- “CMSProduction.ctx:MOP.ctx:FNAL.ctx:JetMet.ctx”
- This is still an open issue in RunJob
- Certain pieces of the context can be tied to the extended VOMS proxy (Yujun Wu, FNAL)
- Ie- A runjob-context-init client can be run after voms-proxy-init to obtain proper context information.

## Runjob Contexts

## Conclusion

- RunJob is a modular framework for job creation
- Script driven, can be used as provenance (XML coming)
- Ability to express data flow as well as metadata flow
- Ability to specify constraints on metadata
- Delegation for multiple composable runtime environments
- Composable contexts for on-the-fly modification of macro scripts.
- Near term plans
- CLARENS based Web Services decomposition of Linker
- Explore context serving client on top of VOMS extended proxy
- Full XML Schema specification of RunJob macro scripts and contexts.
- Updated documents circulated for comment.
- Existing documentation: CHEP 2003 (TUCT007)
- But this even is already dated.