# Presentation: 201910

## RunJob in CMS

- Greg Graham
- Discussion Slides

## RunJob in CMS

- RunJob is an Application Configuration and Job Creation Tool
- RunJob uses metadata to abstract application steps and bolt them together into a workflow (Configurators)
- RunJob uses metadata to abstract service invocations and iterators to parallelize job creation (other Configurators)
- RunJob has an abstract model of jobs.
-  These can be translated into concrete jobs for a variety of environments (local or grid)  later depending on what modules are plugged in

## RunJob in CMS

- RunJob does job creation in scripts (parallelized or not)
- RunJob scripts support external service invocations, looping over data things, inline job submission, metadata constraint modeling among applications and services, persistence, and conventional provenance.
- Scripts can be factored into simpler scripts and contexts.  Contexts contain site level details, administrative details, logical details about constraints on the simpler scripts.
- Saving the context scripts can enable “meta-provenance” in the sense that recording constraints provides clues as to why certain parameters are set the way they are instead of just their value.

## RunJob in CMS

- Specific Uses
- Official Monte Carlo Generation
- Over 30 sites, over 75 million GEANT events, controlled centrally by central RefDB (pull model)
- Data Challenge 2004
- Used at CERN to govern online reconstruction of simulated data at 25 Hz.
- Used at FNAL to create physics plots from reconstructed data transferred there.

## RunJob in CMS

- Extensions to RunJob
- XMLP: An XML persistency mechanism
- shREEK: Runtime extensions to RunJob, monitoring, and late job building.
- logger: An event logger
- scriptObjects: Abstract descriptions of jobs that serve as input to code generators producing actual jobs for specific environments
- CAST: A GUI for construction Configurators and workflows.

## Databases

**& Catalogs**

**Local/Grid **
**Resources**

**Linker:** A container for Configurators that enables
- constraint propagation, workflow modeling, provenance,
- scripting, and context resolution of scripts

**Configurator**: An Object Oriented, Metadata
- based description of services or applications with
- APIs for extending behaviors to many environments

**Applications**

**Execution**
**Managers**

**RunJob Scripts**

**RunJob Contexts**

- User

- Admin

**ScriptObjects**

## Relationship to Physics

- Physics Parameters Specification
- By physics group: in a context file written by a special user designated by the physics group
- Members working within a physics group will inherit these parameters by using a common context
- Constrains multiple applications according to the same set of constants
- By knowledgeable user: in a personal MCRunjob script file or in a personal context file.
- By novice user: on the command line of a pre-existing MCRunjob script written by a knowledgeable user.
- In this case, the novice is restricted to specifying parameters foreseen by the author of the MCRunjob script in question.

## Linker: A container for Configurators that enables 

- constraint propagation, workflow modeling, provenance,
- scripting, and context resolution of scripts

**RunJob Scripts**

**RunJob Contexts**

- User

- Admin

**-User’s Job Description**
-   eg- attach Pythia
-         attach BatchManager
**-Physics Parameters of User Interest**
-   eg- TopMass=275 GeV/c2
**-Specification of Which Context **
**                    ****To Use**

**-Physics Parameters set by Physics Group**
-    eg- HiggsMass=115 GeV/c2
-           TopMass=273 GeV/c2
-           Decay Parameters from a Control DB
**-Physical Description of Site Resources**
-    eg-  BatchManager=FBS
**-Configuration of Site Resources**
-    eg- FBSQueueName=CMS; Submit Jobs Inline

**-Combine all specified contexts**
**-Fully Specified Workflow Description**
-     eg- attach ControlDB
-           attach Pythia
-             HiggsMass=115 GeV/c2
-              TopMass=275 GeV/c2
-              Decay Parameters from a Control DB
-           attach FBS
-              FBS QueueName=CMS; Submit Jobs Inline

**-Read User Script**

**-Read Context(s) specified by User**

**Result Text**

**Results from**
**Configurators**

## Control DB

**Linker:** A container for Configurators that enables
- constraint propagation, workflow modeling, provenance,
- scripting, and context resolution of scripts

**Configurators**: An Object Oriented, Metadata
- based description of services or applications with
- APIs for extending behaviors to many environments

**Pythia**

**FBS**

**-Fully Specified Workflow Description**
-     eg- attach ControlDB
-           attach Pythia
-             HiggsMass=115 GeV/c2
-              TopMass=275 GeV/c2
-              Decay Parameters from a Control DB
-           attach FBS
-              FBS QueueName=CMS; Submit Job

- MySQL

- attach ControlDB
- Get decay parameters

-   **return decay **
**parameters: {x}**

- attach Pythia
-   HiggsMass=115
-   TopMass=275
-   DecayParameters {x}

**return Pythia**
**scriptObject**

- attach FBS
- FBSQueueName=CMS
- Submit Job Now

**FBS Submit**
**    ****Results**

**FBS Submit **
**    ****Results**

- Queue
- CMS

## Relationship to Physics

- Provenance
- By specification of fully reduced workflow specification (all contexts resolved, job level)
- By specification of contexts actually used in modifying a workflow (why does a constant have value x?)
- By specification of