# Presentation: 019312

## CMSSW Configuration in Python

- Rick Wilkinson
- June 19, 2008

## Outline

- Intro to the python language
- Work our way through a typical job config
-  Input, modules & services, execution paths, output
- Available Tools
- Not going to refer back to old configs

## Introduction to Python

- Widely-used scripting language
- Object-oriented
- Nice syntax
- Lots of extensions
- Interfaces with C++ (boost::python)
- In transition now
- Phasing out the old custom config language
- Will be the only config supported in upcoming CMSSW_2_1_0

## More About Python

- No C++-style memory management needed
- Things are usually passed by reference
- Except native numbers and strings
- Be careful about changes that could affect others
- Control flow based on indentation
- No “endif” statements, for example
- Comments use ‘#’
- You can validate that your config file works
**  ****python yourfile.py**

## CMS Python Types

- Most of the objects you’ll create will be of a CMS-specific type
- You’ll build up composite objects by adding other objects as constructor arguments
**import FWCore.ParameterSet.Config as cms**
**jets = cms.EDProducer(“JetReco”,**
**    ****coneSize = cms.double(0.4),**
**    ****debug = cms.untracked.bool(True)**
**)**

## CMS Python Types

| DATA TYPES | COMPOSITES | EXECUTION | MODULES | MODIFIERS |
| --- | --- | --- | --- | --- |
| int32 | vint32 | Process | EDProducer | untracked |
| uint32 | vuint32 | Sequence | EDAnalyzer | ESPrefer |
| int64 | vint64 | Path | EDFilter |  |
| uint64 | vuint64 | EndPath | ESProducer |  |
| double | vdouble | Schedule | ESSource |  |
| bool | vstring | SequencePlaceholder | Source |  |
| string | VPSet |  | SecSource |  |
| PSet | VInputTag |  | Looper |  |
| InputTag | VEventID |  | Service |  |
| EventID |  |  | OutputModule |  |
| FileInPath |  |  |  |  |

## Python Modules

- (Not to be confused with framework modules!)
- A python file that is meant to be included by other files
- Lives in your **Subsystem/Package/python/**** **area, or a subdirectory of it
- Filename conventions
**_cfi.py: **Definition of a single framework object
**_cff.py: **A “fragment” of configuration commands
- To make your module visible to other python modules:
- Be sure your SCRAM environment is set up
- Go to your python area once and ‘**scramv1 build**’
- No need to re-build after each change

## Configuration files

- Controls the final job to be run
- Contains a **cms.Process** object named “**process**”
- Filename convention**:_cfg.py**
- Usually lives in **python** or **test**** **subdirectory
- Can be run by **cmsRun**
**cmsRun HLTExample_cfg.py**

## The Process object

**Process**
-   **process = cms.Process(“JPsiAnalysis”)**
**An input source**
-      **process.s****ource =  cms.Source(“FlatRandomPtGunSource”, ...)**
**Some modules**
-      **process.photons = cms.EDProducer(“PhotonProducer”,  ...)**
**Some services**
-      **process.add_(cms.ESSource(“XMLIdealGeometryESSource”,...))**
**Some execution paths**
-       **process.p1 = cms.Path(process.localreco + process.globalreco)**
**Maybe one or more output modules**
-   **process.diMuonStream = cms.EDAnalyzer(“PoolOutputModule”, ...)**

## How to Import Objects (I)

- To fetch all objects from some other module into this namespace:
**from Configuration.StandardSequences.Digi_cff import ***
- Fetches all objects from:
**Configuration/StandardSequences/****python****/Digi_cff.py**
- Fetches references, not copies
- So any changes you make affect it everywhere!
- Or you can just pick out a single object
**from Configuration.StandardSequences.Reconstruction_cff import btagging**

## How to Import Objects (II)

**import Configuration.StandardSequences.RawToDigi_cff as unpacker**
- Keeps python module object, which acts like a namespace
- Safer & faster
- Again, imports a reference, so changes affect everywhere
**unpacker.ecalDigis.DoRegional = False**
**process.ecalDigis = unpacker.ecalDigis**

## How To Import Objects (III)

- To load everything from a python module into your **process **object, you can say:
**process.load(“Configuration.StandardSequences.Sim_cff”)**

## cloning

- To make a copy of another module, and adding modifications:
**import ElectroWeakAnalysis.ZReco.zToMuMu_cfi**
**    ****as zmumu_cfi**
**zToMuMuGolden = zmumu_cfi.zToMuMu.****clone****(**
**     ****massMin = cms.double(40)**
**   ****)**

## Copying all the parameters from a PSet

** **Give the external PSet’s name as a constructor argument
- after the framework module’s name
- before the named parameters
**KtJetParameters = cms.PSet(**
**  ****strategy = cms.string(“Best”)**
**)**
**ktCaloJets = cms.EDProducer(“KtCaloJetProducer”,**
**  ****KtJetParameters****,**
**  ****coneSize = cms.double(0.7)**
**)**

## Getting Parameters in C++

- Your module will be passed an **edm::ParameterSet **in its constructor
- Values from the configuration can be retrieved in your C++ program using:
**double ptCut = pset.getParameter<double>(“ptCut”);**
**vector<int> weights = pset.getParameter<vector<int> >(“weights”);**

## Modules

- EDProducer
- Add objects into the events
- EDAnalyzer
- Analyze objects from the event
- EDFilter
- Can stop an execution path
- EDLooper
- For multi-pass looping over events
- OutputModule
- Write events to a file, and can specify filters

## Framework Services

- Examples of Framework Services are:
- Geometry, Calibration, MessageLogger
- Types are:
- ESSource
- Provides data which has an Interval Of Validity (IOV)
- ESProducer
- Creates products when IOV changes
- Service
-  Not IOV-dependent, e.g., MessageLogger

## Execution Paths

_**Sequence:**_  a series of modules or sequences, used as components for paths.
** ****trDigi = cms.Sequence(siPixelDigis + siStripDigis)**
_**Path: **_** ** Defines which modules and sequences to be run
-   **p1 = cms.Path(pdigi * reconstruction)**
_**Endpath:**_ a list of analyzers or output modules to be run after all paths have been run.
** ****outpath = cms.EndPath(FEVT)**
_**Schedule:**_ if not specified, defaults to all the paths present plus all the endpaths present
** ****process.schedule  **
**    ****= cms.Schedule(process.p1 + process.outpath)**

## Sequence Operators

**+**    means ‘follows’
-  Use when the previous module’s input isn’t required
*****       means ‘depends on’
-  Use when the module depends on a product created by the previous EDProducer.
- Enforced by the configuration processor
- Has precedence over ‘+’
- Elements can be grouped using (), so
-  **(ecalRecHits + hcalRecHits) * caloTowers**
- means that caloTowers needs both ecalRecHits and HcalRecHits, but ecal and hcal don’t depend on each other

## Filters in paths

- When an **EDFilter** is in a **path, **returning “false” will cause the path to terminate.
- Two operators, ~ and -, can modify this:
- ‘~’ Means “not”.  The path will only continue if filter returns false.
-  ‘-’ means to ignore the result of the filter, and proceed regardless.
**    ****jet500_1000 = cms.Path(**
**      ****~jet1000filter + jet500filter + jetAnalysis**
**    ****)**
- Generic Filters exist in **PhysicsTools/Utilities**

## tracked/untracked

**tracked parameters**
- Will be saved in the event data “provenance”
-  Cannot be “optional”
-  If asked for and not found, will throw exception
**untracked parameters**
-  For parameters which don’t affect results
-  Debug level, for instance
-  Can have default values
-  So they’re used for “optional” parameters, but not really considered safe practice

**Notes:**
For optional parameters, see the slide on ParameterSet Validation

## InputTags

- Used for fetching event data products
- Most commonly by **edm::Event::getByLabel()**
*** ******tagname***** = cms.InputTag(“*****label*****:*****instance*****:*****process”)***
*** ******tagname *****= cms.InputTag(*****label, instance, process)***
_**Label:**_  the name of the producer which produced it
_**Instance**_ (optional):  When a producer makes more than one product
**rpcHits = cms.InputTag(“g4SimHits” ,”MuonRPCHits”)**
**cscHits = cms.InputTag(“g4SimHits” ,”MuonCSCHits”)**
_**Process**_ (optional): The name of the job which created the data.  Useful for when RECO and HLT make the same products.

## Modifying Parameters

- You’re free to reach inside any parameter and change it
**from SimG4Core.Application.g4SimHits_cfi import ***
**g4SimHits.UseMagneticField = False**
**process.source.PGunParameters.PartID = [13,-13]**
- If the right-hand side of the expression is a CMS type, a new parameter can be created
**#  this line will fail, because of typos. Usually safer.**
**g4SimHits.useMagmeticFeeld = False**
**# this line will create a new variable with misspelled name**
**g4SimHits.useMagmeticFeeld = cms.bool(False) **

## Putting it all together

- Your final cfg.py file typically includes:
- A  **process**
- An input **source**
- Often a PoolSource, PythiaSource, or ParticleGunSource
- Possibly a source of conditions
- Either **FakeConditions_cff.py** or **FrontierConditions_GlobalTag_cff.py**
- One or more cff.py or cfi.py files
- Ideally from either **Configuration/StandardSequences **or your Analysis group
- An **OutputModule** which chooses EventContent
**Paths** and an **Endpath**
- For examples, see **Configuration/Examples **area

## Translating Old Config Files

- In FWCore/ParameterSet/python, you’ll find
- translate.py
- Creates a python file based on your config file
- Creates any included files which may be needed
- comments.py
- Transfers comments from old file
- edmTestCfgTranslation
- Translates your config.
- Tests whether the resulting ParameterSets match

## Sample 

- import FWCore.ParameterSet.Config as cms
- process = cms.Process(“SIM”)
- # Input source
- process.source = cms.Source("PoolSource",
-    fileNames = cms.untracked.vstring('file:gen.root')
- )
- # Modules
- process.load(“Configuration.StandardSequences.VtxSmearedGauss_cff”)
- process.load(“SimG4Core.Application.g4SimHits_cfi”)
- process.g4SimHits.UseMagneticField = False
- # Output
- process.load(“Configuration.EventContent.EventContent_cff”)
- process.FEVT = cms.OutputModule("PoolOutputModule",
-     process.FEVTSIMEventContent,
-     fileName = cms.untracked.string('sim.root')
- )
- # Execution paths
- process.p1 = cms.Path(process.VtxSmeared+process.g4SimHits)

## edmPythonTree

- Gives an indented dump of which files are included by which files
- Based on old config versions by Karoly Banicz & Sue Anne Koay
**edmPythonTree RecoExample_GlobalTag_cfg.py**
- + RecoExample_GlobalTag_cfg.py
-    	+ Configuration/StandardSequences/python/Services_cff.py
-       	+ SimGeneral/HepPDTESSource/python/pythiapdt_cfi.py
- 	+Configuration/StandardSequences/python/SimulationRandomNumberGeneratorSeeds_cff.py
-        		+ IOMC/RandomEngine/python/IOMC_cff.py
- 	+ Configuration/StandardSequences/python/Geometry_cff.py
-      		+ Geometry/CMSCommonData/python/cmsIdealGeometryXML_cfi.py
-      		+ Geometry/TrackerNumberingBuilder/python/trackerNumberingGeometry_cfi.py
-      		+ Geometry/TrackerGeometryBuilder/python/trackerGeometry_cfi.py
-      		+ Geometry/MuonNumbering/python/muonNumberingInitialization_cfi.py

## edmPythonSearch

- Provides a ‘grep’-like syntax to search for identifiers within included files
** ****>  ****edmPythonSearch minPt Reconstruction_cff.py**
**RecoTracker/Configuration/python/RecoTracker_cff.py:  newTrajectoryFilter.filterPset.minPt = 0.3**
**TrackingTools/TrajectoryFiltering/python/TrajectoryFilter_cff.py:      minPt = cms.double(0.9),**
**TrackingTools/TrajectoryFiltering/python/TrajectoryFilter_cff.py:      minPt = cms.double(1.0),**
**...**

## Closing Thoughts

- You probably won’t have to start writing config files or C++ modules from scratch
-  Find a similar example
-  Copy it and modify it
- Hopefully, this talk showed you:
-  What concepts CMSSW jobs use
-  Where you need to look to customize your jobs
-  What practices we consider “best”
- Most important: knowing who to ask!