# Presentation: 243223

## CMS Challenge Problem – Review and Lessons Learned

## Grid

**Grid**

**User/Application**

**Fabric (****processing, storage, communication****)**

## The CMS Production Story

**A collaboration between:**
**Physicists & Computer Scientists**
**Vladimir Litvin ****(Caltech CMS)**
**Scott Koranda, Bruce Loftis, John Towns ****(NCSA)**
**Miron Livny, Peter Couvares, Todd Tannenbaum, Jamie Frey ****(UW-Madison Condor)**
**Software**
**CMS, Condor, Globus, PBS, UniTree ...**

## Challenges of a CMS Run

- CMS run naturally divided into two phases

- Specific challenges
- each run generates ~100 GB of data to be moved and archived elsewhere
- many, many runs necessary
- simulation & reconstruction jobs at different sites
- this can require major human effort starting & monitoring jobs, moving data

- physics reconstruction from simulated data
- 100’s of jobs per run
- jobs coupled via Objectivity database access
- ~100 GB data archived

- Monte Carlo detector response simulation
- 100’s of jobs per run
- each generating ~ 1 GB
- all data passed to next phase and archived

## CMS Run on the Grid

**CMS Run on the Grid**

- Caltech CMS staff prepares input files on local workstation
- Pushes “one button” to submit a DAGMan job to Condor
- DAGMan job at Caltech submits secondary DAGMan job to UW Condor pool (~700 CPUs)
- Input files transferred by Condor to UW pool using Globus GASS file transfer

**Caltech workstation**

**Input files via ****Globus GASS**

**UW Condor pool**

## CMS Run on the Grid

- Secondary DAGMan job launches 100 Monte Carlo jobs on Wisconsin Condor pool
- each job runs 12~24 hours
- each generates ~1GB data
- Condor handles checkpointing & migration
- no staff intervention

**Globus**

**Globus**

## CMS Run on the Grid

- When each Monte Carlo job completes, data automatically transferred to UniTree at NCSA by a POST script
- each file ~ 1 GB
- transferred by calling Globus-enabled FTP client “gsiftp”
- NCSA UniTree runs Globus-enabled FTP server
- authentication to FTP server on user’s behalf using digital certificate

**100 Monte Carlo jobs on Wisconsin Condor pool**

**NCSA UniTree with**
**Globus-enabled FTP server**

**100 data files transferred via ****Globus gsiftp**** (~ 1 GB each)**

## CMS Run on the Grid

- When all Monte Carlo jobs complete, Condor DAGMan at UW reports success to DAGMan at Caltech
- DAGMan at Caltech submits another Globus-universe job to Condor to stage data from NCSA UniTree to NCSA Linux cluster
- data transferred using Globus-enabled FTP
- authentication on user’s behalf using digital certificate

**Condor starts job via Globus jobmanager on cluster to stage data**

## CMS Run on the Grid

- Condor DAGMan at Caltech launches physics reconstruction jobs on NCSA Linux cluster
- job launched via Globus jobmanager on NCSA cluster
- no user intervention required
- authentication on user’s behalf using digital certificate

**Master starts reconstruction jobs via Globus jobmanager on cluster**

## CMS Run on the Grid

- When reconstruction jobs at NCSA complete, data automatically archived to NCSA UniTree
- data transferred using Globus-enabled FTP
- After data transferred, DAGMan run is complete, and Condor at Caltech emails notification to staff

**data files transferred via ****Globus gsiftp**** to UniTree for archiving**

## CMS Run Details

- Condor + Globus
- allows Condor to submit jobs to remote host via a Globus jobmanager
- any Globus-enabled host reachable (with authorization)
- Condor jobs run in the “Globus” universe
- use familiar Condor classads for submitting jobs

**universe     = globus**
**globusscheduler **
**             ****= beak.cs.wisc.edu/jobmanager-**
**                  ****condor-INTEL-LINUX**
**environment  = CONDOR_UNIVERSE=scheduler**
**executable   = CMS/condor_dagman_run**
**arguments    = -f -t -l . -Lockfile cms.lock**
**              ****-Condorlog cms.log -Dag **
**              ****cms.dag -Rescue cms.rescue**
**input        = CMS/hg_90.tar.gz**
**remote_initialdir **
**             ****= Prod2001**
**output       = CMS/hg_90.out**
**error        = CMS/hg_90.err**
**log          = CMS/condor.log**
**notification = always**
**queue**

## CMS Run Details

- At Caltech, DAGMan ensures reconstruction job B runs only after simulation job A completes successfully & data is transferred
- At UW, no job dependencies, but DAGMan POST scripts used to stage out data

**# Caltech: main.dag**
**Job jobA_632 Prod2000/hg_90_gen_632.cdr**
**Job jobB_632 Prod2000/hg_90_sim_632.cdr**
**Script pre jobA_632 Prod2000/pre_632.csh**
**Script post jobB_632 Prod2000/post_632.csh**
**PARENT jobA_632 CHILD jobB_632**

**# UW: simulation.dag**
**Job sim_0 sim_0.cdr**
**Script post sim_0 post_0.csh**
**Job sim_1 sim_1.cdr**
**Script post sim_1 post_1.csh**
**# ...**
**Job sim_98 sim98.cdr**
**Script post sim_98 post_98.csh**
**Job sim_99 sim_99.cdr**
**Script post sim_99 post_99.csh**

## 5) UW DAGMan reports success to Caltech DAGMan

**5) UW DAGMan reports success to Caltech DAGMan**

**7) gsiftp fetches data from UniTree**

**NCSA UniTree - Globus-enabled FTP server**

**4) data files transferred via gsiftp, ~ 1 GB each**

**Secondary Condor DAGMan  job on UW pool**

**3) Monte Carlo jobs on UW Condor pool**

**2) Launch secondary DAGMan job on UW pool; input files via Globus GASS**

**Caltech workstation**

**6) DAGMan starts reconstruction jobs via Globus jobmanager on cluster**

**8) Processed objectivity database stored to UniTree**

**9) Reconstruction job reports success to DAGMan**

**1) Submit DAGMan to Condor**

## Lesson #1 Team Effort

- Close collaboration between the developers of the application, middleware and fabric software
- Mutual trust and respect
- Effective coordination
- Common goals

## Lesson #2 Big Payoff

- Computational Grids can deliver effective solutions to data and processing intensive applications
- Local control at submission site
- Powerful middleware available
- Effective harnessing of distribute dresources

## Lesson #3 - it isn’t easy

- Delivering 24-7 capabilities as required by High Throughput applications is a challenge
- Integrated data and storage management
- Monitoring and exception/error handling

## agile!!! 

**Do not be**
**picky, be**
**agile!!!**** **