# Presentation: 658346

## USCMS Integration Grid Testbed

- M. Anzar Afaq
- FermiLab

- Deployment, Operation and Troubleshooting

## Introduction

- Test, Integration and Production Grid Test Beds
- Tier-I/Tier-II resources combined into Grid effort.
- About 230 CPU (750 MHz equivalent, RedHat Linux 6.1)
- Additional 80 CPU at 2.4 GHz running RedHat Linux 7.X
- About 5 TB local disk space plus Enstore Mass storage at FNAL using dCache.
- Caltech, Fermilab, U Florida, UC SanDiego, (UW Madison support).
- CERN/LCG (with 80 CPUs) is joining.
- Dedicated resources, still Grid ?.

## Software Components

- Job Creation and submission
- MCRunJob
- MOP (Condor_G, DAGMAN)
- Applications
- cmsim 125, ORCA 6_2_0, Objectivity, (ROOT)
- Middleware
- Globus and Condor as (VDT 1.1.3 Sever and Client)
- Batch systems
- Condor and fbsng (simpler is better!)
- Monitoring tools
- MonaLisa (CPU load, network usage, disk space).
- Condor_q, NGOP (Farm and job queue monitoring).
- Data Transfer tools (Tony’s scripts !)
- Not very demanding.
- Working/Thinking on better approaches
- Direct (GridFTP) transfer to mass-storage via dCache
- Direct transfer to CERN mass storage (CASTOR).

## Progressing...

**End-to-End ****integration** working fine.
- Smooth start
- 10%  50%  100% in a week
- Throttled submission
- Few sites 1.5  2  3  4 (5th site soon)
- Smaller farms  Larger farms
- Excellent throughput ~20K per day
- SC2002 show
- Running two assignments
- Assignment to produce 1 Million “eGamma Bigjets” events by Christmas 2002, all steps (720K done).
- Assignment to produce 500K additional events, cmsim step only (~50k done!).

## Results (so far)

*Time to process *
*1 event: *
*500 sec @ 750 MHz*
*Speedup: *
*Avg factor of 100 speedup during current run*
*Resources: Approximately 230 CPU @750 MHz equiv. *
*Sustained efficiency: *
*about 43.5%*

## Success counts..

## Limitations..

- Breakdowns----scalability ?
- Unforeseen issues (not any more!)
- Larger submissions (30+)
- Parallel g-u-c (FTSH)
- NFS timeouts
- Larger farms
- Disk management
- Failing job’s garbage collection
- Large number of jobs (kills the system, YP server)
- Configuration issues (condor killing a bug with Gun)
- Mainly bottlenecks (nfs, sockets, disk)
- No dead-end (work arounds exist)
- Throttling, code “adjustments”, tune-ups
- Cleanup and start over

## Troubleshooting.. 

- Several middleware and application software issues/bugs found/fixed/worked-around.
- (RefDB, MCRunJob, MOP, Applications)
- Condor_G/gahp_server failure at higher number of jobs in queue (major issue).
- Gahp_server stop communicating with sites.
- Gahp_server spawns a large number of threads.
- Orphaned jobs (cpu-time wastage/garbage)
- Held jobs (need re-submission).
- Significantly hampers efficiency
- 35-40% of all failures.
- Condor team is working on the problem.
- (kill and restart Condor_G)  ----- Fault Tolerance.

## With ~350 jobs in queue, 17 crashes of gahp_server in ~5 hours

## Operation and Effort 

- End-To-End job tracking is tough.
- Have to look into several log files and queues.
- Need “baby sitting”  (reducing with time).
- Monitoring and alarming helps
- Shift duties
- Need to spread experience
- man-power
- Missing “scheduler” at Grid level
- Manual submission/Resource allocation
- Improving as we go.
- Shift manual is ready 		(http://home.fnal.gov/~anzar/MOP_MASTER/master_manage.html)
- Shift plan in place !
- Deployment guide (~ ready)

## Conclusions & Plans 

- Happy with the results 
- USCMS “production grid” in January 2003
- Improved middleware
- Bugs fixed
- Grid level Scheduling (several options)
- Future development needs understood
- Changes to MCRunJob
- Better tracking
- Changes to MOP (The Show Master)
- More generalized interface
- Multi-site job submission
- BOSS in picture.
- Better organization and closer cooperation
- Shifts, Policies, Bug reporting