# Presentation: 323605

## 
                MONARC Project Status Report
              http://www.cern.ch/MONARC

Harvey NewmanCalifornia Institute of Technlogy
    http://l3www.cern.ch/monarc/monarc_lehman151100.ppt
      DOE/NSF Joint Review of Software and Computing, BNL 
				  November 15, 2000

**                ****MONARC Project Status Report**
**              ****http://www.cern.ch/MONARC**
**Harvey Newman****California Institute of Technlogy**
**    ****http://l3www.cern.ch/monarc/monarc_lehman151100.ppt**
**      ****DOE/NSF Joint Review of Software and Computing, BNL **
**	****	****	****	****  ****November 15, 2000**

##        MONARC: Common Project 

**M****odels ****O****f ****N****etworked ****A****nalysis                                  At ****R****egional ****C****enters**
**Caltech, CERN, Columbia, FNAL, Heidelberg, **
**Helsinki, INFN, IN2P3, KEK, Marseilles, MPI Munich, Orsay, Oxford, Tufts**
_**PROJECT GOALS**_
**Develop “Baseline Models”**
**Specify the main parameters   ****	****	**** characterizing the Model’s ****performance: throughputs, latencies **
**Verify resource requirement baselines:            (computing, data handling, networks)**** **
_**TECHNICAL GOALS**_
** ****Define the ****Analysis Process**
** ****Define ****RC Architectures and Services**
** ****Provide ****Guidelines for the final Models**
** ****Provide a ****Simulation Toolset ****for Further ****	****Model studies**

**2.5 Gbits/s**

**2.5 Gbits/s**

**Univ 2**

**CERN**
**700k SI95 **
**1000+ TB Disk;**
** ****Robot**

**Tier2 Ctr**
**~35k SI95**
**  ****~100 TB Disk**
** ****Robot**

**FNAL/BNL**
**167k SI95**
**650 Tbyte**** Disk; Robot**

**622 Mbits/s**

**N X2.5 Gbits/s**

- Optional Air Freight

**2.5 Gbits/s**

**2.5 Gbits/s**

**Univ**
**1**

**Univ**
**M**

_**Model Circa 2005 or 2006**_

##        MONARC History 

**Spring 1998    ****	****First Distributed Center Models (Bunn; Von Praun)**
**6/1998****	****Presentation to LCB; Project Assignment Plan**
**Summer 1998****	****MONARC Project Startup (ATLAS, CMS, LHCb)**
**9 - 10/1998****	****Project Execution Plan; Approved by LCB**
**1/1999****	****First Analysis Process to be Modeled**
**2/1999****	****First Java Based Simulation Models (I. Legrand)**
**Spring 1999****	****Java2 Based Simulations; GUI**
**4/99; 8/99; 12/99****	****Regional Centre Representative Meetings**
**6/1999****	****Mid-Project Progress Report****	****Including MONARC Baseline Models**
**9/1999****	****Validation of MONARC Simulation on Testbeds****	****Reports at LCB Workshop (HN, I. Legrand)**
**1/2000****	****Phase 3 Letter of Intent (4 LHC Experiments)**
**2/2000****	****Six Papers and Presentations at CHEP2000:****	****  *****D385, F148, D127, D235, C113, C169***
**3/2000****	****Phase 2 Report**
**Spring 2000****	****New Tools: SNMP-based Monitoring; S.O.N.N.**
**5/2000****	****Phase 3 Simulation of ORCA4 Production;****	****Begin Studies with Tapes**
**Spring 2000****	****MONARC Model Recognized by Hoffmann WWC Panel;****	****Basis of Data Grid Efforts in US and Europe **

##           MONARC Working Groups/Chairs

**                    **
_**Analysis Process Design WG**__**	**_** ****P. Capiluppi (Bologna, CMS)****	**Studied the analysis workload, job mix and profiles, time to complete the 	reco. and analysis jobs. Worked with the Simulation WG to verify that the 	specified resources in the models could handle the workload.
_**Architectures WG**_** ****	****	****   ****Joel Butler (FNAL, CMS)****	**Studied the site and network architectures, operational modes and 	services provided by Regional Centres, data volumes stored and 		analyzed, candidate architectures for CERN, Tier1 (and Tier2) Centres
_**Simulation WG**_**	****         ****K. Sliwa (Tufts, ATLAS)****	**Defined the methodology, then (I. Legrand et al.) designed, built and 		further developed the simulation system as a toolset for users. 	Validated the simulation with the Testbeds group.
_**Testbeds WG**_**                 ****	****   ****L. Luminari (Rome, ATLAS) ****	**Set up small and larger prototype systems at CERN, several INFN   and US sites and Japan, and used them to characterize the performance	of the main elements that could limit throughput in the simulated systems
_**Steering Group**_**	****	****Laura Perini  (Milan, ATLAS) **
**                  ****	****Harvey Newman (Caltech, CMS)**
** ****Regional Centres Committee**

##   Regional Center Architecture   Example by I. Gaines 

- Tape Mass Storage
-  & Disk Servers
- Database Servers

**Physics**
**Software **
**Development**

**R&D Systems**
**and Testbeds**

**Info servers**
**Code servers**

**Web Servers**
**Telepresence**
**Servers**

**Training**
**Consulting**
**Help Desk**

**Production**
_**Reconstruction**_
**Raw/Sim ******** ESD**
**Scheduled,**
** ****predictable**
**experiment/**
**physics groups**

**Production**
_**Analysis**_
**ESD ******** AOD**
**AOD ******** DPD**
**Scheduled**
**Physics groups**

**Individual **
_**Analysis**_
**AOD ******** DPD**
**and plots**
**Chaotic**
**Physicists**

**Desktops**

**Tier 2**

**Local **
**institutes**

**CERN**

- Tapes

- Data Import

- Data Export

## MONARC Analysis Model

_**Hierarchy of Processes (Experiment, Analysis Groups, Individuals)**_

**Reconstruction**

**Selection**

**Analysis**

**Re-****processing**
**3 per year**

**Iterative selection**
**Once per month**

**Different Physics cuts**
**& MC comparison**
**~Once per day**

**Experiment-**
**Wide Activity**
**(10****9**** events)**

**~20 Groups’**
**Activity**
**(10****9 ******** ****10****7**** events)**

**~25 Individual**
**per Group**
**Activity**
**(10****6**** –10****8**** events)**

**New detector **
**calibrations**
**Or understanding**

**Trigger based and**
**Physics based**
**refinements**

**Algorithms applied**
**to data**
**to get results**

**3000 SI95sec/event**
**1 job year**

**3000 SI95sec/event**
**3 jobs per year**

**25 SI95sec/event**
**~20 jobs per month**

**10 SI95sec/event**
**~500 jobs per day**

**Monte Carlo**

**5000 SI95sec/event**

##      Simulation Validation: LAN   Measurements (Y. Morita et al)

**Machine A:**
**	****Sun Enterprise450 (400MHz 4x CPU)**
**Machine B:**
**	****Sun Ultra5 (270MHz): The Lock Server**
**Machine C:**
**	****Sun Enterprise 450 (300MHz 2x CPU)**
**Tests:**
** ****(1) Machine A local (2 CPU)**
** ****(2) Machine C  local (4 CPU)**
** ****(3) Machine A (client) and Machine C (server)**
**	****  ****number of client processes: 1, 2, 4, ..., 32**

**Job on**
** ****Machine A**

- Time

- Event

**Job on**
** ****Machine C**

**CPU 17.4 SI95 I/O 207MB/s @ 54MB file**

**CPU 14.0 SI95 I/O 31MB/s @ 54MB file**

##  MONARC Simulation System Multitasking  Processing  Model

**	****	**_**“**__**Interrupt” driven scheme:**_** **
**For each new task or when one task is finished, an interrupt is generated and all “times to completion” are recomputed.**

** ****Assign active tasks (CPU, I/O, network) to Java threads**
** ****Concurrent running tasks share resources (CPU, memory, I/O)**

##     Example : Physics Analysis at    Regional Centres

**    ****Example : Physics Analysis at    Regional Centres**

**     **

- Similar data processing jobs are performed in  each of several RCs
- There is profile of jobs,each submitted to a job scheduler
- Each Centre has “TAG”and “AOD” databases replicated.
- Main Centre provides “ESD” and “RAW” data
- Each job processes AOD data, and also aa fraction of ESD and RAW data.

##    Example: Physics Analysis

**     **

##       Validation Measurements          AMS Data Across a LAN 

- Raw Data

- DB

**LAN**

**4 CPUs Client**

**Simulation**

**Measurements**

**Distribution of 32 Jobs’ **
** ****Processing Time**

**Simulation**
**mean 109.5**

**Measurement**
**mean 114.3**

## ORCA Production on CERN/IT-LoanedEvent Filter Farm Test Facility  

**     **

**Pileup**
**DB**

**Pileup**
**DB**

**Pileup**
**DB**

**Pileup**
**DB**

**Pileup**
**DB**

**HPSS**

**Pileup**
**DB**

**Pileup**
**DB**

**Output **
**Server**

**Output **
**Server**

**Lock **
**Server**

**Lock **
**Server**

**SUN**

**...**

**FARM  140 Processing Nodes **

**17 Servers**

**9 Servers**

**Total 24 Pile Up Servers**

**The strategy is to use many commodity PCs as Database Servers**

##        Network Traffic & Job Efficiency 

**       ****Network Traffic & Job Efficiency **

**Mean measured Value ~48MB/s**

**Measurement**

**Simulation**

##     Total Time for Jet & Muon Production Jobs

## Job Scheduling in Distributed SystemsSelf-Organizing Neural Network (SONN)

** ****Efficient job scheduling policies in large distributed systems, ****	****which evolve dynamically, is one of  challenging tasks in HEP ****	****computing.****	****	****Analyze a large number of parameters describing the jobs and the time ****	****	****dependent state of the system. The problem is more difficult when not all ****	****	****these parameters are correctly identified, when  knowledge about the ****	****	****system state is incomplete and/or known with a certain delay.**
** **
** ****This study  is to develop tools able to generate effective job ****	****scheduling policies in distributed architectures, based on a ****	****“Self Organizing Neural Network” (SONN) system ****	****able to ****	****	****dynamically learn and cluster information in a large dimensional ****	****parameter space. **
**	****An adaptive middleware layer, aware of current available resources and ****	****learning from "past experience”, developing decision rules heuristically. **
** ****We applied the SONN approach to the problem of distributing  ****	****jobs among regional centers.  The evaluation for this job ****	****	****scheduling procedure has been done with the Monarc ****	****	****Simulation Tool. **** **

## Intuitive Scheduling Model 

**Intuitive **
**scheduling**
**scheme**

**Job description parameters  {J}**

**External RCs state description {R}**

**Job Scheduling decision {D}**

**Knowledge & experience**
**( +constants )**

**Local RC State {S}**

**Execution**

**Job**

**Evaluation**

**Performance quantification {X}**

**Job**

## A simple toy example 

**A simple toy example **

**We assume that the time to execute a job in the local farm having a certain load (********) is:**
**Where t****0**** is the theoretical time to perform the job and  f(********) describes the effect of the farm load in the job execution time.**
**If the job is executed on a remote site, an extra factor (********) is introduced in the response time:**

## Evaluating the SONN Scheduling Scheme with the MONARC Simulation

**DECISION **

**Intuitive **
**scheduling**
**scheme**

**RCs**
**  ****SIMULATION**

** ****Activities **
** ****Simulation**

**Job**

**Job**

**Job**

**Warming up**

**The learning process of the Self Organizing Network dynamically adapts itself to changes in the system configuration.**
** ****It may require a "classical" scheduling algorithm as a starting point with the aim  to dynamically improve it ****in time.**

## 2 RCs  Learning to Export Jobs

**Day = 9 **

**Cern**
**30  CPUs**

**Caltech**
**25 CPUs**

**No Activity**

**Kek**
**20 CPUs**

**0.8MB/s ;     200 ms RTT**

**<E> =0.30****             ****<E> =0.70**

**<E> =0.69**

## 2 RCs  Learning to Export Jobs

**caltech**

**kek**

**cern**

**Day 0 **

**Day 1**** **

**Day 2 **

**Day 9 **

**Day 6**

## MONARC Simulation: I. LeGrand Workplan (2000)

**May 2000: ****	****CMS HLT - simulation**
_**http://www.cern.ch/MONARC/sim_tool/Publish/CMS/publish/**_
_**http://home.cern.ch/clegrand/MONARC/CMS_HLT/sim_cms_hlt.pdf**_
**June 2000 ****	****Tape usage study**
_**http://www.cern.ch/MONARC/sim_tool/Publish/TAPE/publish/**_
**Aug 2000  ****	****Update of the Simulation tool for large scale simulations****.**
_**http://home.cern.ch/clegrand/MONARC/WSC/wsc_final.pdf **_**(to be presented at the IEEE Winter Simulation Conference: WSC2000)**
_**http://home.cern.ch/clegrand/MONARC/ACAT/sim.ppt**_
**Oct 2000   ****	****A study in using SONN for job scheduling**
_**http://www.cern.ch/MONARC/sim_tool/Publish/SONN/publish/**_
_**http://home.cern.ch/clegrand/MONARC/ACAT/sonn.ppt**_
**Nov 2000 ****	****Update of the CMS computing needs**
**Based on the new requirements data, to update the baseline ****models for CMS computing**
**Dec 2000 ****	****Simulation of the current CMS Higher Level Trigger production**

## MONARC Simulation: Workplan (2001)

_**Jan 2001**_**	****Update of the MONARC Simulation System****	****New release, including dynamic scheduling and replication modules (policies);****	****Improved Documentation**
_**Feb 2001**_**	****Role of Disk and Tapes in Tier1 and Tier2 Centers**
**	****More elaborate studies to describe Tier2-Tier1 interaction and to evaluate****	****data storage needs**
_**May 2001**_**	****Complex Tier0 - Tier1 - Tier2 simulation: ****	****	****Study the role of Tier2 centers**
**	****Aim is to perform a complete CMS data processing scenario including ****	****all major tasks distributed among regional centers**
_**Jul 2001**_**	****Real SONN module for job scheduling; based on Mobile agents**
**	****Create a Mobile Agents framework able to provide the basic mechanism for ****	****scheduling between regional centers**
_**Sep 2001**_**	****Add monitoring agents for network and system states****	****	****based on (SNMP)**
**	****Collect system dependent parameters using SNMP and integrate them into****	****the mobile agents used for scheduling**
_**Dec 2001**_**	****Study of the correlation between data replication and job ****	****	****scheduling**
**	****Combine the scheduling policies with data replication to optimize different cost ****	****functions; Integrate this into the Mobile Agents framework**

## MONARC Status

**MONARC is on the way to specifying baseline Models   ****	****  representing cost-effective solutions to LHC Computing**
**MONARC’s Regional Centre hierarchy model has been accepted by all four LHC Experiments**
** ****And is the basis of HEP Data Grid work.**
**A powerful simulation system has been developed, and is being used both for further Computing Model, Strategy Development, and Grid-component studies. **
**There is strong synergy with other advanced R&D projects:****PPDG, GriPhyN, EU HEP Data Grid, ALDAP and others.**
**Example Computing Models have been provided, and are ****being updated**
** ****This is  important input for the Hoffmann LHC ****	****	****	****Computing Review**
**The MONARC Simulation System is now being applied to key Data Grid issues, and Grid-tool design and Development**

##      MONARC Future: Some “Large”     Grid Issues to Be Studied

**     ****MONARC Future: Some “Large”     Grid Issues to Be Studied**

**Query estimation and transaction-design for****replica management**
**Queueing and co-scheduling strategies**
**Strategy for use of tapes**
**Strategy for resource sharing among sites ****and activities**
**Packaging of object-collections into blocks for ****transport across networks; integration with databases**
**Effect on Networks of Large windows, QoS, etc.**
**Behavior of the Grid Services to be developed**

## From UserFederation    To Private Copy

**CD**

**CH**

**MD**

**MH**

**TH**

**MC**

**UF.boot**

**MyFED.boot**

**User**
**Collection**

**MD**

**CD**

**MC**

**TD**

**AMS**

**ORCA 4 tutorial, part II - 14. October 2000**

**From UserFederation ****   To Private Copy**