# Presentation: 064607

## The LCG Service Challenges: Ramping up the LCG Service

Jamie Shiers, CERN-IT-GD-SC
April 2005

**The LCG Service Challenges: **_**Ramping up the LCG Service**_
- Jamie Shiers, CERN-IT-GD-SC
- April 2005

## Agenda

**Agenda**

- Goals and Timelines of the LCG Service Challenges
- Review of SC1 and SC2
- Summary of LHC Experiments’ Computing Models
- Outline of SC3 and SC4
- After that it’s the **FULL PRODUCTION SERVICE!**
- Plans for involving Tier2 sites in the Service Challenges
- Detailed SC3 planning

## Acknowledgements / Disclaimer

- These slides are the work of many people
- LCG Storage Management Workshop, Service Challenge Meetings etc
- Plus relevant presentations I found with Google!
- The work behind the slides is that of many more...
- All those involved in the Service Challenges at the many sites...
- I will use “LCG” in the most generic sense...
- Because that’s what I know / understand... (?)

## Partial Contribution List

- James Casey: SC1/2 review
- Jean-Philippe Baud: DPM and LFC
- Sophie Lemaitre: LFC deployment
- Peter Kunszt: FiReMan
- Michael Ernst: dCache
- gLite FTS: Gavin McCance
- ALICE: Latchezar Betev
- ATLAS: Miguel Branco
- CMS: Lassi Tuura
- LHCb: atsareg@in2p3.fr

## LCG Service Challenges - Overview

- LHC will enter production (physics) in April 2007
- Will generate an enormous volume of data
- Will require huge amount of processing power
- LCG ‘solution’ is a world-wide Grid
- Many components understood, deployed, tested..
- But...
- Unprecedented scale
- Humungous challenge of getting large numbers of institutes and individuals, all with existing, sometimes conflicting commitments, to work together
- LCG must be ready at full production capacity, functionality and reliability in less than 2 years from now
- Issues include h/w acquisition, personnel hiring and training, vendor rollout schedules etc.
_**Should not limit ability of physicist to exploit performance of detectors nor LHC’s physics potential**_
- Whilst being stable, reliable and easy to use

## LCG Deployment Schedule

## Service Challenges

- Purpose
- Understand what it takes to operate a real grid service – run for days/weeks at a time (outside of experiment Data Challenges)
- Trigger/encourage the Tier1 & large Tier-2 planning – move towards real resource planning – based on realistic usage patterns
- Get the essential grid services ramped up to target levels of reliability, availability, scalability, end-to-end performance
- Set out milestones needed to achieve goals during the service challenges
- NB: This is focussed on Tier 0 – Tier 1/large Tier 2
- Data management, batch production and analysis
- Short term goal – by end 2004 – have in place a robust and reliable data management service and support infrastructure and robust batch job submission

- Ian Bird – ian.bird@cern.ch

***From early proposal, May 2004***

## Why Service Challenges?

_**To test Tier-0 **__****__** Tier-1 **__****__**Tier-2 services**_
- Network service
- Sufficient bandwidth: ~10 Gbit/sec
- Backup path
- Quality of service: security, help desk, error reporting, bug fixing, ..
- Robust file transfer service
- File servers
- File Transfer Software (GridFTP)
- Data Management software (SRM, dCache)
- Archiving service: tapeservers,taperobots, tapes, tapedrives, ..
- Sustainability
- Weeks in a row un-interrupted 24/7 operation
- Manpower implications: ~7 fte/site
- Quality of service: helpdesk, error reporting, bug fixing, ..
_**Towards a stable production environment for experiments**_

***Kors Bos – Presentation to LHCC, March 7 2005***

## Whither Service Challenges?

- First discussions: GDB  May - June 2004
- May 18 -** Lessons from Data Challenges and planning for the next steps (+ Discussion)** (1h10') (   _[transparencies](http://agenda.cern.ch/askArchive.php?base=agenda&categ=a04113&id=a04113s1t6/transparencies)_ )
- June 15 -** Progress with the service plan team** (10') (   _[document](http://agenda.cern.ch/askArchive.php?base=agenda&categ=a04114&id=a04114s1t2/document)_ )
- Other discussions: PEB June 2004
- June 8 - **Service challenges - proposal** (40') (   _[transparencies](http://agenda.cern.ch/askArchive.php?base=agenda&categ=a042347&id=a042347s1t16/transparencies)_ )
- June 29 - **Service challenges - status and further reactions** (30') (   _[transparencies](http://agenda.cern.ch/askArchive.php?base=agenda&categ=a042862&id=a042862s1t11/transparencies)_ )
- May 2004 HEPiX
-   _[LCG Service Challenges](http://hepwww.rl.ac.uk/hepix/nesc/servicechallenges.ppt)_ Slides from Ian Bird (CERN)
- My involvement: from January 2005
- Current Milestones: _[http://lcg.web.cern.ch/LCG/PEB/Planning/deployment/Grid%20Deployment%20Schedule.htm](http://lcg.web.cern.ch/LCG/PEB/Planning/deployment/Grid%20Deployment%20Schedule.htm)_

## Key Principles

**Key Principles**

- Service challenges result in a _**series**_ of services that exist in _**parallel**_ with _**baseline production**_ service
- Rapidly and successively approach production needs of LHC
- Initial focus: core (data management) services
- Swiftly expand out to cover _**full spectrum**_ of production and analysis chain
- Must be as realistic as possible, including end-end testing of key experiment _**use-cases**_ over extended periods with recovery from _**glitches**_ and _**longer-term**_ outages
_**Necessary resources and commitment pre-requisite to success!**_
- Effort should not be under-estimated!

## To be done over the next six months (PEB)

- End January - Service Challenge planning paper (Service Challenge manager, Jamie Shiers)
- February 2005 - Detailed plan for deploying reliable file transfer service (GDA)
- March 2005 - Tier-0/Tier-1 middleware agreement - inc. workload management, catalog (PEB)
- March 2005 - Detailed plan/agreement for deploying mass storage implementations - inc. SRM 2, light-weight disk pool manager (GDA/GDB)
- End March - Applications area long term plan (AA manager)
- End March 2005 - Initial plan for Tier-0/Tier-1 networking (GDB/Network WG, David Foster)
- Early April 2005 - Final proposal for MoU (MoU Task Force)
- Early April 2005 - Re-assessment of Tier-0, Tier-1 plans in light of new model/requirement models (P2P group)
- Pasta IV complete (Sverre Jarp)
- April 2005 - Database strategy (3D Project, Dirk Duellmann)
- Middleware rollout plan (MWA, Frédéric Hemmer)
- End June 2005 - Production of the TDR (TDR Editor)

***Status of planning in January 2005***

## Informal Overview

- In addition to planned GDB meetings, Service Challenge Meetings, Network Meetings etc:
- Visits to all Tier1 sites (initially)
- Goal is to meet as many of the players as possible
- Not just GDB representatives! Equivalents of Vlado etc.
- Current Schedule:
- Aim to complete many of European sites by Easter
- “Round world” trip to BNL / FNAL / Triumf / ASCC in April
- Need to address also Tier2s
- Cannot be done in the same way!
- Work through existing structures, e.g.
- HEPiX, national and regional bodies etc.
- e.g. GridPP (12)
- Talking of SC Update at  May HEPiX (FZK) with more extensive programme at Fall HEPiX (SLAC)
- Maybe some sort of North American T2-fest around this?

***Status of planning in January 2005***

## SC1 Review

- SC1 _**did not**_ complete its goals successfully
- Dec04 - Service Challenge I complete
- mass store (disk) - mass store (disk)
- 3 T1s (Lyon, Amsterdam, Chicago) (others also participated...)
- 500 MB/sec (individually and aggregate)
- 2 weeks sustained
- Software; GridFTP plus some scripts
_**We did not meet the milestone of 500MB/s for 2 weeks**_
- We need to do these challenges to see what actually goes wrong
- A lot of things do, and did, go wrong
- We need better test plans for validating the infrastructure before the challenges (network throughput, disk speeds, etc...)
- OK, so we’re off to a great start with the Service Challenges...

## “Service Challenge 2 Review”

***LCG Storage Workshop***

**“****Service Challenge 2 Review****”**

- James Casey, IT-GD, CERN
- CERN, 5th April 2005

## Overview

- Reminder of
- Targets for the Service Challenge
- Plan and timescales
- CERN Tier-0 Configuration
- Tier-1 Configuration
- Throughput phase of Service Challenge 2
- Transfer Software
- Monitoring
- Outages and problems encountered
- SC2 single site tests

## SC2 - Overview	

- “Service Challenge 2”
- Throughput test from Tier-0 to Tier-1 sites
- Started 14th March
- Set up Infrastructure to 7 Sites
- NIKHEF/SARA, IN2P3, FNAL, BNL, FZK, INFN, RAL
- 100MB/s to each site
- 500MB/s combined to all sites at same time
- 500MB/s to a few sites individually
- Goal : by end March, sustained 500 MB/s at CERN

## SC2 met its throughput targets

- >600MB/s daily average for 10 days was achieved:Midday 23rd March to Midday 2nd April
- Not without outages, but system showed it could recover rate again from outages
- Load reasonable evenly divided over sites (give network bandwidth constraints of Tier-1 sites)

## Division of Data between sites

| Site | Average throughput (MB/s) | Data Moved (TB) |
| --- | --- | --- |
| BNL | 61 | 51 |
| FNAL | 61 | 51 |
| GridKA | 133 | 109 |
| IN2P3 | 91 | 75 |
| INFN | 81 | 67 |
| RAL | 72 | 58 |
| SARA | 106 | 88 |
| TOTAL | 600 | 500 |

## CERN Tier-0 Configuration (1/3)

- Service Challenge 1 was on an experimental hardware setup
- 10 HP IA64 nodes within the opencluster
- Not a standard service
- No LEMON monitoring
- No operator coverage
- No standard configuration
- Plan was to move to a “standard” configuration
- IA32 “worker nodes” running CASTOR SRM/gridftp
- Still serving data from local disks
- Also to move to “production” networking
- The opencluster is a test and research facility
- Again, non-monitored network hardware

## CERN Tier-0 Configuration (2/3)

- This didn’t all go to plan...
- First set of 20 IA32 worker nodes weren’t connected to right network segments
- They were installed, configured and tested before this was discovered
- Replacement nodes (20 IA32 disk servers) were supplied too late to get installed, configured and tested before SC2
- Had to fallback to openlab IA64 nodes for SC2
- These weren’t connected to the production network on their inward facing interface (towards CERN LAN)
- Added 10 extra IA64 nodes to system – 16 in total were used for data transfer in SC2

## CERN Tier-0 Configuration (3/3)

## Tier-1 Storage Configuration

- Small set of storage configurations
- Most sites ran “vanilla” Globus gridftp servers
- SARA, INFN, IN2P3, FZK
- The rest of the sites ran dCache
- FNAL, BNL, RAL
- Most sites used local or system-attached disk
- FZK used SAN via GPFS
- FNAL used production CMS dCache, including tape
- Load-balancing for most plain gridftp sites was done at the RADIANT layer
- INFN deployed “n-1” DNS alias – highest loaded machine was replaced in alias every 10 minutes
- Alleviated problems see with pure round-robin on gridftp servers

## Tier-1 Network Connectivity

- Sites are in the middle of their upgrade path to LCG networking
- 10Gb will be the norm – now it’s more like 1Gb
- All this is heavily tied to work done in the LCG T0-T1 networking group
- Most sites were able to provided dedicated network links
- Or at least links where we could get most of the supplied bandwidth
- IN2P3, BNL still were on shared links with a bit more congestion
- Needed to be dealt with differently
- Upped the number of concurrent TCP streams per file transfer

## Tier-1 Network Topology

**Tier-1 Network Topology**

## Transfer Control Software (1/3)

- LCG RADIANT Software used to control transfers
- Prototype software interoperable with the gLite FTS
- Plan is to move to gLite FTS for SC3
- Initial promising results presented at Lyon SC Meeting in Feb.
- More details in LCG Storage Workshop tomorrow
- Run on a single node at CERN, controlling all Tier-0 to Tier-1 “channels”
- Transfers done via 3rd-party gridftp
- ‘radiant-load-generator’ was used to generate transfers
- Configured for each channel to load-balance where appropriate
- Specifies number of concurrent streams for a file transfer
- This was normally =1 for a dedicated link
- Ran from cron to keep transfer queues full

## Transfer Control Software (2/3)

- Control of load on a channel via number of concurrent transfers
- Final Configuration:

- #Chan : State : Last Active      :Bwidth: Files: From   : To
- FZK   :Active :05/03/29 15:23:47 :10240 :5     :cern.ch :gridka.de
- IN2P35:Active :05/03/29 15:16:32 :204   :1     :cern.ch :ccxfer05.in2p3.fr
- IN2P33:Active :05/03/29 15:20:30 :204   :1     :cern.ch :ccxfer03.in2p3.fr
- INFN  :Active :05/03/29 15:23:07 :1024  :8     :cern.ch :cr.cnaf.infn.it
- IN2P32:Active :05/03/29 15:21:46 :204   :1     :cern.ch :ccxfer02.in2p3.fr
- FNAL  :Inactiv:Unknown           :10240 :0     :cern.ch :fnal.gov
- IN2P3 :Active :05/03/29 15:23:40 :204   :1     :cern.ch :ccxfer01.in2p3.fr
- IN2P34:Active :05/03/29 15:18:00 :204   :1     :cern.ch :ccxfer04.in2p3.fr
- BNL   :Inactiv:Unknown           :622   :24    :cern.ch :bnl.gov
- NL    :Active :05/03/29 15:22:54 :3072  :10    :cern.ch :tier1.sara.nl
- RAL   :Active :05/03/29 15:23:09 :2048  :12    :cern.ch :gridpp.rl.ac.uk

## Transfer Control Software (3/3)

- RADIANT controlled transfers were pure gridftp only
- SRM components did not get developed in time
- FNAL and BNL did SRM transfers by pulling from their end
- FNAL used PhEDEx to pull data
- BNL used simple driver scripts around srmCopy command
- BNL started some radiant-controlled gridftp transfers too
- Used the exact transfer nodes as the rest of the sites (radiantservice.cern.ch)
- SRM interactions helped debug issues with dCache SRM and DNS load-balanced SRM/gridftp servers
- Fixed java DNS alias problems seen at FNAL
- Did not work smoothly for most of the challenge with FNAL mode of operation
- Main problem solved and fix in place by end of throughtput phase
- How to do srmCopy’s with 1000s of files is still not well understood

## Monitoring @ CERN

**Monitoring @ CERN**

- MRTG Graphs of network usage out of cluster
- LEMON monitoring of cluster
- CPU usage
- Disk usage
- Network usage
- Gridftp logfile monitoring

## Tier-1 Monitoring

- Gridftp Monitoring (http://challenge.matrix.sara.nl/SC/)
- Main monitoring tool used during the SC2
- Hourly throughput per site
- Hourly throughput per host
- Daily throughput per site
- Daily throughput per host

## Service Outages (1/2)

- Progress page kept in SC wiki of
- All tunings made to the system
- All outages noted during the running
- Any actions needed to cleanup and recover service
- <http://service-radiant.web.cern.ch/service-radiant/wiki/ow.asp?ChallengeSC2Progress>
- No real 24x7 service in place
- Manual monitoring of monitoring webpages
- Best-effort restart of service
- Also at Tier-1 sites – problems communicated to service challenge teams, but this was not a 24x7 coverage

## Service Outages (2/2)

- Capacity in the cluster meant that we could recover from one site not being active
- Other sites would up their load a bit automatically due to gridftp stream rates increasing
- Only thing that killed transfers were CERN outages
- We did not do any scheduled outages during the SC
- No procedures for starting a schedule outage
- If we had done one to move to managed network infrastructure, it would have removed some of the scheduled ones

## Outage Breakdown - CERN

- Myproxy instability
- Locks up under high load
- Understood by developers of myproxy
- Can be handled by watchdog job
- Particular problem on restart after network outage
- CERN LAN Network outages
- Had 2 long-ish outages (~12 hours)
- Issue with being on non-managed network hardware
- Database quota limit hit
- Tablespace was being monitored but not quota
- Quota monitoring added
- Database load problems
- Caused intermittent drops in throughput due to new jobs not being scheduled
- In-memory job queues in the transfer agents meant these we’re a big problem

## DNS load-balancing of gridftpd

**DNS load-balancing of gridftpd**

- An issue raised at Lyon SC Meeting
- “load-balanced” gridftp servers
- Not really load-balanced with round-robin – no awareness of state or behaviour of other nodes
- Can end up with all transfers going to one host, with all others idle
- CNAF put in place “worst one out” load-balancing
- Heaviest loaded node is taken out of alias every 10 minutes
- Smoothed out load

## Individual site tests

- Overlapped with LCG Storage Management Workshop
- Sites can pick days in next two weeks when they have the capacity
- 500MB/s to disk
- 60MB/s to tape
- FNAL was running 500MB/s disk tests at the time...

## FNAL Failover

- it shows
- that when the starlight link was cut, our transfers fell back to our ESNET connection,
- when our network folks rerouted it to our 1 GE link
- when starlight came back.

## SC2 Summary

- SC2 met it’s _**throughput**_ goals – and with _**more**_ sites than originally planned!
- A big improvement from SC1
_**But we still don’t have something we can call a service**_
- Monitoring is better
- We see outages when they happen, and we understood why they happen
- First step towards operations guides
- Some advances in infrastructure and software will happen before SC3
- gLite transfer software
- SRM service more widely deployed
- We have to understand how to incorporate these elements

## SC1/2 - Conclusions

- Setting up the infrastructure and achieving reliable transfers, even at much lower data rates than needed for LHC, is complex and requires a lot of technical work + coordination
- Even within one site – people are working very hard & are stressed. Stressed people do not work at their best. Far from clear how this scales to SC3/SC4, let alone to LHC production phase
- Compound this with the multi-site / multi-partner issue, together with time zones etc and you have a large “non-technical” component to an already tough problem (example of technical problem follows...)
- But... the end point is fixed (time + functionality)
- We should be careful not to over-complicate the problem or potential solutions
- And not forget there is still a humungous amount to do...
- (much much more than we’ve done...)

## Computing Model Summary - Goals

- Present key features of LHC experiments’ Computing Models in a consistent manner
- High-light the commonality
- Emphasize the key differences
- Define these ‘parameters’ in a central place (LCG web)
- Update with change-log as required
- Use these parameters as input to requirements for Service Challenges
- To enable partners (T0/T1 sites, experiments, network providers) to have a clear understanding of what is required of them
_**Define precise terms and ‘factors’**_

## LHC Computing Models

## Where do these numbers come from?

_**Obtained from LHC Computing Models as reviewed in January**_
- Part of plan is to understand how sensitive overall model is to variations in key parameters
- Iteration with experiments is on-going
- i.e. I have tried to clarify any questions that I have had
_**Any mis-representation or mis-interpretation is entirely my responsibility**_
- Sanity check: compare with numbers from MoU Task Force
- (Actually the following LCG document now uses these numbers!)
- _[http://cern.ch/LCG/documents/LHC_Computing_Resources_report.pdf](http://cern.ch/LCG/documents/LHC_Computing_Resources_report.pdf)_

## Slide 41

| Nominal | These are the raw figures produced by multiplying e.g. event size x trigger rate. |
| --- | --- |
| Headroom | A factor of 1.5 that is applied to cater for peak rates. |
| Efficiency | A factor of 2 to ensure networks run at less than 50% load. |
| Recovery | A factor of 2 to ensure that backlogs can be cleared within 24 – 48 hours and to allow the load from a failed Tier1 to be switched over to others. |
| Total
Requirement | A factor of 6 must be applied to the nominal values to obtain the bandwidth that must be provisioned. 

Arguably this is an over-estimate, as “Recovery” and “Peak load” conditions are presumably relatively infrequent, and can also be smoothed out using appropriately sized transfer buffers.

But as there may be under-estimates elsewhere... |

## All numbers presented will be nominal unless explicitly specified

## LHC Parameters (Computing Models)

| Year | pp operations |  | Heavy Ion operations |  |
| --- | --- | --- | --- | --- |
|  | Beam time
(seconds/year) | Luminosity
(cm-2s-1) | Beam time
(seconds/year) | Luminosity
(cm-2s-1) |
| 2007 | 5 x 106 | 5 x 1032 | - | - |
| 2008 | (1.8 x) 107
 | 2 x 1033 | (2.6 x) 106 | 5 x 1026 |
| 2009 | 107 | 2 x 1033 | 106 | 5 x 1026 |
| 2010 | 107 | 1034 | 106 | 5 x 1026 |

- (Real time given in brackets above)

## LHC Schedule – “Chamonix” workshop

- First collisions: two months after first turn on in August 2007
- 32 weeks of operation, 16 weeks of shutdown, 4 weeks commissioning = 140 days physics / year (5 lunar months)

## Overview of pp running

| Experiment | SIM | SIMESD | RAW | Trigger  | RECO | AOD | TAG |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ALICE | 400KB | 40KB | 1MB | 100Hz | 200KB | 50KB | 10KB |
| ATLAS | 2MB | 500KB | 1.6MB | 200Hz | 500KB | 100KB | 1KB |
| CMS | 2MB | 400KB | 1.5MB | 150Hz | 250KB | 50KB | 10KB |
| LHCb |  | 400KB | 25KB | 2KHz | 75KB | 25KB | 1KB |

## pp questions / uncertainties

- Trigger rates essentially independent of luminosity
- Explicitly stated in both ATLAS and CMS CM docs
- Uncertainty (at least in my mind) on issues such as zero suppression, compaction etc of raw data sizes
- Discussion of these factors in CMS CM doc p22:
- RAW data size ~300kB (Estimated from MC)
- Multiplicative factors drawn from CDF experience
- MC Underestimation factor 1.6
- HLT Inflation of RAW Data, factor 1.25
- Startup, thresholds, zero suppression,.... Factor 2.5
- Real initial event size more like **1.5MB**
- Could be anywhere between 1 and 2 MB
- Hard to deduce when the even size will fall and how that will be compensated by increasing Luminosity
- _i.e. total factor = 5 for CMS raw data_
- N.B. must consider not only Data Type (e.g. result of Reconstruction) but also how it is used
- e.g. compare how Data Types are used in LHCb compared to CMS
- All this must be plugged into the meta-model!

## Overview of Heavy Ion running

| Experiment | SIM | SIMESD | RAW | Trigger  | RECO | AOD | TAG |
| --- | --- | --- | --- | --- | --- | --- | --- |
| ALICE | 300MB | 2.1MB | 12.5MB | 100Hz | 2.5MB | 250KB | 10KB |
| ATLAS |    |  | 5MB | 50Hz |  |  |  |
| CMS |  |  | 7MB | 50Hz | 1MB | 200KB | TBD |
| LHCb | N/A | N/A | N/A  | N/A | N/A | N/A | N/A |

## Heavy Ion Questions / Uncertainties

- Heavy Ion computing models less well established than for pp running
- I ***was*** concerned about model for 1st/2nd/3rd pass reconstruction and data distribution
*“****We therefore require that these data (Pb-Pb) are reconstructed at the CERN T0 and exported over a four-month period after data taking. This should leave enough time for a second and third reconstruction pass at the Tier 1’s” *****(ALICE)**
- Heavy Ion model has major impact on those Tier1’s supporting these experiments
- All bar LHCb!
- These issues have since been clarified:
- Raw data export will be spread over shutdown;
- First pass reconstruction should complete at least 6 months prior to following year’s AA data taking (ALICE) or during shutdown (CMS);
- 2nd pass (ALICE) will not involve the full data sample;
- 3rd pass is a complete pass which should complete prior to following year’s AA run
- Implies data out of CERN roughly constant; will vary per T1 (pp/AA/shutdown) depending on experiments supported

## Data Rates from MoU Task Force

| MB/Sec | RAL | FNAL | BNL | FZK | IN2P3 | CNAF | PIC | T0 Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ATLAS | 106.87 | 0.00 | 173.53 | 106.87 | 106.87 | 106.87 | 106.87 | 707.87 |
| CMS | 69.29 | 69.29 | 0.00 | 69.29 | 69.29 | 69.29 | 69.29 | 415.71 |
| ALICE | 0.00 | 0.00 | 0.00 | 135.21 | 135.21 | 135.21 | 0.00 | 405.63 |
| LHCb | 6.33 | 0.00 | 0.00 | 6.33 | 6.33 | 6.33 | 6.33 | 31.67 |
| T1 Totals MB/sec | 182.49 | 69.29 | 173.53 | 317.69 | 317.69 | 317.69 | 182.49 | 1560.87 |
| T1 Totals Gb/sec | 1.46 | 0.55 | 1.39 | 2.54 | 2.54 | 2.54 | 1.46 | 12.49 |
|  |  |  |  |  |  |  |  |  |
| Estimated T1 Bandwidth Needed |  |  |  |  |  |  |  |  |
| (Totals * 1.5(headroom))*2(capacity) | 4.38 | 1.66 | 4.16 | 7.62 | 7.62 | 7.62 | 4.38 | 37.46 |
|  |  |  |  |  |  |  |  |  |
| Assumed Bandwidth Provisioned | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 | 10.00 | 70.00 |

***http://cern.ch/LCG/MoU%20meeting%20March%2010/Report_to_the_MoU_Task_Force.doc***

## Tier1 Sites

| Centre | ALICE | ATLAS | CMS | LHCb | Target Data Rate MBytes/sec |
| --- | --- | --- | --- | --- | --- |
| ASCC |  | X | X |  | 110 |
| CNAF | X | X | X | X | 220 |
| PIC |  | X | X | X | 200 |
| CC-IN2P3 | X | X | X | X | 220 |
| FZK | X | X | X | X | 220 |
| RAL | X | X | X | X | 220 |
| BNL |  | X |  |  | 154 |
| FNAL |  |  | X |  | 50 |
| TRIUMF |  | X |  |  | 65 |
| NIKHEF | X | X |  | X | 175 |
| NORDIC DATA GRID FACILITY | X | X |  | X | 90 |
| Target data rate at CERN |  |  |  |  | 1,600 |

- Target data rates calculated from raw computing model numbers during pp running
- No (in)efficiency factors, no overhead, etc.
- Assume each T1 takes equal fraction (except BNL)
- Balanced by fraction of resources allocated per experiment?

## Heavy Ion Data Rates

- ATLAS / CMS data rates limited by online system
- LHCb does not participate in Heavy Ion programme
- Current model is that data is distributed during shutdown, rather than inline with data taking

## pp / AA data rates (equal split)

| Centre | ALICE | ATLAS | CMS | LHCb | Rate into T1 | Rate into T1 (AA) |
| --- | --- | --- | --- | --- | --- | --- |
| ASCC, Taipei | 0 | 1 | 1 | 0 | 118.7 | 28.2 |
| CNAF, Italy | 1 | 1 | 1 | 1 | 205.0 | 97.2 |
| PIC, Spain | 0 | 1 | 1 | 1 | 179.0 | 28.2 |
| IN2P3, Lyon | 1 | 1 | 1 | 1 | 205.0 | 97.2 |
| GridKA, Germany | 1 | 1 | 1 | 1 | 205.0 | 97.2 |
| RAL, UK | 1 | 1 | 1 | 1 | 205.0 | 97.2 |
| BNL, USA | 0 | 1 | 0 | 0 | 72.2 | 11.3 |
| FNAL, USA | 0 | 0 | 1 | 0 | 46.5 | 16.9 |
| TRIUMF, Canada | 0 | 1 | 0 | 0 | 72.2 | 11.3 |
| NIKHEF/SARA, Netherlands | 1 | 1 | 0 | 1 | 158.5 | 80.3 |
| Nordic Centre | 1 | 1 | 0 | 0 | 98.2 | 80.3 |
| Totals | 6 | 10 | 7 | 6 |   |  |

## Streaming

- All experiments foresee RAW data streaming, but with different approaches:
- CMS: O(50) streams based on trigger path
- Classification is immutable, defined by L1+HLT
- Atlas: 4 streams based on event types
- Primary physics, Express line, Calibration, Debugging and diagnostic
- LHCb: >4 streams based on trigger category
- B-exclusive, Di-muon, D* Sample, B-inclusive
- Streams are not created in the first pass, but during the “stripping” process
-  Not clear what is the best/right solution. Probably bound to evolve in time.

- Francesco Forti, Pisa

## Reprocessing

- Data need to be reprocessed several times because of:
- Improved software
- More accurate calibration and alignment
- Reprocessing mainly at T1 centers
- LHCb is planning on using the T0 during the shutdown – not obvious it is available
- Number of passes per year
- But experience shows the reprocessing requires huge effort!
- Use these numbers in the calculation but 2 / year will be good going!

| Alice | Atlas | CMS | LHCb |
| --- | --- | --- | --- |
| 3 | 2 | 2 | 4 |

- Francesco Forti, Pisa

## Base Requirements for T1s

**Provisioned bandwidth comes in units of 10Gbits/sec although this is an evolving parameter**
*From *Reply to Questions from Computing MoU Task Force...
- Since then, some parameters of the Computing Models have changed
- Given the above quantisation, relatively insensitive to small-ish changes
- Important to understand implications of multiple-10Gbit links, particularly for sites with Heavy Ion programme
- Spread of AA distribution during shutdown probably means 1 link sufficient...
_**For now, planning for 10Gbit links to all Tier1s**_

##  Tier1 SUMMARY

- High data rate transfer tests Karlsruhe to CERN already underway
- 10G available now from Bologna to CERN
- Testing of 10G lambdas from Lyon and Amsterdam can commence from July 2005
- Amsterdam (and Taipei) will use Netherlight link to CERN until GÉANT2 paths are available
- Testing of Barcelona link at 10G from October 2005
- Nordic distributed facility restricted to 1G until late-2006 when 10G available
- RAL could operate between 2 and 4 x 1GE (subject to scheduling and NetherLight/CERN agreement) until late-2006 when 10G available. Interconnection of UKLight to GEANT2 might make an earlier transition to higher capacities possible.

**Tier0/Tier1/Tier2 NREN Connectivity Overview**
**John Chevers, Project Manager, DANTE **
**Amsterdam 8****th**** April 2005**

## Data Rate / Site - Conclusions

- It is clear that these are only estimates
- Experiments will almost certainly split data into streams which will not be of equal size
- The share of resources that each T1 provides to a given experiment will also influence the amount of data that is sent there
- But it is impossible (and not relevant?) to do a precise calculation
- Which in any case becomes further clouded when the reprocessing and analysis is folded in...

## Dedicated connections for SCs

| Tier1 | Location | NRENs | Status dedicated link |
| --- | --- | --- | --- |
| ASCC | Taipei, Taiwan | ASnet, SURFnet | 1 Gb via SURFnet, testing |
| BNL | Upton, NY, USA | ESnet, LHCnet | 622 Mbit shared |
| CNAF | Bologna, Italy | Geant2, GARR | 1 Gb now, 10 Gb in Sept |
| FNAL | Batavia, ILL, USA | ESnet, LHCnet | 10 Gb, tested |
| IN2P3 | Lyon, France | Renater | 1 Gb now, 10 Gb in Sept |
| GridKa | Karlsruhe, Germany | Geant2, DFN | 10 Gb, tested |
| SARA | Amsterdam, NL | Geant2, SURFnet | 10 Gb, testing |
| NorduGrid | Scandinavia | Geant2, Nordunet | Not participating yet |
| PIC | Barcelona, Spain | RedIris, Geant2 | Not participating yet |
| RAL | Didcot, UK | Geant2, Ukerna | 2 x 1 Gb via SURFnet soon |
| Triumf | Vancouver, Canada | Canet, LHCnet | 1 Gb via SURFnet, testing |

***Kors Bos, LCG-LHCC Referees Meeting, March 2005***

## T1/T2 Roles

_**Tier1**_
- Keep certain portions of RAW, ESD, sim ESD
- Full copies of AOD + TAG, calibration data
- Official physics group large scale data analysis
- ALICE + LHCb:
- also contribute to simulation

_**Tier2**_
- Keep certain portions of AOD and full copies of TAG for real + simulated data
- LHCb: sim only at T2s
- Selected ESD samples
- Produce simulated data
- General end-user analysis

- Based on “T1 Services for T2 Centres” document
***(Just type this into Google)***

## MC Data

|  | Units | ALICE |  | ATLAS | CMS | LHCb |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | p-p | Pb-Pb | p-p | p-p |  |
| Time to reconstruct 1 event | kSI2k sec | 5.4 | 675 | 15 | 25 | 2.4 |
| Time to simulate 1 event | kSI2k sec | 35 | 15000 | 100 | 45 | 50 |

| Parameter | Unit | ALICE |  | ATLAS | CMS | LHCb |
| --- | --- | --- | --- | --- | --- | --- |
|   |   | p-p | Pb-Pb |   |   |   |
| Events/year | Giga | 1 | 0.1 | 2 | 1.5 | 20 |
| Events SIM/year | Giga | 1 | 0.01 | 0.4 | 1.5 | 4 |
| Ratio SIM/data | % | 100% | 10% | 20% | 100% | 20% |

**Tier2 sites offer 10 – 1000 kSI2K years**
**ATLAS: 16MSI2K years over ~30 sites in 2008**
**CMS: 20MSI2K years over ~20 sites in 2008**

## GridPP Estimates of T2 Networking

|  | Number of T1s | Number of T2s | Total T2 CPU | Total T2 Disk | Average T2 CPU | Average T2 Disk | Network In | Network Out |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  | KSI2K | TB | KSI2K | TB | Gb/s | Gb/s |
| ALICE | 6 | 21 | 13700 | 2600 | 652 | 124 | 0.010 | 0.600 |
| ATLAS | 10 | 30 | 16200 | 6900 | 540 | 230 | 0.140 | 0.034 |
| CMS | 6 to 10 | 25 | 20725 | 5450 | 829 | 218 | 1.000 | 0.100 |
| LHCb | 6 | 14 | 7600 | 23 | 543 | 2 | 0.008 | 0.008 |

- The CMS figure of 1Gb/s into a T2 comes from the following:
-  Each T2 has ~10% of current RECO data and 1/2 AOD (real+MC sample)
-  These data are refreshed every 3 weeks
-  compatible with frequency of (possible) major selection pass at T1s
-  See CMS Computing Model S-30 for more details

## The (ATLAS) System

- Tier2 Centre  ~200kSI2k

- Event Builder

- Event Filter~7.5MSI2k

- T0 ~5MSI2k

- UK Regional  Centre (RAL)

- US Regional  Centre

-  French Regional Centre

- Dutch Regional Centre

- Sheffield

- Manchester

- Liverpool

**Lancaster ~0.25TIPS**

- Workstations

- ~100 Gb/sec

- ~3 Gb/sec raw

- 100 - 1000 Mb/s links

- Castor

- MSS

- Some data for calibration and monitoring to institutes
- Calibrations flow back

**Each of ~30 Tier 2s have ~20 physicists (range) working on one or more channels**
**Each Tier 2 should have the full AOD, TAG & relevant Physics Group summary data**
**Tier 2 do bulk of simulation**

- Physics data cache

- ~Pb/sec

- ~ 75MB/s/T1 raw for ATLAS

- Tier2 Centre  ~200kSI2k

- Tier2 Centre  ~200kSI2k

- 622Mb/s links

**Tier 0**

**Tier 1**

**Desktop**

**PC (****2004****) = ~1 kSpecInt2k**

**Tier 2**

**~200 Tb/yr/T2**
**(or 50Mbps)**

**~2MSI2k/T1**
**~2 Pb/yr/T1**

**~5 Pb/year**
**No simulation**

- 622Mb/s links

**The (ATLAS) System**

**10 Tier-1s reprocess **
**house simulation**
**Group Analysis**

## ALICE Computing Needs

| From <http://pcaliweb02.cern.ch/NewAlicePortal/en/Collaboration/Documents/TDR/Computing.html> as posted 25 Feb. 2005 |  |  |  |  |
| --- | --- | --- | --- | --- |
| Table 2.6 | T0 | Sum T1s | Sum T2s | Total |
| CPU (MSI2K) [Peak] | 7.5 | 13.8 | 13.7 | 35 |
| Transient Storage (PB) | 0.44 | 7.6 | 2.5 | 10.54 |
| Permanent storage (PB/year) | 2.3 | 7.5 | 0 | 9.8 |
| Bandwidth in (Gbps) | 8 | 2 | 0.075 |   |
| Bandwidth out (Gbps) | 6 | 1.5 | 0.27 |   |

## Analysis Model

- The ‘soft’ association of a T2 with a T1 handles the MC case and >90% of the analysis case
- Posit: the <10% left does not necessarily mean generalising the entire model
- See next slide for a ‘simple’ example
- LHCb have a simple model for handling this by official ‘stripping’ phases ~4 times per year
- I would propose addressing now 100% of the MC case and >90% of the analysis cases
- And further discussion – perhaps joint w/s with ARDA?

## Analysis Example (ATLAS)

- 	A typical analysis scenario might begin with the physicist issuing a query against a **very large tag dataset**, e.g. the latest **reconstruction** of all data **taken to date**. For example, the query might be for events with **three** **leptons** and **missing transverse energy** above some threshold. The result of this query is used to define a dataset with the AOD information for these events. The analyst could then provide an **Athena algorithm** to make further event selection by refining the **electron** quality or missing **transverse** energy calculations. The new output dataset might be used to create an n-tuple for further analysis or the AOD data for the selected events could be copied into new files. A subset of particularly striking events identified in one of these samples could be used to construct a dataset that includes the ESD and perhaps even RAW data for these events. The physicist might then redo the electron reconstruction for these events and then use it to create a new AOD collection or n-tuple.

## Service Challenge 3

## Service Challenge 3 - Phases

_**High level view:**_
_**Setup phase**_ (includes Throughput Test)
- 2 weeks sustained in July 2005
- “Obvious target” – GDB of July 20th
- Primary goals:
- 150MB/s disk – disk to Tier1s;
- 60MB/s disk (T0) – tape (T1s)
- Secondary goals:
- Include a few named T2 sites (T2 -> T1 transfers)
_**Encourage remaining T1s to start disk – disk transfers**_
_**Service phase**_
- September – end 2005
- Start with ALICE & CMS, add ATLAS and LHCb October/November
- All offline use cases except for analysis
- More components: WMS, VOMS, catalogs, experiment-specific solutions
- Implies production setup (CE, SE, ...)

## SC3 – Production Services

- SC3 is a relatively small step wrt SC2 ***in terms of throughput!***
- We know we can do it technology-wise, but do we have a solution that will scale?
**Let’s make it a priority for the coming months to streamline our operations**
- And not just throw resources at the problem...
- which we don’t have...
- Whilst not forgetting ‘real’ goals of SC3... ***i.e. services!***

## SC3 – Service Phase

- It sounds easy:**“all offline Use Cases except for analysis”**
- And it some senses it is:these are well understood and tested
- So its clear what we have to do:
- Work with the experiments to understand and agree on the experiment-specific solutions that need to be deployed
- Agree on a realistic and achievable work-plan that is consistent with overall goals / constraints
- Either that or send a ’droid looking for Obi-Wan Kenobi...

## Service Phase - Priorities

- Experiments have repeatedly told us to focus on reliability and functionality
- This we need to demonstrate as a first step...
_**But cannot lose sight of need to pump up data rates – whilst maintaining production service – to pretty impressive “DC” figures**_

## SC3 Preparation Workshop

- This (proposed)  workshop will focus on very detailed technical planning for the whole SC3 exercise.
- It is intended to be as interactive as possible, i.e. not presentations to an audience largely in a different (wireless) world.
- There will be sessions devoted to specific experiment issues, Tier1 issues, Tier2 issues as well as the general service infrastructure.
- Planning for SC3 has already started and will continue prior to the workshop.
- This is an opportunity to get together to iron out concerns and issues that cannot easily be solved by e-mail, phone conferences and/or other meetings prior to the workshop.

## SC3 Preparation W/S Agenda

- 4 x 1/2 days devoted to experiments
- in B160 1-009, phone conferencing possible
- 1 day focussing on T1/T2 issues together with output of above
- In 513 1-024, VRVS available
- Dates are 13 – 15 June (Monday – Wednesday)
- Even though conference room booked tentatively in February, little flexibility in dates even then!

## SC3 on

- SC3 is _**significantly**_ more complex than previous challenges
- It includes experiments s/w, additional m/w, Tier2s etc
- Proving we can transfer dummy files from A-B proves nothing
- Obviously need to show that basic infrastructure works...
- Preparation for SC3 includes:
- Understanding experiments’ Computing Models
- Agreeing involvement of experiments’ production teams
- Visiting all (involved) Tier1s (multiple times)
- Preparing for the involvement of 50-100 Tier2s
- Short of resources at all levels:
- “Managerial” – discussing with experiments and Tier1s (visiting)
- “Organizational” – milestones, meetings, workshops, ...
- “Technical” – preparing challenges and running CERN end – 24 x 7 ???

## Prepare for the next service challenge (SC3)
  --  in parallel with SC2 (reliable file transfer) – 

Build up 1 GByte/s challenge facility at CERN
The current 500 MByte/s facility used for SC2 will become the testbed from April onwards (10 ftp servers, 10 disk servers, network equipment)

Build up infrastructure at each external centre
Average capability ~150 MB/sec at a Tier-1 (to be agreed with each T-1)

Further develop reliable transfer framework software
Include catalogues, include VO’s

- Prepare for the next service challenge (SC3)
-   --  in parallel with SC2 (reliable file transfer) –
- Build up 1 GByte/s ***challenge***** **facility at CERN
**The current 500 MByte/s facility used for SC2 will become the *****testbed***** from April onwards (10 ftp servers, 10 disk servers, network equipment)**
- Build up infrastructure at each external centre
**Average *****capability***** ~**150 MB/sec** at a Tier-1 (to be agreed with each T-1)**
- Further develop reliable transfer framework software
**Include catalogues, include VO’s**

**2005 Q1  -  ****SC3 preparation**

**disk-network-disk bandwidths**

## 2005 Sep-Dec -  SC3 Service

- 50% Computing Model Validation Period
- The service exercised in SC3 is made available to experiments as a stable, permanent service for computing model tests
- Additional sites are added as they come up to speed
- End-to-end *sustained* data rates –
- 500 Mbytes/s at CERN (aggregate)
- 60 Mbytes/s at Tier-1s
- Modest Tier-2 traffic

## SC3 – Milestone Decomposition

- File transfer goals:
- Build up disk – disk transfer speeds to 150MB/s with 1GB/s out of CERN
- SC2 was 100MB/s – agreed by site
- Include tape – transfer speeds of 60MB/s with 300MB/s out of CERN
- Tier1 goals:
- Bring in additional Tier1 sites wrt SC2 (at least wrt the original plan...)
- PIC and Nordic most likely added later: SC4?
- Tier2 goals:
- Start to bring Tier2 sites into challenge
- Agree services T2s offer / require
- On-going plan (more later) to address this via GridPP, INFN etc.
- Experiment goals:
- Address main offline use cases ***except*** those related to analysis
- i.e. real data flow out of T0-T1-T2; simulation in from T2-T1
- Service goals:
- Include CPU (to generate files) and storage
- Start to add additional components
- Catalogs, VOs, experiment-specific solutions etc, 3D involvement, ...
- Choice of software components, validation, fallback, ...

## SC3 – Experiment Goals

- Meetings on-going to discuss goals of SC3 and experiment involvement
- Focus on:
- First demonstrate robust infrastructure;
- Add ‘simulated’ experiment-specific usage patterns;
- Add experiment-specific components;
- Run experiments offline frameworks but don’t preserve data;
- Exercise primary Use Cases ***except*** analysis (SC4)
- Service phase: data is preserved...
- Has significant implications on resources beyond file transfer services
- Storage; CPU; Network... Both at CERN and participating sites (T1/T2)
- May have different partners for experiment-specific tests (e.g. not all T1s)
_**In effect, experiments’ usage of SC during service phase = data challenge**_
- Must be _**exceedingly clear**_ on goals / responsibilities during each phase!

## SC3 – Experiment Involvement Cont.

- Regular discussions with experiments have started
- ATLAS: at DM meetings
- ALICE+CMS: every ~2 weeks
- LHCb: no regular slot yet, but discussions started...
- Anticipate to start first with ALICE and CMS (exactly when TDB) ATLAS and LHCb around October
- T2 sites being identified in common with these experiments
- More later...
- List of experiment-specific components and the sites where they need to be deployed being drawn up
- Need this on April timeframe for adequate preparation & testing

## ALICE Physics Data Challenge ’05 and LCG Service Challenge 3 

- Latchezar Betev / ALICE
- Geneva, 6 April 2005
**LCG Storage Management Workshop **

## Outline  

- Goals and structure of PDC’05
- Basic parameters and participation plans
- Services, processing layout
- Timeline

## Goals and structure  

- The transfer framework can only be discussed as a part of the overall ALICE Grid strategy
- PDC’05 : Test and validation of the remaining parts of the ALICE Offline computing model:
- Quasi-online reconstruction of RAW data at CERN (T0), without calibration
-  ‘push’ of the data from CERN to T1’s
- Second phase (delayed) reconstruction at T1’s ***with calibration*** and remote storage
***Do all of the above ***_***again***_*** ENTIRELY on the GRID***
- Structure – logically divided in three phases:
- Phase 1 – Slow production of events on the GRID, storage at CERN
- Phase 2 (during SC3) – Pass 1 reconstruction at CERN, push data from CERN to T1’s, Pass 2 reconstruction at T1s with calibration and storage:
- Phase 2 (throughput phase of SC3) – how fast we can push data out
- Phase 3 – Analysis of data (batch) and ***interactive analysis with PROOF***

## Parameters of the exercise  

- Data production:
- Physics driven, list of physics signals to be produced is being collected now
- The processed data has to be preserved – capacity for its storage has been pledged by the participating computing centres for ALICE
- Potentially as large data volume as in PDC’04 (50-80TB) – ALICE detector and physics studies require new MC generated signals and improved statistics of the already produced ones
- SC3 - ALICE participation currently being discussed:
- Computing and storage capacities
- Network capacity: reserved/shared and elements (PDC data transfer volume ~200 TB)
- Availability of services (more later)
- Monitoring: experiment specific and interaction with the SC3 tools

## Services availability  

- Transfer and storage services (part 1 of SC3):
- GridFTP
- gLite FTS
- SRM
- Experimental solution
- Other
- How will these become ‘stable services’ for the experiments and with what functionality
- Be deployed on sufficiently large computing and storage capacities on the sites (more than on test-beds)
- Other Grid services (part 2 of SC3):
- We assume that EGEE middleware available on LCG-SC3

## ALICE PDC’05 layout

**AliEn **
**CE/SE**

***Central services:***
**Catalogue**
**Task queue**
**Job optimization**
**-etc.**

**AliEn **
**CE/SE**

**AliEn **
**CE/SE**

**LCG **
**CE/SE**

**LCG **
**CE/SE**

**LCG **
**CE/SE**

- Job submission

- File registration

- Phase 1: Event production and storage at CERN
- Phase 2: pass 1 reconstruction ->  fast push to T1 -> pass 2 reconstruction/calibration -> store at T1

## PDC’05 timeline

**Apr05 **

**–**

**SC2 Complete**

**service**

- Apr

- May

- Jun

- Jul

- Aug

- Sep

- Oct

- Nov

- Dec

**2005**

- Start event production:         - at all available computing capacities (AliEn and LCG)

- SC3 Throughput test

- ALICE data ‘push’:                 - reserved/shared bandwidth   - test of available services

- SC3 Service test:               - add more complexity to the picture, data analysis

## PDC’05 continuation

- Last step:
- Analysis of the reconstructed data
- That is at the end of SC3:
- We have some more time to think about it

## ATLAS File Transfer Framework and SC3

## Baseline Requirements

- Baseline requirements: simplest system ATLAS can use and must absolutely be production-quality
- Requirements for a File Transfer Service:
- GridFTP
- SRM get and put. SRM copy useful.
- Delegate user credentials (GSI, MyProxy)
- FIFO queue
- All requirements already met with proposed interface
- Now will check gLite FTS implementation
- Other possibilities:
- Globus RFT: SRM support?
- CMS PhEDEX

## Future plans

- gLite FTS integration tests:
- Instead of DQ doing SRM/GridFTP calls, it will use gLite FTS as the ‘data transfer layer’
- Simple use of FIFO queue: feed gLite FTS with SURL->SURL requests
- Keep logic of handling requests and grid catalog interactions within ATLAS framework
- Schedule:
- Planned for July early-phase SC3
- ATLAS will do its own testing of gLite FTS on private testbed prior before July: starting as soon as possible
- ATLAS sees benefits on trying gLite FTS as soon as possible
- To see ASAP whether it meets medium term requirements
- Data transfer will require _x0015__significant effort_ to ramp up!
- Help debugging gLite FTS
- Transfers between more sites
- A real usage with test data
- Uniform low-level file transfer layer?

## CMS & Service Challenge 3(Preliminary)

## Service Challenge 3CMS transfers

- PhEDEx will be used for SC3 CMS transfer tests
- Available to help set up if others have interest
- Not the only CMS service that needs setting up at the sites
- Transfer features expected to be tested
- Simultaneous data import, export and serving for local processing
- Must be representative of real experiment data flow
- Must use realistic files and realistic storage
*This will become the next production service, right?*
- To/from tape at least on some Tier 1 sites
- We are working on file size
- Cannot afford to fail
- Suggest testing a couple of different configurations according to region/site preferences: SRM-SRM transfers, GridFTP only, FTS
- EGEE FTS not a high priority for CMS, may be for some sites?
*Risk for using for all sites is too high, not clear why for e.g. U.S.*

- !

## Service Challenge 3Site services

- Transfers: import and export
- For CMS tests, using PhEDEx installation at site
- Serving data to bulk data processing applications
- Simulated and/or real applications
- This requires several other services to be available
- Computing element, job submission, output harvesting for transfer, software installation + publishing into the information system, bookkeeping / monitoring databases for production, file catalogue, PubDB or successor
- The above are expected to be available concurrently
- Throughput phase: concurrent import/export transfers only
- Service phase: all, but top throughput not required

## Service Challenge 3Schedule

- CMS will participate in the “early phase” (cf. Jamie)
- July: throughput phase
- T0/T1/T2 simultaneous import/export
- To and from tape at T1s — *application level end-to-end transfers*
- Real files, real storage
- August: setup phase
- Agents work while we all enjoy our holidays?
- September: service phase 1 — modest throughput
- Demonstrate bulk data processing, simulation at T1, T2s
- Requires software, job submission, output harvesting, monitoring, ...
- Not everything everywhere, something reasonable at each site
- November: service phase 2 — modest throughput
- Phase 1 + continuous data movement
- Precursor for next production service

## Summary

- Data transfers is a substantial topic
- The interesting world is beyond “SRM-or-FTS?”...
- CMS has a production data transfer system
- Many major and smaller sites already involved
- Handles large scale continuous transfers, mostly on background
- Relatively low overhead
- Planning next steps
- Significant amount of work to ramp up everything
- Major issues remain to be sorted out
- That’s why we are here, doing service challenges
- That’s why CMS visit sites directly for technical contact
- We have to press on, service challenge or not
- It’s an exciting time, but have to move swiftly :-)

## LHCb & Service Challenges

- We would like to get access to the FTS system as early as possible
- May ?
- Our own evaluation of stability
- Start with one central Request Store instance
- Add more instances as necessary
- Do bulk transfers for Stripping data distribution to Tier1 centers
- September
- T0-T1, T1-T1 transfers
- Not part of LHCb Data Challenge

## SC3 – Milestone Decomposition

- File transfer goals:
- Build up disk – disk transfer speeds to 150MB/s
- SC2 was 100MB/s – agreed by site
- Include tape – transfer speeds of 60MB/s
- Tier1 goals:
- Bring in additional Tier1 sites wrt SC2
- PIC and Nordic most likely added later: SC4?
**Tier2 goals:**
**Start to bring Tier2 sites into challenge**
**Agree services T2s offer / require**
**On-going plan (more later) to address this via GridPP, INFN etc.**
- Experiment goals:
- Address main offline use cases ***except*** those related to analysis
- i.e. real data flow out of T0-T1-T2; simulation in from T2-T1
- Service goals:
- Include CPU (to generate files) and storage
- Start to add additional components
- Catalogs, VOs, experiment-specific solutions etc, 3D involvement, ...
- Choice of software components, validation, fallback, ...

## A Simple T2 Model

_**N.B. this may vary from region to region**_
- Each T2 is configured to upload MC data ***to*** and download data ***via*** a given T1
- In case the T1 is logical unavailable, _**wait and retry**_
- MC production might eventually stall
- For data download, _**retrieve**_ via _**alternate**_ route / T1
- Which may well be at lower speed, but hopefully rare
- Data residing at a T1 other than ‘preferred’ T1 is transparently delivered through appropriate network route
- T1s are expected to have at least as good interconnectivity as to T0
- Each Tier-2 is associated with a Tier-1 who is responsible for getting them set up
- Services at T2 are _**managed storage**_ and _**reliable file transfer**_
- DB component at T1; user agent also at T2
- 1GBit network connectivity – shared (less will suffice to start with, more maybe needed!)

## Initial Tier2 Sites for SC3

| Site | Tier1 | Experiment |
| --- | --- | --- |
| Bari, Italy | CNAF, Italy | Alice, CMS |
| Turin, Italy | CNAF, Italy | Alice |
| DESY, Germany | FZK, Germany | ATLAS, CMS |
| Lancaster, UK | RAL, UK | ATLAS |
| London, UK | RAL, UK | CMS |
| ScotGrid, UK | RAL, UK | LHCb |
| US Tier2s | BNL / FNAL | ATLAS / CMS |

## Prime Tier-2 sites

- For SC3 we aim for
- DESY   FZK  (CMS + ATLAS)
- Lancaster   RAL (ATLAS)
- London   RAL (CMS)
- Scotgrid   RAL (LHCb)
- Torino   CNAF (ALICE)
- US sites   FNAL (CMS)
- Responsibility between T1 and T2 (+ experiments)
- CERN’s role limited
- Develop a manual “how to connect as a T2”
- Provide relevant s/w + installation guides
- Assist in workshops, training etc.
- Other interested parties: Prague, Warsaw, Moscow, ..
_**Also attacking larger scale problem through national / regional bodies**_
- GridPP, INFN, HEPiX, US-ATLAS, US-CMS

| Site | Tier1 | Experiment |
| --- | --- | --- |
| Bari, Italy | CNAF, Italy | CMS |
| Turin, Italy | CNAF, Italy | Alice |
| DESY, Germany | FZK, Germany | ATLAS, CMS |
| Lancaster, UK | RAL, UK | ATLAS |
| London, UK | RAL, UK | CMS |
| ScotGrid, UK | RAL, UK | LHCb |
| US Tier2s | BNL / FNAL | ATLAS / CMS |

**More sites are appearing!**
**For CMS, also Legnaro, Rome and Pisa**
**For Atlas, sites will be discussed next week**
**Plans for a workshop in Bari end-May advancing well.**
**Tutorials also foreseen in the UK.**
**Pilot also with FZK / DESY – see May HEPiX**
**Also foreseen for Spain, France, ...**
**US sites through US-expt?**

## Slide 100

| Tier2 Region | Coordinating Body | Comments |
| --- | --- | --- |
| Italy | INFN | A workshop is foreseen for May during which hands-on training on the Disk Pool Manager and File Transfer components will be held. |
| UK | GridPP | A coordinated effort to setup managed storage and File Transfer services is being managed through GridPP and monitored via the GridPP T2 deployment board. |
| Asia-Pacific | ASCC Taipei | The services offered by and to Tier2 sites will be exposed, together with a basic model for Tier2 sites at the Service Challenge meeting held at ASCC in April 2005. |
| Europe | HEPiX | A similar activity will take place at HEPiX at FZK in May 2005, together with detailed technical presentations on the relevant software components. |
| US | US-ATLAS and US-CMS | Tier2 activities in the US are being coordinated through the corresponding experiment bodies. |
| Canada | Triumf | A Tier2 workshop will be held around the time of the Service Challenge meeting to be held in Triumf in November 2005. |
| Other sites | CERN | One or more workshops will be held to cover those Tier2 sites with no obvious regional or other coordinating body, most likely end 2005 / early 2006. |

## Tier2 and Base S/W Components

- Disk Pool Manager (of some flavour...)
- e.g.  dCache, DPM, ...
- gLite FTS client (and T1 services)
- Possibly also local catalog, e.g. LFC, FiReMan, ...
- Experiment-specific s/w _**and**_ services ( ‘agents’ )
- 1 – 3 will be bundled with LCG release.
- Experiment-specific s/w will not...

## Tier2 Software Components

- Overview of dCache, DPM, gLite FTS
- + LFC and FiReMan

## File Transfer Software SC3

- Gavin McCance – JRA1 Data Management Cluster
- LCG Storage Management Workshop
- April 6, 2005, CERN

## Reliable File Transfer Requirements

- LCG created a set of requirements based on the Robust Data Transfer Service Challenge
- LCG and gLite teams translated this into a detailed architecture and design document for the software and the service
- A prototype (radiant) was created to test out the architecture and was used in SC1 and SC2
- Architecture and design have worked well for SC2
- gLite FTS (“File Transfer Service”) is an instantiation of the same architecture and design, and is the candidate for use in SC3
- Current version of FTS and SC2 radiant software are interoperable

## gLite FTS Summary

- Core FTS software is in good shape
- On its way to becoming software that can provide a manageable, reliable service
- Stress-testing underway
- Gaining operational experience with it early
- Increasing as we head to SC3
- We are now understanding how the experiment frameworks can plug onto it
- Interested to discuss further and work with experiments
_**‘**__**Pilot’ to be setup at CERN in May**_
- To allow experiments to start to evaluate / gain experience

## Disk Pool Manager (DPM) aims

- Provide a solution for the small Tier-2s in LCG-2
- This implies 1 to 10 Terabytes in 2005
- Focus on manageability
- Easy to install
- Easy to configure
- Low effort for ongoing maintenance
- Easy to add/remove resources
- Support for multiple physical partitions
- On one or more disk server nodes
- Support for different space types – volatile and permanent
- Support for multiple replicas of a file within the disk pools

## DPM Manageability

- Few daemons to install
- Disk Pool Manager
- Name Server
- SRM
- No central configuration files
- Disk nodes request to add themselves to the DPM
- Easy to remove disks and partitions
- Allows simple reconfiguration of the Disk Pools
- Administrator can temporarily remove file systems from the DPM if a disk has crashed and is being repaired
- DPM automatically configures a file system as “unavailable” when it is not contactable

## DPM Features

- DPM access via different interfaces
- Direct Socket interface
- SRM v1
- SRM v2 Basic
- Also offer a large part of SRM v2 Advanced
- Global Space Reservation (next version)
- Namespace operations
- Permissions
- Copy and Remote Get/Put (next version)
- Data Access
- Gridftp, rfio (ROOTD, XROOTD could be easily added)
- DPM Catalog shares same code as LCG File Catalog
- Possibility to act as a “Local Replica Catalog” in a distributed catalog

## DPM Status

- DPNS, DPM, SRM v1 and SRM v2 (without Copy nor global space reservation) have been tested for 4 months
- The secure version has been tested for 6 weeks
- GsiFTP has been modified to interface to the DPM
- RFIO interface is in final stage of development

## DPM - Proposal for SC3

- Provide a possible solution for the small Tier-2s
- This implies 1 to 10 Terabytes in 2005
- Focus on manageability
- Easy to install
- Easy to configure
- Low effort for ongoing maintenance
- Easy to add/remove resources
- Support for multiple physical partitions
- On one or more disk server nodes
- Replacement of ‘Classic SE’
- Only metadata operations needed (data does not need to be copied)
_**Working with Tier2s regarding evaluation**_

## CASTOR status and plans: proposals for SC3 

- 5/4/2005

## Outline

- CASTOR status & plans
- SC3 proposal
- CASTOR version
- SRM version

## CASTOR status & plans

- The CASTOR version running in production follows the original CASTOR design defined in 1999 – 2000 and fully deployed for production use in early 2002
- Name space in Oracle, MySQL  good scalability (currently >33M files)
- Known limitations:
- The stager disk residence catalog is in memory  scales to O(200k) resident files but not beyond
- No file access request throttling
- No controlled resource sharing
- Stager limitations has lead to a deployment with many stager instances
- ~40 instances, each with each its own disk pool
- Bad sharing and loadbalancing
- Difficult to operate

## CASTOR status & plans

- New stager (disk pool manager)
- Disk file residence catalog in Oracle (later also MySQL)  good scalability
- Externalized request scheduling  throttling and controlled resource sharing
- Many features targeted for easy operation and automated management
- New stager status:
- Developments started in 2003
- First version of complete new system was ready for testing in late 2004
- Running in testbed since 4 months
- Functionality OK
- Performance:
- Tier-0 requirements heavily tested with satisfactory results
- Tier-1 requirements tests still ongoing. Request handling requires more tuning
- New stager API not compatible with old stager  new SRM required

## New stager

- Not going to present the architecture here. Please refer to CASTOR presentations presented at various occasions in 2004-2005, e.g.
**IEEE MSST 2004**
- _[http://cern.ch/castor/PRESENTATIONS/2004/MSST2004/MSST2004-40.html](http://cern.ch/castor/PRESENTATIONS/2004/MSST2004/MSST2004-40.html)_** **
- _[http://cern.ch/castor/PRESENTATIONS/2004/MSST2004/MSST2004-CASTOR-1.pdf](http://cern.ch/castor/PRESENTATIONS/2004/MSST2004/MSST2004-CASTOR-1.pdf)_** **
**CHEP04**
- _[http://cern.ch/castor/PRESENTATIONS/2004/chep04/index.html](http://cern.ch/castor/PRESENTATIONS/2004/chep04/index.html)_** **
**Operational experience**
- _[http://cern.ch/castor/PRESENTATIONS/2005/CCM-20050118/NewStager.htm](http://cern.ch/castor/PRESENTATIONS/2005/CCM-20050118/NewStager.htm)_** **
- _[http://cern.ch/castor/PRESENTATIONS/2005/CCM-20050322/ALICE-experience.html](http://cern.ch/castor/PRESENTATIONS/2005/CCM-20050322/ALICE-experience.html)__ _
**API documentation**
- _[http://cern.ch/castor/DOCUMENTATION/CODE/STAGE/NewAPI/index.html](http://cern.ch/castor/DOCUMENTATION/CODE/STAGE/NewAPI/index.html)_

## CASTOR status & plans

- New stager deployment
- Roll-out plan is being prepared
- It is too early to give a definite timescale
- Gain operational experience with test instances
- Configuration, monitoring, automated management
- Understand the Oracle requirements
- Hardware, tuning, administration
- User interfaces may require more discussions with experiments
- RFIO backward compatible
- Stager commands are only partially backward compatible
- Stager API is completely new (_[http://cern.ch/castor/DOCUMENTATION/CODE/STAGE/NewAPI/index.html](http://cern.ch/castor/DOCUMENTATION/CODE/STAGE/NewAPI/index.html)__ _)
- Work out a good deployment architecture exploiting the advantages of the new stager
- Scalable catalog and resource sharing facilities fewer instances

## SC3 proposal

- CASTOR version:
- Throughput phase
- Use the new stager
- Service phase
- Can use new stager, but...
- Sharing data with production disk pools only possible if the participating experiments have been migrated
- New stager cannot share disk pools with an old stager instance

## Possible SC3 deployment using new stager

**Possible SC3 deployment using new stager**

**SRM**

- ALICE pool
- RFIO, ROOT

- CASTOR stager

- ATLAS pool
- RFIO, ROOT

- CMS pool
- RFIO, ROOT

- LHCb pool
- RFIO, ROOT

- SC3 transfer pool
- gridftp

**Internal file replication **

**gridftp**

**Stager managed disk pools**

## SRM & SC3 – proposal

- SRM version
- Strongly recommend to stay with V1.1
- It works and is well tested and known to interoperate but it took a long time and a lot of efforts to get there
- Enough functionality for the throughput phase
- Going for a newer version is associated with high risks
- Will not interoperate out-of-the-box!
- True (and respected) space reservations is well understood, e.g. what does 1TB reserved space mean?
- I can write a 1TB file without hitting ENOSPC
- I can write 1000 x 1GB files, each without hitting ENOSPC
- V2.1 definition does not address this problem
- The operational nightmare: “ls”
- You would better gridftp the SOAP message resulting from srmLs on some castor directories

## Status of SRM definition

- SRM v1.1 insufficient – mainly lack of pinning
- SRM v3 not required – and timescale too late
- Require Volatile, Permanent space; Durable not practical
- Global space reservation: reserve, release, update (mandatory LHCb, useful ATLAS,ALICE). Compactspace NN
- Permissions on directories mandatory
- Prefer based on roles and not DN (SRM integrated with VOMS desirable but timescale?)
- Directory functions (except mv) should be implemented asap
- Pin/unpin high priority
- srmGetProtocols useful but not mandatory
- Abort, suspend, resume request : all low priority
- Relative paths in SURL important for ATLAS, LHCb, not for ALICE

*CMS input/comments not included yet*

## SC3 – Milestone Decomposition

- File transfer goals:
- Build up disk – disk transfer speeds to 150MB/s
- SC2 was 100MB/s – agreed by site
- Include tape – transfer speeds of 60MB/s
- Tier1 goals:
- Bring in additional Tier1 sites wrt SC2
- PIC and Nordic most likely added later: SC4?
- Tier2 goals:
- Start to bring Tier2 sites into challenge
- Agree services T2s offer / require
- On-going plan (more later) to address this via GridPP, INFN etc.
- Experiment goals:
- Address main offline use cases ***except*** those related to analysis
- i.e. real data flow out of T0-T1-T2; simulation in from T2-T1
- Service goals:
- Include CPU (to generate files) and storage
- Start to add additional components
- Catalogs, VOs, experiment-specific solutions etc, 3D involvement, ...
- Choice of software components, validation, fallback, ...

## Service Goals

- Expect relatively modest increase in ‘service’ components
- File catalog based on agreement from Baseline Services WG
- Other services agreed by BSWG
- Experiment-specific components and impact on other services, e.g. Distributed Database Services, need to be clarified as soon as possible
- Similarly, requirements for processing power and storage at all sites involved (T0, T1, T2)
- (This is for both Service and Challenge phases: where we run the experiments’ s/w and store the output!)

## SC3 – Milestone Decomposition

- File transfer goals:
- Build up disk – disk transfer speeds to 150MB/s
- SC2 was 100MB/s – agreed by site
- Include tape – transfer speeds of 60MB/s
- Tier1 goals:
- Bring in additional Tier1 sites wrt SC2
- PIC and Nordic most likely added later: SC4?
- Tier2 goals:
- Start to bring Tier2 sites into challenge
- Agree services T2s offer / require
- On-going plan (more later) to address this via GridPP, INFN etc.
- Experiment goals:
- Address main offline use cases ***except*** those related to analysis
- i.e. real data flow out of T0-T1-T2; simulation in from T2-T1
- Service goals:
- Include CPU (to generate files) and storage
- Start to add additional components
- Catalogs, VOs, experiment-specific solutions etc, 3D involvement, ...
- Choice of software components, validation, fallback, ...

## Ramp-up of FTS and T1s

- As expected, this is proving to be a significant amount of work!
- Need to keep up momentum:
- Increasing file transfer performance;
- Adding tape;
- Building up network links;
- Bringing additional T1s into the challenges...

## File Transfer Milestones

- Need to understand how infrastructure at CERN will build up to address file transfer goals of SC3
- Internal / External network infrastructure
- File transfer Server nodes
- Access to Experiment files
- For each Tier1 need to understand how local infrastructure and network connectivity will be build up to meet requirements
- For Tier2s, assume “Minimal Usable Model” (see earlier slide)...
- Assume DB for FTS will run at T1 with “client” also at T2
- Transfers can be initiated from both ends

## ‘SC3’ pilot services

- Not really restricted to SC3, except timewise...
- gLite FTS pilot
- For experiments to start to get experience with it, integrating with their frameworks etc
- File catalogs
- Both LFC and FiReMan... and RLS continues for the time being...
- Expect these to start in May, based on s/w delivered this week (code free April 15) for release end May
- Max one more cycle for SC3
- end June is too late for throughput phase!
- LFC is already part of LCG releases

## up to 4Gbps

- _[edoardo.martelli@cern.ch](mailto:edoardo.martelli@cern.ch)_ 20050408

- cernh7

-  General Purpose Network backbone

- Tapes

- Disks

- Database

- 10Gb link

- 1Gb link

- 2x1Gb links

**up to 4Gbps**

**up to 8Gbps**

- 44/88 Machines

- 192.16.160.0/22

- 192.16.160.1

- 192.16.160.254

- 4x1Gb

- nx1Gb

## 2005 Sep-Dec -  SC4 preparation

- In parallel with the SC3 model validation period,in preparation for the first  2006 service challenge (SC4) –
- Using 500 MByte/s test facility
**test PIC and Nordic T1s**
**and T2’s that are ready (Prague, LAL, UK, INFN, ..)**
- Build up the production facility at CERN to 3.6 GBytes/s ** **
- Expand the capability at all Tier-1s to full nominal data rate

***Historical slides from Les / Ian***

## 2006 Jan-Aug  -  SC4

- SC4 – full computing model services
- 	- Tier-0, ALL Tier-1s, all major Tier-2s operational        at full target data rates (~2 GB/sec at Tier-0)-  acquisition -  reconstruction - recording – distribution,       ***PLUS *** ESD skimming, servicing Tier-2s
- Goal – stable test service for one month – April 2006
- 100% Computing Model Validation Period (May-August 2006)
- 	Tier-0/1/2 full model test - **All experiments**
- 	- 100% nominal data rate, with processing load scaled to 2006 cpus

***Historical slides from Les / Ian***

## SC4 Planning

- Discussing a joint workshop with ARDA focussing on SC4 after the summer
- Tentative dates: during week of October 10 – 14
- Clash with GDB and HEPiX?
- After we have concrete results from SC3?
- When a more concrete model for analysis has appeared?
_**In all events, early enough for SC4 planning...**_

## 2006 Sep – LHC service available

- The SC4 service becomes the permanent LHC service – available for experiments’ testing, commissioning, processing of cosmic data, etc.
- All centres ramp-up to capacity needed at LHC startup
- TWICE nominal performance
- Milestone to demonstrate this 3 months before first physics data    April 2007

***Historical slides from Les / Ian***

## Key dates for Connectivity

## Key dates for Services

## Additional threads started to address:

Experiment involvement;
Bringing T2s in SC3;
Longer-term goals of bringing all T2s into the LCG Service (Challenges)

The enthusiasm and support provided to these new activities is much appreciated

We have a lot of work ahead...

...but the problem is beginning to become tractable(?)

- Additional threads started to address:
- Experiment involvement;
- Bringing T2s in SC3;
- Longer-term goals of bringing all T2s into the LCG Service (Challenges)
- The enthusiasm and support provided to these new activities is much appreciated
- We have a lot of work ahead...
- ...but the problem is beginning to become tractable(?)

## Conclusions

- To be ready to fully exploit LHC, significant resources need to be allocated to a series of _**Service Challenges**_ by all concerned parties
- These challenges should be seen as an _**essential**_ on-going and _**long-term**_ commitment to achieving production LCG
- The countdown has started – we are already in (pre-)production mode
- Next stop: 2020

## Postscript

**The ****Service**** is ****the ****Challenge**