# Presentation: 125357

## SC4 Workshop

**Dantong Yu**

## SC4 Workshop

- Update On LCG OPN.
- FTS Status and Direction.
- DPM/LFC.
- dCache.
- SRM/FTS/ Interoperability Issues.
- Services and Server Levels, and Operation Model.
- 3D projects.
- User Cases -> ATLAS User Cases For SC4.
- June SC4 Tier 2 Workshop

## LHC Networking – T0, T1, T2

- Composed of many different Infrastructures
- Relies on many different networks
- Dedicated Circuits
**USLHCNet (dedicated links)**
- Research Infrastructures
**National Research Networks (NRENS), Projects (GEANT)**
- Internet connectivity
- Has different requirements
- Raw data transmission, production, analysis
- Some objectives very clear
- To be able to get the data from the T0 to the T1’s as quickly and reliably as possible after data taking
- Imagine an “optical private network”, OPN, for this purpose, although this is part of the overall networking that will be used.

## Tier0 / Tier1 / Tier2 Networks

## Slide 5

## FTS-Support

- FTS: Point to point file movement tool for the sites to allow them to easily control, prioritize and service the transfer requests of their VOs.
- SRM v1 3rd party copy mostly understood
- Can obtain good rates / fair stability with this
- SRM copy not really understood
- Harder to debug problems
- Consequent higher support load on T1 sites
- Is it needed?
- Still to develop SRM v2 support and test properly at high transfer rates
- This will take time and effort: just starting now
- Work is common with SRM v2 support for LCG utils / GFAL
- Goal is for the SC4 service phase
- Mixed SRM version support is not understood
- What is the plan for this?
- Do we need this?

## FTS

- Development goals
- Stability, stability, stability...
- Need to work on operational issues or overall service will not be stable
- Functional improvements
- SRM v2 support
- Plug-in support from gLite 1.5 – still need to understand / agree deployment model for this
- “Site grouping” to allow easier management of “point-to-point” channels
- Point-to-point production channels will go over the OPN
- Less well used channels (e.g. random T2 to random T2) will likely go over general network
- Roll out plan
- Sites update to 1.4 as soon as possible
- Upgrade and install guides on the Wiki
- 1.4 clients are being distributed in LCG 2.7.0
- We’ll work on understanding the new features of 1.5

## CASTOR Direction: SC3 setup castorgridsc

**CASTOR Direction: SC3 setup **castorgridsc

-  Imbalanced WAN pool of 3 H/W types, 2 architectures causing performance problems

**Tier-1 WAN access**

**CERN internal network**

**cmsprod**

**lhcbprod**

**atlprod**

**aliprod**

## Castor-2 setup for SC4

## dCache Direction

*SRM V2 *
- Data Transfer and Directory methods/function available
- Explicit Space Management is in development phase
*Information Provider*
*Smart Tape System flushing (Ping-Pong Style to improve disk I/O)*
*Generic Tertiary Storage Interface*
*dCache partitioning*
*VOMS integration: *
- Pluggable authorization, Storage Authorization Call-outs, uid/gid mapping still done locally
*Replica Manager (Resilient dCache ****no HSM back end****)*
*New Protocol: xRootd: **we need help from experts*

## dCache

- Separate Pnfs server: single dCache Instance, pnfs manager proxy (/pnfs) : multiple pnfs servers: /atlas, /cms/, /generic
*dCache Book: www.dCache.ORG*
- dCache, the Book
- need specific help for you installation or help in designing your dCache instance.
*support@dCache.ORG*
*user-forum@dCache.ORG*

## WLCG Service Requirements: Define the site services

- Who is responsible ?
- What level of service is required ?
- What capacity of service ?
- What is the support structure ?
- Who pays for what ?

**WLCG Service Requirements: Define the site services**

## Service catalog approach

- A service catalog consists
- Service Class – Criticality
- Calendar – Variation with time
- Product – What application
- Customer – Which VO
- Service =
- Service Class x Calendar x Product x Customer

## Service class

- https://uimon.cern.ch/twiki/bin/view/LCG/ScFourServiceDefinition

| Class | Description | Downtime | Reduced | Degraded | Avail |
| --- | --- | --- | --- | --- | --- |
| C | Critical | 1 hour | 1 hour | 4 hours | 99% |
| H | High | 4 hours | 6 hours | 6 hours | 99% |
| M | Medium | 6 hours | 6 hours | 12 hours | 99% |
| L | Low | 12 hours | 24 hours | 48 hours | 98% |
| U | Unmanaged | None | None | None | None |

## Service questions

- What service levels are required for each calendar period ?
- Who is providing support for the application ?
- Who supports the infrastructure ?
- How should the support be contacted?
- What support service do they provide?

## Operations questions

- How are problems identified ?
- Local console ?
- Grid Monitoring ?
- Who should be contacted to resolve the problem ?
- Who should be informed of the problem ?
- What new procedures / operations guides are required ?
- What is the local coverage for nights / weekends ?
- How does local and Grid operations interwork ?

## Build Production Servers/Services

- What does production mean?
- Reach MoU agreements
- Avoid single points of failure for the services:
- In terms of hardware
- In terms of software
- In terms of people
- ‘Procedurise’:
- Production/Failure recover/maintenance/...

## General remarks

- Production Service Managers ‡ Software/Service Developer/Guru
- Procedure needed for:S/w installation, upgrade, checking,...
- Requirements needed for:h/w: disk, memory, backup, high availability
- Operations: 24/7, 9-5, piquet, problem escalation

## GRID Service issues

- Most GRID Services do not have built in failover functionality
- Stateful servers: Information lost on crashDomino effects on failuresScalability issues: e.g. >1 CE

## Different approaches

- Server hardware setup:
- Cheap ‘off the shelf’ h/w satisfies only for batch nodes (WN)
- Improve reliability (cost ^)
- Dual Power supply (and test it works)
- UPS
- Raid on system/data disks (hot swappable)
- Remote console/BIOS access
- Fiber Channel disk infrastructure
- CERN: ‘Mid-Range-(Disk-)Server’

## Various Techniques

- Where possible, increase number of ‘servers’ to improve ‘service’:
- WN, UI (trivial)
- BDII: possible, state information is very volatile and re-queried
- CE, RB,... more difficult, but possible for scalability

## Load Balancing

- For multiple Servers, use DNS alias/load balancing to select the right one:
- Simple DNS aliasing for selecting master or slave server: Useful for scheduled interventions
- Pure DNS Round Robin (no server check)configurable in DNS: to equally balance load
- Load Balancing Service, based on Server load/availability/checks: Uses monitoring information to determine best machine(s)[See CHEP talk by Vlado Bahyl]

## High Availability Linux

- Add-on to standard Linux
- Switches IP address between two servers:
- If one Server crashes
- On request
- Machines monitor each other
- To further increase high availability:State information can be on (shared) FC

## Site Function Test

- What is SFT?
- “agent” or test job sent to all sites (CEs) on regular basis (every 3 hours)
- set of tests for different aspects of functionality
- high level - service or functionality oriented
- Most important changes:
- MySQL and R-GMA instead of directory structure for storage
- tests executed in parallel
- publishing results using asynchronous web service
- additional dimension: VO - possibility of VO specific tests
- About 20 tests:
- General: Software Version, CA certs version, CSH, BrokerInfo, R-GMA (+ secure conn.), Apel
- Replica management (lcg-utils+RLS): infosys, copyAndRegister, copy, replicate, 3rd party replication, delete + similar set using LFC
- Security tests (separate restricted zone): CRL validity
- “Official” test jobs submitted in *dteam* VO (high priority queue) every 3 hours
- On demand submissions when necessary (SFT Admin page in Poznan)
- Prototype CE availability metric aggregated on site, region, grid level

## SFT workflow

- Tests are performed on/from 1 WN
- Results are related to a single CE
- Only “default” nodes of other types are tested (SE, MON, ...)

## Service Level metrics- Monitoring and Reporting

- Based on SFT, visualized by GridView
- Provide high level (managerial) views of the current status and history of the Tier 0 and Tier 1 grid services
- Keep this simple and provide a first result quickly (end of 1Q 2006)
- Plan the views to be qualitative and quantitative so they can be compared to the LCG site Memoranda of Understanding service targets for each site.
- Satisfy the recent LHCC LCG review observation:
- ‘A set of Baseline Services provided by the LCG Project to the experiments has been developed since the previous comprehensive review. A measure of evaluating the performance and reliability of these services is being implemented.’

## Service Level Agreement

- An objective is to measure the services we deliver against the MoU targets
- Data transfer rates, Service availability and time to resolve problems
- Resources available at a site (as well as measured usages)
- Resources specified in the MoU are cpu capacity (in KSi2K), tbyes of disk storage, tbytes of tape storage and nominal WAN data rates (Mbytes/sec)

| Service | Maximum delay in responding to operational problems |  |  | Average availability measured on an annual basis |  |
| --- | --- | --- | --- | --- | --- |
|  | Service stoppage | Degradation ... by more than 50% | Degradation ... by more than 20% | During accelerator operation | At all other times |
| Acceptance of data from the Tier-0 Centre during accelerator operation | 12 hours | 12 hours | 24 hours | 99% | n/a |
| Networking service to the Tier-0 Centre during accelerator operation | 12 hours | 24 hours | 48 hours | 98% | n/a |
| Data-intensive analysis services, including networking to Tier-0, Tier-1 Centres outside accelerator operation | 24 hours | 48 hours | 48 hours | n/a | 98% |
| All other services – prime service hours | 2 hour | 2 hour | 4 hours | 98% | 98% |
| All other services – outside prime service hours | 24 hours | 48 hours | 48 hours | 97% | 97% |

## Nominal MoU pp running data rates CERN to Tier 1 per VO

| Centre | ALICE | ATLAS | CMS | LHCb | Rate into T1 (pp)MB/s |
| --- | --- | --- | --- | --- | --- |
| ASGC, Taipei | - | 8% | 10% | - | 100 |
| CNAF, Italy | 7% | 7% | 13% | 11% | 200 |
| PIC, Spain | - | 5% | 5% | 6.5% | 100 |
| IN2P3, Lyon | 9% | 13% | 10% | 27% | 200 |
| GridKA, Germany | 20% | 10% | 8% | 10% | 200 |
| RAL, UK | - | 7% | 3% | 15% | 150 |
| BNL, USA | - | 22% | - | - | 200 |
| FNAL, USA | - | - | 28% | - | 200 |
| TRIUMF, Canada | - | 4% | - | - | 50 |
| NIKHEF/SARA, NL | 3% | 13% | - | 23% | 150 |
| Nordic Data Grid Facility | 6% | 6% | - | - | 50 |
| Totals | - | - | - | - | 1,600  |

***These rates must be sustained to tape 24 hours a day, 100 days a year.***
***Extra capacity is required to cater for backlogs / peaks.***

## Required Tier1 Services

| Service | VOs | Class |
| --- | --- | --- |
| SRM 2.1 | All VOs | H/M |
| LFC | ALICE, ATLAS | H/M |
| FTS | ALICE, ATLAS, LHCb, (CMS) | H/M |
| CE |  | H/M |
| Site BDII |  | H/M |
| R-GMA |  | H/M |

***Many also run e.g. an RB etc. ***

## Some discussion points

- Can we improve the consistency of the various grid monitoring interfaces:
- GRIDVIEW access point: http://gridview.cern.ch/GRIDVIEW/
- Grid Operations Centre access point  http://goc.grid-support.ac.uk/gridsite/monitoring/
- Which includes links to
- Gstat via http://goc.grid.sinica.edu.tw/gstat/
- SFT reports via https://lcg-sft.cern.ch:9443/sft/lastreport.cgi
- Gridice at http://gridice2.cnaf.infn.it:50080/gridice/site/site.php
- Who should use which monitoring displays and for what ?
- Should we try to join the CERN FIO (Fabric Infrastructure and Operation) service view framework ?
- How many levels of quality of service do we need ?
- What threshold values to define the quality of a service e.g. for
- Time for a bdii to respond to an ldap query
- Time for an RB to matchmake (find a suitable CE)
- Time for simple batch job to complete
- Time to register a new LFC entry
- What fraction of MoU targets is ‘acceptable’
- How to properly handle a null result for a test – reuse last value ?
- How to automate calculation of average KSi2K rating under a CE ?

##  Operation Models: Goal

- Operation process, includes:
- Problem detection
- Reporting
- Problem solving
- Escalation procedures
- To have a single operation process
- No EGEE – LCG division; we have a single infrastructure and should have a single process
-  In place when SC4 starts

##  EGEE Operations Structure

- Operations Management Centre (OMC)
- Core Infrastructure Centres (CIC)
- Manage daily grid operations – oversight, troubleshooting
- “Operator on Duty”
- Run infrastructure services
- UK/I, Fr, It, CERN, Ru,Taipei
- Regional Operations Centres (ROC)
- Front-line support for user and operations issues
- Provide local knowledge and adaptations
- One in each region – many distributed
- User Support Centre (GGUS)
- In FZK: provide single point of contact (service desk) + portal.

## Operations plan - Evolve support flow. March

***1******st****** level support***

***2******nd****** level support***

***3******rd****** level support***

**Operations plan - **Evolve support flow. March

## Round of the T1s : RAL, CNAF, SARA, IN2P3, BNL, FNAL, FZK, TRIUMF, ASGC, NDGF, PIC and T0

- IN2P3, RAL, FZK - we have a good automatic callout already, high availability services needed as no operator on site foreseen.
- CNAF - we fixed serious problems within 24 hours at Xmas holidays. general grid expert on call.
- FNAL – we have people for 24/7. Operator with modus operandi at extra-site call-center ~ 500$/month.
- NDGF - we will be aiming for redundancy and expect most problems to be during the first year only.
- PIC - we will rely on calling to experts and high availability services.
- BNL, Triumf - being a national lab they have 24 by 7 operators.
- Sara - we have 24 hour coverage of network operations.
- ASGC - we are going to hire four technicians.
- CERN - full cover with escalation layers from operator to sysadmin to expert.

## Distributed Database Deployment

- Availability, scalability, consistency, location independency.
- Shared deployment and administration of this infrastructure during 24 x 7 operation.
- Propose to setup for first 6 month For Tier 1.
- 2/3  dual-cpu database nodes with 2GB or more
- Setup as RAC cluster (preferably) per experiment
- ATLAS: 3 nodes with  300GB storage (after mirroring)
- LHCb: 2 nodes with 100GB storage (after mirroring)
- Shared storage (eg FibreChannel) proposed to allow for clustering
- 2-3  dual-cpu Squid nodes with 1GB or more
- Squid s/w packaged by CMS will be provided by 3D
- 100GB storage per node
- Need to clarify service responsibility (DB or admin team?)
- Target s/w release: Oracle 10gR2
- RedHat Enterprise Server to insure Oracle support
- Used for Grid Middleware and Applications (pool, cool)

## LCG 3D Service Architecture

## ATLAS plans for 2006:Computing System Commissioningand Service Challenge 4

**Dario Barberis**
**CERN & Genoa University**

## Computing System Commissioning Goals

- Main aim of Computing System Commissioning will be to test the software and computing infrastructure that we will need at the beginning of 2007:
- Calibration and alignment procedures and conditions DB
- Full trigger chain
- Event reconstruction and data distribution
- Distributed access to the data for analysis
- At the end (autumn-winter 2006) we will have a working and operational system, ready to take data with cosmic rays at increasing rates

## ATLAS Computing Model

- Tier-0:
- Copy RAW data to Castor tape for archival
- Copy RAW data to Tier-1s for storage and reprocessing
- Run first-pass calibration/alignment (within 24 hrs)
- Run first-pass reconstruction (within 48 hrs)
- Distribute reconstruction output (ESDs, AODs & TAGS) to Tier-1s
- Tier-1s:
- Store and take care of a fraction of RAW data
- Run “slow” calibration/alignment procedures
- Rerun reconstruction with better calib/align and/or algorithms
- Distribute reconstruction output to Tier-2s
- Keep current versions of ESDs and AODs on disk for analysis
- Tier-2s:
- Run simulation
- Keep current versions of AODs on disk for analysis

## ATLAS Tier-0 Data Flow

- EF

- CPUfarm

- T1

- T1

- T1s

- Castorbuffer

- RAW

- 1.6 GB/file
- 0.2 Hz
- 17K f/day
- 320 MB/s
- 27 TB/day

- ESD

- 0.5 GB/file
- 0.2 Hz
- 17K f/day
- 100 MB/s
- 8 TB/day

- AOD

- 10 MB/file
- 2 Hz
- 170K f/day
- 20 MB/s
- 1.6 TB/day

- AODm

- 500 MB/file
- 0.04 Hz
- 3.4K f/day
- 20 MB/s
- 1.6 TB/day

- RAW

- AOD

- RAW

- ESD (2x)

- AODm (10x)

- RAW

- ESD

- AODm

- 0.44 Hz
- 37K f/day
- 440 MB/s

- 1 Hz
- 85K f/day
- 720 MB/s

- 0.4 Hz
- 190K f/day
- 340 MB/s

- 2.24 Hz
- 170K f/day (temp)
- 20K f/day (perm)
- 140 MB/s

- Tape

## BNL Data Flow (2008)

- Tier-0

- CPUfarm

- BNL
- diskbuffer

- BNL
- Tape

- diskstorage

**Plus simulation ****&**** analysis data flow**

**Real data storage, reprocessing and distribution**

- 234MB*n
- analysis

## ATLAS SC4 Tests

- Complete Tier-0 test
- Internal data transfer from “Event Filter” farm to Castor disk pool, Castor tape, CPU farm
- Calibration loop and handling of conditions data
- Including distribution of conditions data to Tier-1s (and Tier-2s)
- Transfer of RAW, ESD, AOD and TAG data to Tier-1s
- Transfer of AOD and TAG data to Tier-2s
- Data and dataset registration in DB (add meta-data information to meta-data DB)
- Distributed production
- Full simulation chain run at Tier-2s (and Tier-1s)
- Data distribution to Tier-1s, other Tier-2s and CAF
- Reprocessing raw data at Tier-1s
- Data distribution to other Tier-1s, Tier-2s and CAF
- Distributed analysis
- “Random” job submission accessing data at Tier-1s (some) and Tier-2s (mostly)
- Tests of performance of job submission, distribution and output retrieval

## ATLAS SC4 Plans (1)

- Tier-0 data flow tests:
- Phase 0: 3-4 weeks in March-April for internal Tier-0 tests
- Explore limitations of current setup
- Run real algorithmic code
- Establish infrastructure for calib/align loop and conditions DB access
- Study models for event streaming and file merging
- Get input from SFO simulator placed at Point 1 (ATLAS pit)
- Implement system monitoring infrastructure
- Phase 1: last 3 weeks of June with data distribution to Tier-1s
- Run integrated data flow tests using the SC4 infrastructure for data distribution
- Send AODs to (at least) a few Tier-2s
- Automatic operation for O(1 week)
- First version of shifter’s interface tools
- Treatment of error conditions
- Phase 2: 3-4 weeks in September-October
- Extend data distribution to all (most) Tier-2s
- Use 3D tools to distribute calibration data

## ATLAS SC4 Plans (2)

- ATLAS includes continuous distributed simulation productions (Kaushik)
- SC4: distributed reprocessing tests:
- Test of the computing model using the SC4 data management infrastructure
- Needs file transfer capabilities between Tier-1s and back to CERN CAF
- Also distribution of conditions data to Tier-1s (3D)
- Storage management is also an issue
- Could use 3 weeks in July and 3 weeks in October
- SC4: distributed simulation intensive tests:
- Once reprocessing tests are OK, we can use the same infrastructure to implement our computing model for simulation productions
- As they would use the same setup both from our ProdSys and the SC4 side
- First separately, then concurrently.

## Overview of requirements for SC4

- SRM (“baseline version”) on all storages
- VO Box per Tier-1 and in Tier-0
- LFC server per Tier-1 and in Tier-0
- FTS server per Tier-1 and in Tier-0
- Permanent Storage and Durable Storage.
- separate SRM entry points for permanent and durable storages.
- Disk space is managed by DQ2.
- Counts as *online (“disk”) data *in the ATLAS Computing Model
- Ability to install FTS ATLAS VO agents on Tier-1 and Tier-0 VO Box
- Ability to deploy DQ2 services on VO Box as during SC3
**N****o new requirements on the Tier-2s besides SRM SE**

## Overview of FTS and VO Box

- Hence, an ATLAS VO Box will contain:
- FTS ATLAS agents
- And remaining DQ2 persistent services (less s/w than for SC3 as some functionality merged into FTS in the form of FTS VO agents)
- DQ2 site services will have associated SFTs for testing

## ATLAS SC4 Requirement (PPS)

- Small testbed with (part of) CERN, a few Tier-1s and a few Tier-2s to test our distributed systems (ProdSys, DDM, DA) prior to deployment
- It would allow testing new m/w features without disturbing other operations
- We could also tune properly the operations on our side
- The aim is to get to the agreed scheduled time slots with an already tested system and really use the available time for relevant scaling tests
- This setup would not interfere with concurrent large-scale tests or data transfers run by other experiments
- A first instance of such a system would be useful already now!
- April-May looks like a realistic request

## Summary of requests

- March-April (pre-SC4): 3-4 weeks in for internal Tier-0 tests (Phase 0)
- April-May (pre-SC4): tests of distributed operations on a “small” testbed (PPS)
- Last 3 weeks of June: Tier-0 test (Phase 1) with data distribution to Tier-1s
- 3 weeks in July: distributed processing tests (Part 1)
- 2 weeks in July-August: distributed analysis tests (Part 1)
- 3-4 weeks in September-October: Tier-0 test (Phase 2) with data to Tier-2s
- 3 weeks in October: distributed processing tests (Part 2)
- 3-4 weeks in November: distributed analysis tests (Part 2)

## WLCG Tier 2 Workshop

- _[https://twiki.cern.ch/twiki/bin/view/LCG/WorkshopAndTutorials](https://twiki.cern.ch/twiki/bin/view/LCG/WorkshopAndTutorials)_
- _[http://indico.cern.ch/conferenceDisplay.py?confId=1148&view=egee_meeting&showDate=all&showSession=all&detailLevel=contribution](http://indico.cern.ch/conferenceDisplay.py?confId=1148&view=egee_meeting&showDate=all&showSession=all&detailLevel=contribution)_
- Four Experiment Activities Introduction.
- MC Simulation User Cases
- An Overview of Calibration & Alignment
- Analysis Use Cases
- Services Required at / for Tier2s (Grid, Application).
- Support and Operation Issues.
- Happen in the middle of June.