# Presentation: 075964

## LHC Computing Grid Deployment

- Ian Bird
- IT Division, CERN
- LCG Deployment Area Manager
- US LHC Software and Computing Review
- January 14, 2003

## Outline

- LCG Phase I Deployment Goals
- Deployment Organisation
- LCG Deployment Plan
- Middleware
- Test beds and Services
- Certification, Testing, Packaging, Distribution
- Operations and Security
- Support
- Collaborative projects
- Summary

- US LHC Computing Review / Jan 14,2003

## LCG-1 Deployment Goals

- Production service for Data Challenges in 2H03 & 2004
- Initially focused on batch production work
- Experience in close collaboration between the Regional Centres
- Must have wide enough participation to understand the issues,
- Learn how to maintain and operate a global grid
- Focus on a production-quality service
- Robustness, fault-tolerance, predictability, and supportability take precedence; additional functionality gets prioritized
- LCG should be integrated into the sites’ physics computing services – should not be something apart
- This requires coordination between participating sites in:
- Policies and collaborative agreements
- Resource planning and scheduling
- Operations and Support

- US LHC Computing Review / Jan 14,2003

## Elements of an LCG Service

- Middleware:
- Testing and certification
- Packaging, configuration, distribution and site validation
- Support – problem determination and resolution; feedback to middleware developers
- Operations:
- Grid infrastructure services
- Site fabrics run as production services
- Operations centres – trouble and performance monitoring, problem resolution – 24x7 globally
- Support:
- Experiment integration – ensure optimal use of system
- User support – call centres/helpdesk – global coverage; documentation; training

- US LHC Computing Review / Jan 14,2003

## Deployment Organisation

- Grid Deployment Board (GDB) – chair Mirco Mazzucato
- Representatives from the experiments and from each country with an active Regional Centre taking part in the LCG Grid Service
- Forges the agreements, takes the decisions, defines the standards and policies that are needed to set up and manage the LCG Global Grid Services
- Coordinates the planning of resources for physics and computing data challenges
- First task is the detailed definition of LCG-1
- includes defining the set of grid middleware tools to be deployed
- LCG Deployment Group - coordinated by Deployment Area Manager
- Teams at CERN and Regional Centres
- Implement the elements of the LCG service:
- Certification, testing etc; Operations; Support; Scheduling
- Guided by agreements negotiated by GDB
- Reports to Project Execution Board; SC2

- US LHC Computing Review / Jan 14,2003

## GDB Status

- 1st Meeting in Milano – Oct 4, 2002, 4 since
- Set up Technical Working groups:
- WG1: Define LCG-1 functionality and services
- WG2: Define the schedule for rolling out the infrastructure and resources.  Propose process & metrics to be used for allocation, accounting, and reporting.
- WG3: Define a straightforward security and authentication model to be used in LCG-1, and identify the technical issues. Set up agreements to enable implementation.
- WG4: Define ops procedures & responsibilities. Make agreements to ensure coordination of these activities.  Define the requirements for a Grid Operations Centre to coordinate operational activities.
- WG5: Propose a support model for LCG-1, including the scope of responsibilities for call centre/helpdesk (delayed until Feb 03).
- Status:
- LCG-1 on track to be defined by end Jan 03 – certainly in essentials needed to progress – functionality and services, participating sites, resources and schedules, initial operational and security models
- Working group final reports end ~Jan 03 – should indicate where LCG should focus effort, process for agreements, etc.

- US LHC Computing Review / Jan 14,2003

## LCG Deployment PlanLevel 1 Milestones

| M1.1 - July 03 | First Global Grid Service (LCG-1) available |
| --- | --- |
| M1.2 - June 03 | Hybrid Event Store (Persistency Framework) available for general users |
| M1.3a - November 03 | LCG-1 reliability and performance targets achieved |
| M1.3b - November 03 | Distributed batch production using grid services |
| M1.4 - May 04 | Distributed end-user interactive analysis from  “Tier 3” centre |
| M1.5 - December 04 | “50% prototype” (LCG-3) available |
| M1.6 - March 05 | Full Persistency Framework |
| M1.7 - June 05 | LHC Global Grid TDR |

- US LHC Computing Review / Jan 14,2003

## LCG Phase I Timescale in a nutshell

- LCG-1 must be defined – end Jan 03
- 2 major areas to be addressed by GDB working groups
- Define LCG-1 in terms of required functionality and services
- Deployment schedule
- Set up distributed organisational structure
- Resources and scheduling,
- Policies – security, authentication, etc.
- Operational agreements and responsibilities
- Support services
- LCG-1 service must be in place – July 2003
- 6 months testing, integration, certification, packaging and deployment
- Need to demonstrate performance – end 2003
- This should include adding current production services into LCG
- Provide production service for data challenges in 2004
- LCG-3 Follows LCG-1 by 1 year – provides “50% complexity” service in 2005.

- US LHC Computing Review / Jan 14,2003

## Activities to Achieve:Initial Availability of First Global Service

- Define LCG-1 in terms of
- functionality, resources, operations, security, support
- Deploy a series of evolving pilot services for testing, with increasing resources
- First pilot service – Feb 1 2003
- Incremental deployment to Tier 1 and Tier 2 centres:
- ~10 in 3 continents by June
- Testing, certification, packaging and release of software
- Certification, testing, release process defined – January 2003.
- Packaging/configuration mechanism defined– March 2003.
- Delivery of middleware software packages – March 1, 2003
- Iterative, incremental release cycle, with major functional releases:
- V1.0 – June 1, 2003

- US LHC Computing Review / Jan 14,2003

## activities – cont.

- Set up infrastructure and operational procedures
- Certificate Authorities and VO management systems in place – May 2003
- Based on existing EU and US inter-operating systems
- Resource accounting and reporting procedures set up – May 2003
- Security procedures defined and agreed and in place– June 2003
- Incident response and security management
- Set up operations centre and help desk (call centre)
- Identify operations and call centre locations – February 1, 2003
- In place by June 2003
- LCG-1 commissioning and acceptance
- 30 day commissioning period with user productions and stress tests, including 7 day acceptance period

- US LHC Computing Review / Jan 14,2003

## Activities to Achieve:LCG-1 Fully operational

- Define LCG-1 performance goals – July 2003
- In concert with experiments and their data challenge requirements, set performance goals in terms of capacity, throughput, reliability, etc.  A GDB working group.
- 10 Regional Centres participating – October 2003
- WG2 defines the implementation schedule – may be adjusted in July.
- LXBatch service merged into LCG-1 – October 2003
- All resources of LXBATCH will be grid-enabled and accessible as part of the LCG-1 service.  This is a CERN activity but hopefully reflects what happens at other sites too.
- Milestone release of middleware – October 2003
- V1.1 release with improved functionality – October 2003
- Review of service – November 2003
- The LCG-1 service level should be that required for the 2004 data challenges.  The determination and acceptance of achieving the target will be done in a review of the service by representatives from the experiments, the regional centres and LCG.

- US LHC Computing Review / Jan 14,2003

## Deployment Details

## Middleware

- Deployed middleware to be based on US and EU toolkits:
- VDT
- Globus, Condor, GLUE schema, EDG CA and VO tools, etc.
- EDG
- Resource broker
- Reptor - WP2 (Data Management) – using RLS
- WP4 will be used to manage CERN fabric (available for others)
- VOMS
- Monitoring tools
- Initially based on work done by Worldgrid (iVDGL + DataTag)
- Specifics (functionality, version, delivery) being firmed up now by GDB WG1 – final by beginning February.
- This will provide the initial basic functionality and will evolve significantly
- LCG will focus on building a robust service – changes in basic functionality driven by experiments
- Deployment of these components involves not only obtaining the software but also agreeing the essential support and maintenance.

- US LHC Computing Review / Jan 14,2003

## Testbeds and Services

- The deployed systems will be in several versions and functions:
- Certification testbeds – both local and distributed
- Integration of middleware components, Controlled changes, in-depth application testing
- Prepare for release
- Production service – deployed at Regional Centres
- Development service – deployed at Regional Centres
- Certification testbeds parallel Production and Development services – i.e. need to debug and stabilise production release in parallel with development
- This is 2 prongs of the “Gordon trident” –
- 3rd prong are the grid projects’
- development systems:

- iVDGL
- Datagrid

**Production Grid**

**developers’  testbed**

**development testbed**

- LCG

- US LHC Computing Review / Jan 14,2003

## Timelines – LCG Phase 1

**Incremental middleware releases**

**Incrementally add regional centres  ******

- ATLAS

- LHCb

_**Data Challenges**_

**LCG-1 Defined**

**LCG-1 Initial Service Available**

**LCG-1 Full Service Available**

**LCG-1 Fulfils Performance Goals**

**LCG-3 Fulfils Performance Goals**

**Computing TDR**

- US LHC Computing Review / Jan 14,2003

## Certification and Testing

- Will be an ongoing major activity of LCG
- Part of what will make LCG a production-level service
- Goals:
- Certify/validate that middleware behaves as advertised and provides the required functionality (HEPCAL)
- Stabilise and robustify middleware
- Provide debugging, problem resolution *and feedback to developers*
- Testing activities at all levels
- Component/unit tests
- Basic functional tests, including tests of distributed (grid) services
- Application level tests – based on HEPCAL use-cases
- Driven/implemented by the experiments – GAG set up by SC2
- Experiment beta-testing before release
- Site configuration verification
- JTB collaborative project  - LCG, Trillium, EDG
- Gather existing tests
- Write/obtain missing tests

- US LHC Computing Review / Jan 14,2003

## Certification & TestingTestbeds

- CERN testbed
- Several “clusters” forming a local grid
- Basic tests, basic grid functionality
- Distributed testbed
- CERN testbed + testbeds at a few other remote sites
- Grid functionality
- Application benchmarks and experiment beta testing
- Needs several versions
- Current production version – for reproducing and fixing problems
- Development version
- + OS versions ...

- US LHC Computing Review / Jan 14,2003

## Certification & TestingRelease Strategy

- Small release cycles with incremental functionality, rather than major releases where many things change
- Somewhat depends on technology suppliers and their responsiveness to LCG needs, since LCG is not in control of development
- There will however, be milestone functional releases in June and October 2003.
- Continuous, evolutionary process
- Each release goes through certification/test cycle
- Only way to keep control of bugs
- Goal is stability and robustness ...

- US LHC Computing Review / Jan 14,2003

## Packaging and distribution

- Obviously a major issue for a deployment project
- Joint activity started –
- Discussions LCG, EDG, VDT, EDT, iVDGL, etc.
- Have produced a draft discussion document
- Will soon lead to a JTB joint project
- Want to provide a tool that satisfies needs of the participating sites,
- Interoperate with existing tools where appropriate and necessary
- Does not force solution on sites with established infrastructure
- Solution for sites with nothing
- Configuration is essential component
- Essential to understand and validate correct site configuration
- Effort will be devoted to providing configuration tools
- Verification of correct configuration will be required before sites join LCG

- US LHC Computing Review / Jan 14,2003

## LCG Operations

- Responsible for operating and maintaining the grid infrastructure and associated services
- Gateways, information services, resource broker etc. – i.e. grid specific services
- Will be a coordination between teams at CERN and at Regional Centres
- Responsible also for the VO infrastructure, Authentication and Authorisation services
- Security operations – incident response etc.
- Build Grid Operations Centre(s)
- Performance and problem monitoring; troubleshooting and coordination with site operations, user support, network operations etc.
- Leverage existing experience/ideas (WorldGrid – iVDGL,EDT, etc.)
- Started discussions with DataTag about future developments
- Indian group provide development effort
- LCG site to lead this (FZK? – not certain yet)
- Once have a activity lead will expand collaborative activities
- Assemble monitoring, reporting, performance, etc. tools
- Start with what exists, understand what is missing and needed and build from there

- US LHC Computing Review / Jan 14,2003

## Grid Operation		                          

- User

- Network
- Operations
-  Centre

- Local operation

- Local user support

- Grid Operations Centre

- Call Centre

- Local site

- Grid operations

**Grid **
**information**
**service**

**Virtual**
**Organisation**

**Grid **
**logging &**
**bookkeeping**

**queries**
**monitoring & alarms**
**corrective actions**

- US LHC Computing Review / Jan 14,2003

## Security

- GOAL: Do not want to make exceptions for LCG services – they must run integrated into a site infrastructure, and be subject to all usual security and good management procedures and policies
- BUT: Initially, certain to need exceptions and compromises since until now most grid middleware has sidestepped security issues
- THUS: We must have a sound security policy and an agreed plan that provides for these exceptions in the short term, but shows a clear path to reach the state that the sites require

- US LHC Computing Review / Jan 14,2003

## User Support

- Essential for a production service
- Two aspects
- Experiment integration/ consultancy
- Work directly with the experiments’ computing projects to ensure efficient use of LCG services, and optimum use of resources
- Act as liaison to ensure experiment specific issues are resolved
- User support
- Helpdesk/call centre operation
- Globally distributed – 24x7, ensure single point of contact for user
- Collaborative and distributed operation
- Documentation
- Training, tutorials, etc.

- US LHC Computing Review / Jan 14,2003

## Collaborative Projects

- LCG is not a middleware development project and can only succeed by leveraging the existing and ongoing work of the various grid development projects – and (hopefully) becoming a focus for them.
- There are many opportunities for common solutions, which are being actively pursued
- HICB – JTB
- GLUE
- Schema definitions & interoperability work
- New collaborative activities:
- Validation and Test Suites
- Distribution, Meta-Packaging, Configuration
- Grid Operations Centre
- DataTag, iVDGL, DTF, etc
- Storage interfaces; e.g. SRM
- Authentication, authorisation and security
- Security managers are beginning to collaborate in the context of LCG
- HEPiX/LCCWS as collaborative vehicle for RC managers, site coordinators
- E.g. certification process for operating environments; upgrade procedures; configuration management; helpdesk tools, etc.
- GGF – production grids area, etc.

- US LHC Computing Review / Jan 14,2003

## Deployment Summary

- Deploy middleware to support essential functionality, but goal is to evolve and incrementally add functionality
- Added value is to robustify, support and make into a 24x7 production service
- How?
- Certification & test procedure – tight feedback to developers
- must develop support agreements with grid projects to ensure this
- Define missing functionality – require from providers
- Provide documentation and training
- Provide missing operational services
- Provide a 24x7 Operations and Call Centre
- Guarantee to respond
- Single point of contact for a user
- Make software easy to install – facilitate new centres joining
- Deployment is a major activity of LCG
- Encompasses all operational and practical aspects of a grid
- There is a lot of work already done that must be leveraged
- Many opportunities for synergy and collaboration

- US LHC Computing Review / Jan 14,2003