# Presentation: 369992

## EDG WP4 (fabric mgmt): status&plans

- Large Cluster Computing Workshop
- FNAL, 22/10/2002
- Olof Bärring

## Outline

- What’s “EDG” and “WP4” ??
- Recap from LCCWS 2001
- Architecture design and the ideas behind...
- Subsystem status&plans&issues
- Configuration mgmt
- Installation mgmt
- Monitoring
- Fault tolerance
- Resource mgmt
- Gridification
- Conclusions

## “EDG” == EU DataGrid project

- Project started 1/1/2001 and ends 31/12/2003
- 6 principal contractors: CERN, CNRS, ESA-ESRIN, INFN, NIKHEF/FOM, PPARC
- 15 assistant contractors
- ~150FTE
- _[http://www.](http://www.eu-datagrid.org/)__[eu](http://www.eu-datagrid.org/)__[-](http://www.eu-datagrid.org/)__[datagrid](http://www.eu-datagrid.org/)__[.org](http://www.eu-datagrid.org/)_
- 12 workpackages

## “WP” == workpackage

_**EDG WPs**_
**WP1: Workload Management**
**WP2: Grid Data Management**
**WP3: Grid Monitoring Services**
**WP4: Fabric management**
**WP5: Mass Storage Management**
**WP6: Integration Testbed – Production quality International Infrastructure**
**WP7: Network Services**
**WP8: High-Energy Physics Applications**
**WP9: Earth Observation Science Applications**
**WP10: Biology Science Applications**
**WP11: Information Dissemination and Exploitation**
**WP12: Project Management**

## WP4: main objective

**“****To deliver a computing fabric comprised**
**  ****of all the necessary tools to manage a center**
**  ****providing grid services on clusters of**
**  ****thousands of nodes****.”**

## WP4: structure

- ~14 FTEs (6 funded by the EU). Presently split over ~ 30 - 40 people
- 6 partners: CERN, NIKHEF, ZIB, KIP, PPARC, INFN
- The development work divided into 6 subtasks

## Recap from LCCWS-1

- EDG WP4 presentations in LCCWS-1 //-sessions

| Session | What we said | What happened |
| --- | --- | --- |
| Installation | Plans for using the LCFG tool from Edinburgh Univ. as an interim installation/maintenance system | LCFG in production on EDG testbed since 12 months. Will be replaced by new system 2Q03. |
| Monitoring | PEM vs. WP4. Design for node autonomy where possible | System deployed on EDG testbed since one month |
| Grid | Early architecture design ideas and development plans up to Sept. 2001 | Architecture design refined and adopted. Delivery OK. |

- Not everything worked smoothly
- Architecture design: had to reach consensus between partners with different agendas and motivations.
- Delivered software: we learned some lessons and had taken some uncomfortable decisions

## Architecture design and the ideas behind

- Information model. Configuration is distinct from monitoring
- Configuration == desired state (what we want)
- Monitoring == actual state (what we have)
- Aggregation of configuration information
- Good experience with LCFG concepts with central configuration template hierarchies
- Node autonomy. Resolve local problems locally if possible
- Cache node configuration profile and local monitoring buffer
- Scheduling of intrusive actions
- Plug-in authorization and credential mapping

## DataGrid Architecture

**Collective Services**

**Information & Monitoring**

**Replica Manager**

**Grid Scheduler**

**Local Application**

**Local Database**

**Underlying Grid Services**

**Computing Element Services**

**Authorization Authentication and Accounting**

**Replica Catalog**

**Storage Element Services**

**SQL Database Services**

**Fabric services**

**Configuration**
**Management**

**Node **
**Installation &**
**Management**

**Monitoring**
**and**
**Fault Tolerance**

**Resource Management**

**Fabric Storage**
**Management**

- Grid

- Fabric

- Local Computing

- Grid

**Grid Application Layer**

**Data Management**

**Job Management**

**Metadata Management**

**Object to File Mapping**

**Service Index**

- WP4 tasks

## WP4 Architecture logical overview

**WP4 Architecture logical overview**

**- ****Interface Grid-wide services with local fabric**
**- Provides local authorization and mapping of grid credentials**.

**- ****provides transparent access (both job and admin) to different cluster batch systems**
**- enhanced capabilities (extended scheduling policies, advanced reservation, local accounting)**

**provides a central storage and management of all fabric configuration information**
**Compile HLD templates to LLD node profiles**
**- central DB and set of protocols and APIs to store and retrieve information**

**- ****provides the tools to install and manage all software running on the fabric nodes**
**Agent to install, upgrade, remove and configure software packages on the nodes**
**-bootstrap services and software repositories**

**- ****provides the tools for gathering monitoring information on fabric nodes**
**central measurement repository stores all monitoring information**
** ****fault tolerance correlation engines detect failures and trigger recovery actions**

## User job management (Grid and local)

**Farm A (LSF)**

**Farm B (PBS**)

**Grid User**

- (Mass storage,

- Disk pools)

**Local User**

**Monitoring**

**Fabric****Gridification**

**Resource****Management**

**Grid Info****Services****(WP3)**

**Resource****Broker****(WP1)**

**Data Mgmt****(WP2)**

**Grid Data****Storage****(WP5)**

**- Submit job**

**- Optimized selection of site**

**Authorization**
**Map grid ******** local credentials**

**Select an optimal batch queue and submit**
**Return job status and output**

**- publish resource and accounting information**

## Automated management of large clusters

**- Update configuration templates**

**- Node malfunction detected**

**Remove node from queue**
**Wait for running jobs(?)**

**- Trigger repair**

**- Repair (e.g. restart, reboot, reconfigure, ...)**

**- Node OK detected**

**Put back node in queue**

## Node autonomy

**Cfg cache**

**Monitoring**
**Buffer**

**Correlation**
**engines**

**Node mgmt**
**components**

**Monitoring**
**Measurement**
**Repository**

**Configuration**
**Data Base**

**Central (distributed)**

**Buffer copy**

**Cache Node profile**

**Local recover if possible**
**(e.g. restarting daemons)**

**Automation**

## Subtasks: configuration management

- Server

- XML

- HLDL

- PAN

- DBM

- Notification
- + Transfer

- Low Level API

- Access API

**Template**

**Subtasks: configuration management**

## Configuration templates like this ...

**####################################**
**object template**** TEST_i386_rh72;**
**"/system/platform" = "i386_rh72";**
**"/system/network/interfaces/0/ip" = “192.168.0.1";**
**"/system/network/hostname" = “myhost";**
**include**** node_profile;**

**# ****Default node profile**
**####################################**
**template**** node_profile;**
**# ****Include validation functions**
**##############################**
**include**** functions;**
**# ****Include basic type definitions**
**################################**
**include**** hardware_types;**
**include**** system_types;**
**include**** software_types;**
**# ****Include default configuration data**
**####################################**
**include**** default_hardware;**
**include**** default_system;**
**include**** default_software;**

**# ****SYSTEM: Default configuration**
**#########################**
**template**** default_system;**
**# ****Include default system configuration**
**######################################**
**include**** default_users;**
**include**** default_network;**
**include**** default_filesystems;**

**# ****SYSTEM: Default network configuration**
**###################################**
**template**** default_network;**
**"/system/network" = ****value****("//network_" + **
**	****	****value****("/system/platform") +**
**	****	****"/network");**

**Configuration templates like this ...**

**Notes:**
The template hierarchy describes the aggregation of information into complete deployable configurations.

## ... generate XML profile like this

-  - <**nlist name**="**profile**" **derivation**="**TEST_i386_rh72,node_profile,functions,hardware_types,...**
**    ...****..**
**   ****-**** <****nlist name****="system"**** derivation****="TEST_i386_rh72"**** type****="record">**
**       **** **** ****<****string name****="platform"**** derivation****="TEST_i386_rh72">i386_rh72</****string****>**
**        ****-**** <****nlist name****="network“**
**              ****derivation****="TEST_i386_rh72,default_network,network_i386_rh72,std_network“**
**               ****type****="record">**
** ****               ****<****string name****="hostname"**** derivation****="functions,std_network">myhost</****string****>**
**              ****-**** <****list name****="interfaces"**** derivation****="std_network">**
**                   ****-**** <****nlist name****="0"**** derivation****="std_network_interface,std_network"**** type****="record">**
**          ****- ****<****string name****="name"**** derivation****="std_network_interface">eth0</****string****>**
**                        ****-****  <****string name****="ip"**** derivation****="functions,std_network_interface">192.168.0.1</****string****>**
**                        ****- ****<****boolean name****="onboot"**** derivation****="std_network_interface">true</****boolean****>**
**                       ****</****nlist****>**
**                  ****</****list****>**
**   ...****..**

**... ****generate XML profile like this**

- Description of the High Level Definition Language (HLDL), the compiler and the Low Level Definition Language (LLDL) can be found at: _[http://](http://cern.ch/hep-proj-grid-fabric-config)__[cern](http://cern.ch/hep-proj-grid-fabric-config)__[.](http://cern.ch/hep-proj-grid-fabric-config)__[ch](http://cern.ch/hep-proj-grid-fabric-config)__[/hep-](http://cern.ch/hep-proj-grid-fabric-config)__[proj](http://cern.ch/hep-proj-grid-fabric-config)__[-grid-fabric-](http://cern.ch/hep-proj-grid-fabric-config)__[config](http://cern.ch/hep-proj-grid-fabric-config)_

## Global configuration schema tree

**hardware**

**system**

**software**

**CPU**

**harddisk**

**memory**

**...****.**

**sys_name**

**interface_type**

**size**

**...****.**

**network**

**platform**

**partitions**

**services**

**...****.**

**hda1**

**size**

**type**

**id**

**hda2**

**...****.**

**packages**

**known_repositories**

**edg_lcas**

**edg_lcas**

**...****.**

**version**

**repositories**

**...****.**

**cluster**

**...****.**

**Component specific configuration**

- The population of the global schema is an ongoing activity
- _[http://edms.cern.ch/document/352656/1](http://edms.cern.ch/document/352656/1)_

**i386_rh72**

**Notes:**
The global configuration schema tree describes the organisation of the configuration parameter space.

## Subtask: installation management

**Node Configuration Deployment**
**Base system installation**
**Software Package Management**

## Node configuration deployment

- Server

- XML

- HLDL

- PAN

- DBM

- Notification
- + Transfer

- Low Level API

- Access API

**Template**

**Node configuration deployment**

## Node configuration deployment infrastructure

**Node View Access (NVA) API**

**DBM Cache**

- Invocation

- registration &
- notification

- XML profiles

- Configure()

- “low level” API

**Configuration Dispatch daemon (cdispd)**

## Component example

**sub Configure {**
**  ****my ($self) = @_;**
**  ****# ****access configuration information**
**  ****my $config=NVA::Config->new(); **
**  ****my $arch=$config->getValue('/system/platform’); # ****low-level API**
**  ****$self->Fail (“not supported") unless ($arch eq ‘i386_rh72’);**
**  ****# ****(re)generate and/or update local config file(s)**
**  ****open (myconfig,’/etc/myconfig’); **
**      ****...**
**  ****# ****notify affected (SysV) services if required**
**  ****if ($changed) {**
**    ****system(‘/sbin/service myservice reload’); ...**
**  ****}**
**}**

## Base Installation and Software Package management

- Use of standard tools
- Base installation
- Generation of kickstart or jumpstart files from node profile
- Software package management
- Framework with pluggable packager
- rpm
- pkg
- ??
- It can be configured to respect locally installed packages, ie. it can be used for managing only a subset of packages on the node (useful for desktops)

## Software Package Management (SPM)

- Packages (RPM, pkg)

- RPM db

- Local
- Config
- file

- Transaction set

- HTTP(S), NFS, FTP

- Installed pkgs

- “desired”
- configuration

- SPM Component

## Installation (&configuration): status

- LCFG (Local Configuration) tool from Univ. of Edinburgh has been in production at the EDG testbed since more than 12 months
- Learned a lot from it to understand what we really want
- Used at almost all EDG testbed sites  very valuable feedback from a large O(5-10) group of site admins
- Disadvantages with LCFG
- Enforces a private per component configuration schema
- High level language lacks possibilities to attach compile time validation
- Maintains propriety solutions where standards exist (e.g. base installation)
- New developments progress well and complete running system is expected by April 2003

## Subtask: fabric monitoring

- Framework for
- Collecting monitoring information from sensors running on the nodes
- Store the information in a local buffer
- Assures that data is collected and stored even if network is down
- Allows for local fault tolerance
- Transports the data to a central repository database
- Allows for global correlations and fault tolerance
- Facilitate generation of periodic resource utilisation reports
- Status: framework deployed on EDG testbed. Enhancements will come
- Oracle DB repository backend. MySQL and/or PostgreSQL also planned
- GUIs: alarm display and data analysis

## Fabric monitoring

**Fabric monitoring**

**Sensor**

**Sensor**

**Sensor**

**Agent**

**Cache**

**Repository**
**server**

**DB**

**Application**

**Sensor API**

**Cache used by local fault tolerance**

**Native DB**
** ****API (e.g. SQL)**

**Repository API**
**(SOAP RPC)**

**Transport (UDP or TCP)**

**Nodes**

**Server node**

**Desktop**

**Repository API**
**(Local access)**

**Notes:**
The fabric monitoring architecture has been designed to fulfil the following requirements
All measurements from all nodes in the computer centre are stored in a central (possibly distributed) database.  facilitate correlation and analysis of monitoring data from different nodes, network and storage devices.
Any consumer of monitoring data should use the central repository. The worker nodes should not be allowed to be clobbered in an uncontrolled way by other monitoring consumers.  Minimise footprint from monitoring
At each monitored node there is a local cache (or more correctly “buffer”) stores all recent data gathered on the node.  Allow for local fault tolerance actions without needing to contact the central repository. The local buffer is also useful to store the sometimes precious monitoring data during moments when the node cannot reach the network (for instance, the node has hung and has to be rebooted).
Using the sensor API it is easy to plug-in new metrics to the system. Currently a C++ API exists but the protocol is simple enough to allow script sensors (shell, perl) to send their data directly to the agent through a pipe.
A repository API library can be used to query the repository for monitoring data. The API also allows for subscription to new measurements of metrics. The API implementation is based on SOAP RPC. The repository API is useful for simple queries. For more complex queries the native DB interface should be used.
The current implementation of the repository server supports a flat-file database and Oracle. In later releases either MySQL or PostgreSQL will also be supported.

## Subtask: fault tolerance

- Framework consists of
- Rule editor
- Enter metric correlation algorithms and bind them to actions (actuators)
- Correlation engines implements the rules
- Subscribe to the defined set of input metrics
- Detect exception conditions determined by the correlation algorithms and report to the monitoring system (exception metric)
- Try out the action(s) and report back the success/failure to the monitoring system (action metric)
- Actuators
- Plug-in modules (scripts/programs) implementing the actions
- Status: first prototype expected by mid-November 2002

## Subtask: resource management

- Manage grid jobs and local jobs. Layer between grid scheduler and local batch system. Allows for enhancing scheduling capabilities if necessary
- Advanced reservations
- Priorities
- Provides common API for administrating underlying batch system
- Scheduling of maintenance jobs
- Draining node/queues from batch jobs
- Status: prototype exists since a couple of months. Not yet deployed on EDG testbed.

## Resource management prototype (R1.3)

-    queues                    resources
- Batch system: PBS, LSF, etc.

- Scheduler

- Runtime Control
- System

*Grid*

*Local fabric*

- Gatekeeper
- (Globus or WP4)

- job 1

- job 2

- job n

- JM 1

- JM 2

- JM n

**scheduled jobs**

**new jobs**

**user queue 2**

** ****execution queue**

- stopped, visible for users

- started, invisible for users

**submit**

**user queue 1**

**get job info**

**move**

**move job**

**exec job**

**RMS components**

**PBS-, LSF-Cluster**

**Globus components**

**Resource management prototype (R1.3)**

## Subtask: gridification

- Layer between local fabric and the grid
- Local Centre Authorisation Service, LCAS
- Framework for local authorisation based on grid certificate and resource specification (job description)
- Allows for authorisation plug-ins to extend the basic set of authorisation policies (gridmap file, user ban lists, wall-clock time)
- Local Credential Mapping Service, LCMAPS
- Framework for mapping authorised user’s grid certificates onto local credentials
- Allows for credential mapping plug-ins. Basic set should include uid mapping and AFS token mapping
- Job repository
- Status: LCAS deployed in May 2002. LCMAPS and job repository expected 1Q03.

## Conclusions

- Since last LCCWS we have learned a lot
- We do have an architecture and a plan to implement it
- Development work is progressing well
- Adopting LCFG as interim solution was a good thing
- Experience and feedback with a real tool helps in digging out what people really want
- Forces middleware providers and users to respect some rules when delivering software
- Automated configuration has become an important for implementing quality assurance in EDG
- Internal and external coordination with other WPs and projects result in significant overhead
- Sociology is an issue (see next 30 slides...)