# Presentation: 526698

## The GridKa Installation for HEP Computing

- Forschungszentrum Karlsruhe GmbH
- Central Information and Communication Technologies Department
- Hermann-von-Helmholtz-Platz 1
- D-76344 Eggenstein-Leopoldshafen
- Holger Marten
**http://grid.fzk.de**

## Helmholtz Foundation of German Research Centres (HGF)

**Helmholtz Foundation of German Research Centres (HGF)**

**  ****15 German research centres**
**  ****24.000 employees**
**  ****largest German scientific organization**
**  ****6 main research areas**
**  ****Structure of Matter**
**  ****Earth & Environment**
**  ****Traffic & Outer Space**
**  ****Health**
**  ****Energy**
**  ****Key Technologies**

## Slide 3

## Forschungszentrum Karlsruhe

**  ****40 institutes and divisions**
**  ****3.500 employees**
**  ****13 research programs for**
**  ****Structure of Matter**
**  ****Earth & Environment**
**  ****Health**
**  ****Energy**
**  ****Key Technologies**
**  ****many close collaborations**
**   ****with TU Karlsruhe**

## Your Key

**performance computers and IT solutions for each purpose.**
**vector computers, parallel computers, Linux Clusters, workstations, ~2500 PCs,**
**online storage, tape robots, networking infrastructure, printers and printing**
**services, central software, user support,....**
**About 90 persons in 7 departments**

**Central Information and Communication Technologies Dept.**
**(Hauptabteilung Informations- und Kommunikationstechnik, HIK)**

**Your Key**
**to Success**

## Organisational Structure

- Datendienste
- Anwendungen
- System-
- überwachung
- Infrastruktur
- (R. Kupsch)

**DASI**

- Hochleistungs-
- rechnen
- (F. Schmitz)

**HLR**

- Grid-Computing
- Infrastruktur
- und
- Service
- GridKa
- (H. Marten)

**GIS**

- Grid-Computing
- und
- e-Science
- Competence
- Centre
- (M. Kunze)

**GES**

- Netzinfrastruktur
- und
- Netzanwendungen
- (K. -P. Mickel)

**NiNa**

- PC-Betreuung
- und
- Büro-
- kommunikation
- (A. Lorenz)

**PC/BK**

- Reprografie
- (G.Dech)

**Repro**

**Zentralabtlg.**

**Sekretariat**

**HIK**
- (K.-P. Mickel)

- Zentralabteilung und Sekretariat:  IT-Innovationen, Accounting, Billing, Budgetierung, Ausbildungskoordination, sonstige zentrale Aufgaben

## Grid Computing Centre Karlsruhe -

**The mission**

**German Tier-1 Regional Centre for 4 LHC HEP-Experiments**
-    2001-04   test phase
-    2005-07   main set-up phase
-    2007+      production phase
**German Computing Centre for 4 non-LHC HEP-Experiments**
-    2001-04+   production environment for BaBar, CDF, D0, Compass

## Regional Data and Computing Centre Germany, Requirements

**Regional Data and Computing Centre Germany, Requirements**

- For LHC (Alice, Atlas, CMS, LHCb) **+ BaBar, Compass, D0, CDF**

- 2001-2004:  Test Phase
- 2005-2007:  LHC Setup Phase
-        2007+:  Operation

- + services ....  + other sciences

- ▲
- starts in 2001 !

** ****Including resources for BaBar Tier-A**

## Organization of GridKa

** ****It’s a project with 41 user groups from 19 German institutions**

**  ****Project Leader & Deputy**
**    ****H. Marten, M. Kunze**
**  ****Overview Board**
**     ****controls execution & financing, arbitrates in case of conflicts**
**  ****Technical Advisory Board**
**    ****defines technical requirements**

## German users of GridKa

**German users of GridKa**
** ** 19 institutions
-   41 user groups
-   ~ 350 scientists

## The GridKa Installation

## Support for multiple experiments I

**  ****experiments ask for RedHat 6.2, 7.2, 7.1 or Fermi Linux, SuSE 7.2 ...**
**in different environments**
**  ****experiments ask for Grid (Globus, EDG,...), batch & interactive login**
**  ****split the CPUs into 8 parts ?**
**  ****would be administrative challenge**
**  ****likely, whole machine would be busy only part time**
**  ****reconfigure for each experiment ?**
**  ****non-LHC experiments produce and analyse data all the time**
**  ****who should define a time schedule ?**
**  ****what about other sciences ?**

## Support for multiple experiments II

- Strategy as starting point:
-   Compute Nodes = general purpose, shared resources
-     -  GridKa Technical Advisory Board agreed to RedHat 7.2
-   Experiment-specific software server
-     -  “arbitrary” development environments at the beginning
-     -  pure software servers and/or Globus gatekeepers later-on

## Experiment Specific Software Server

**8x dual PIII for Alice, Atlas, BaBar,..., each with**
-   2 GB ECC RAM,  4x 80 GB IDE-Raid5,  2x Gbit Ethernet
-   Linux & basic software on demand:
-     RH 6.2, 7.1.1, 7.2, Fermi-Linux, SuSE 7.2
**Used as**
-   Development environment per experiment
-   Interactive login & Globus gatekeeper per experiment
-   Basic admin (root) by FZK
-   Specific software installation by experiment admin

## Grid LAN Backbone

**Extreme Black Diamond 6808**
-   redundant power supply
-   redundant management board
-   128 Gbit/s back plane
-   max. 96 Gbit ports
-   currently 80 ports available

## Compute Nodes

**124x dual PIII, each with**
-   1 GHz or 1.26 GHz
-   1 GB ECC RAM
-   40 GB HDD IDE
-   100 Mbit Ethernet

**Total numbers:**
-   5 TB local disk
-   124 GByte RAM
-   Rpeak > 270 GFlops
-   RedHat 7.2
-   OpenPBS
-   automatic installation
-     with NPACI Rocks

## Cluster Installation, Monitoring & Management

-   heterogeneity: different (Intel-based?) hardware over time
-   consistency:  software must be consistent on all nodes
-   manpower:  administration by few persons only

**Cluster Installation, Monitoring & Management**

## Philosophies for Cluster Management

-   hierarchical instead of pure central management
-   combined push and pull for management information
-   info & event handling via separate management network
-   heterogeneity: rpm instead of disk cloning
-   consistency:  distribute software from a central service
-   manpower:  automatise as much as you can

**Philosophies for Cluster Management**

## Architecture for Scalable Cluster Administration

**Architecture for Scalable Cluster Administration**

- Master

- Manager C1

- Manager C2

- Manager Cn

**Management Network**

**Private Compute Network**

**Cabinet 1**

**Cabinet 2**

**Cabinet n**

**Naming scheme:  C02-001...064; F01-003,...**

## Installation - NPACI Rocks with FZK extensions

**Installation - NPACI Rocks with FZK extensions**

**reinstall all nodes in < 1 h**

**  ****DHCP-server for Managers C1...Cn**
**  ****RH kickstart incl. IP conf. for Managers**
**  ****rpm to Managers**

**  ****DHCP-server**
**  ****compute IP**
**  ****install nodes**

**http://rocks.npaci.edu**

## System Monitoring with Ganglia

**System Monitoring with Ganglia**

-   also installed on fileservers
-   CPU usage
-   Bytes I/O
-   Packets I/O
-   disk space
-  ...
-   and published on the Web

**http://ganglia.sourceforge.net**

## System Monitoring with Ganglia - Combined Push-Pull

- Ganglia
-  Master

- Manager C1

- Manager C2

- Manager Cn

**Management Network**

**Private Compute Network**

**System Monitoring with Ganglia - Combined Push-Pull**

**write into round robin DB**
**300 kB / node**

**  ****Ganglia daemon**
**    ****on each node**
**  ****info via multicast**
**  ****no routing**

## Cluster Management with Nagios - Combined Push-Pull

- Nagios
-  Master

- Manager C1

- Manager C2

- Manager Cn

**Management Network**

**Private Compute Network**

**Cluster Management with Nagios - Combined Push-Pull**

**  ****analyse data**
**  ****handle local events**

**syslog**

**  ****analyse data**
**  ****handle events**

**http://www.nagios.org**

**GPL**

## Combining Nagios with Tivoli Enterprise Console ?

- TEC

**SMS**
**Mail**
**inventory DB**
**Remedy workflow**

**Nagios**
**Master**

**GridKa Cluster**

## Infrastructure

**We all want to build a Linux cluster...**

**... do we have the cooling capacity?**

## Closed rack-based cooling system -

**A common development of FZK and Knürr**

- front

- rear

- internal
- heat exchanger

**CPU**
**disk**

**  ****19’’ technique**
**  ****38 units height usable**
**  ****70x 120 cm floor space**
**  ****10 kW cooling**
**  ****redundant DC fans**
**  ****temperature controlled**
**   ****CPU shut-down**

## Closed rack-based cooling system -

**Estimated cost reduction > 70% compared to air conditioning**

## Build a cluster of clusters (or a Campus Grid ?)

- bld. C

- bld. A

- bld. B

- LAN

- bld. D

## Online Storage

-   59 TB brutto
-   45 TB net capacity
-   ~ 500 disk drives
-   mixed IDE, SCSI, FC
-   ext3 & ufs file systems

**  ****DAS:****   ****	****2.6 TB brutto, 7.2k SCSI 120 GB, attached to SUN Enterprise 220 R**
**  ****SAN:****   ****	****2x 5.1 TB brutto, 10k FC 73,4 GB, IBM Fast500**
**  ****NAS:   ****	****42.2 TB brutto, 19x IDE-System, dual PIII 1.0/ 1.26 GHz,**
**    ****	****dual 3Ware Raid Controller, 16x 5.4k IDE 100/ 120/ 160 GB**
** ****SAN-IDE:****  3.8 TB brutto, 2 Systems, 12x 5.4k IDE 160 GB, driven by Linux-PC**

## Available Disk Space for HEP:  36 TByte net

## Scheme of (Disk) Storage

**Scheme of (Disk) Storage**

**Cluster Nodes (clients)**

**Grid backbone**

**Tape**
**Drives**

**MDC**

**Fileserver**
**IDE NAS 1.4 ... 2 TB/System**

## Disk Storage & Management – does it scale?

**  ****>600 automount operations per second for 150 processors**
**  ****measured IDE NAS throughput**
**  ****>150 MB/s local read  (2x RAID5 + RAID0)**
**  ****30-40 MB/s w/r .... over NFS**
**    ****... but  < 10 MB/s with multiple I/O and multiple users**
**  ****150 jobs write to a single NAS box**
**  ****Linux file system limit 2 TB**
**  ****disk volumes of  >50 TB with flexible volume management desirable**
**  ****mature system needed **_**now**_** !**

- We will test gfs & gpfs for Linux

## Available Tape Space for HEP:  106 TByte native

**Available Tape Space for HEP:  106 TByte native**

## FZK SAN

**IBM 3584**
-   ~ 2400 slots LTO Ultrium
-   100 GB/tape
-   106 TB native available
-   8 drives, 15 MByte/s each
-   Backup/Archive with Tivoli Storage Manager

**FZK SAN**

**FZK Tape 2**

**FZK Tape 1**

## Discussion of Tape Storage Management

**  ****HPSS, TSM, SAM-FS, gen. HSM ... do exist**
**  ****for vendor specific file systems**
**  ****on vendor specific disk systems**
**  ****for vendor specific tape systems**

- File systems, data management & mass storage are strongly coupled

## ?

**TSM**

**GridKa disk storage**

**- robotics**
** ****tape error handling**
**- tape recycling**

**backup**

**SAM**

**dCache**

**DataGrid**

**Castor**

- Grid

- Grid

## Summary I  -  The GridKa installation

-   **250 CPUs   **+ 130 PIV this week   + ~60 PIV until April 2003
-   **8 experiment specific server**
-   **35 TB net disk**   + 40 TB net until April 2003
-   **100 TB tape****   **+ 110 TB until April 2003 (or on demand)
-   **a few central servers**   Globus, batch, installation, management, ...
-   **WAN Gbit-test**   Karlsruhe-Cern
-   **FZK-Grid-CA**
**  ****	****	****       ****--****	****	****   ****testbed**
**...  exclusively for HEP and Grid Computing**

**Summary I  -  The GridKa installation**

## Summary II

**There is a whole bunch of questions....**
- Grid, security, gfs, gpfs, Castor, dCache, stability, reliability, HSM,
- certification, hundreds of users, commodity, low cost storage,
- OGSA, authorisation, networks, high throughput, ....

**.... and a pragmatic solution:**
-   start with existing technologies
-   analyse - learn - evolve

## The Federal Ministry of Education and Research, BMBF,

**The Federal Ministry of Education and Research, BMBF,**
** ****considers the construction of GridKa**
** ****and the German contribution to a World Wide Grid**
** ****as a national task.**