# Presentation: 830539

## Managing managed storage CERN Disk Server operations

- HEPiX 2004 / BNL
- Data Services team:
- Vladimír Bahyl, Hugo Caçote, Charles Curran, Jan van Eldik, David Hughes, Gordon Lee, Tony Osborne, Tim Smith

## Outline

- Which are our “Data Services”?
- Disk server hardware @ CERN
- Management tools
- What’s next?

- 2004/05/26

## A lot of hardware

- Disk storage
- 350 “storage in a box” Linux diskservers
- 6700 disks
- 550 TeraBytes of raw disk space
- Tape storage
- 2 robotic installations	each with 5 STK 9310 silos
- 50 9940B drives, 14000 tapes, 2.8 PB
- 20 9840 drives, 8000 tapes, 160 TB

- 2004/05/26

## Many applications

**200 CASTOR!**
- 40 Oracle
- 20 CDR
- 10 AFS scratch
- dCache, _[LHC@home](http://lhcathome.cern.ch/)_, ...
- LCG, OpenLab, EGEE, data challenges
- 40 in repair/spare

*A very heterogeneous environment!*
*And very dynamic too ***

- 2004/05/26

## Players

- Many teams involved:
- Application responsibles / Users
- Service managers
- System administrators team
- Suppliers
- Software often not redundant...	**need to minimize downtime!**
- ...so the hardware should be!

- 2004/05/26

## “Storage in a box”

- 13 different hardware configurations:
- 8 – 26 IDE disks, hot-swappable trays
- 2 – 4 3-Ware RAID controllers
- 2 CPUs
- 2 – 3 power supplies
- GigE network card
*Should be redundant...*

- 2004/05/26

## hardware interventions

- 55 interventions since Sep 1
- disk replacements (70%)
- trays, cables, fans, PSU
- 33% involve (un)scheduled downtime 
- Older hardware harder to maintain
- One supplier out of business
- Incidents to spice up life...

- 2004/05/26

## Disk replacement

- 10 months before case agreed: Head instabilities
- 4 weeks to execute
- 1224 disks exchanged (=18%); And the cages as well

**Christmas**

- 2004/05/26

## 65 Jumbo’s

- 1 – 1.5 TB raw disk space
- 6800 3-Ware controllers
- 600 MHz PIII
- No PXE
- Becoming hard to maintain
- Many still under warranty 
- Make good mini-bars!

- 2004/05/26

## 175 4U servers

- 4U (5U) rack mounted
- 1 – 1.5 TB
- 2 * 3-Ware 7000 series
- 	currently upgrading firmware
- 2 * 1 GHz PIII’s
- No PXE (yet)
- Various maintenance issues

- 2004/05/26

## 115 8U servers

- 8U rack mounted
- 2 – 2.5 Tb
- 3 – 4 * 3-Ware 7500(6)-8
- 2 * 2.4 GHz Xeon
- Well controlled, well maintained, well behaved, after disk replacements 

- 2004/05/26

## Diskserver evolution

- 2004/05/26

## That was then...

- HW RAID1
- Ext2 filesystems *many of them*
- 13 different kernels! *RedHat 6.1/6.2, 7.2/7.3, 2.1ES*
_**Need for automation + standardization**_
- 	ELFms toolsuite
- 	Quattor – installation + configuration
- 	LEMON – performance + exception monitoring
- 	LEAF – Hardware and State Management

- 2004/05/26

## ...this is now

- RedHat 7.3, preparing for SLC3
- Oracle: RHEL 2.1, preparing RHEL 3	*kernel has old 3-Ware driver  ***
- HW RAID5 + hot spare disk
- Up to 50% more usable space
- On 3-Ware 7000 controller *with up-to-date firmware*
- SW RAID0 + XFS
- Improved performance expected *iozone benchmark*
- Old XFS version
- Improved kernel / elevator tuning

- 2004/05/26

## Updating the toolbox

- SMART – to predict disk failuredaily and weekly self-tests, on every disk
- IPMI v1.5
- HW monitoring and event control
- Power control, resets
- Lm_sensors – temperature monitoring
*Hardware and software specific*
*All data flows into Lemon repository*

- 2004/05/26

## Wintertime?

- 2004/05/26

## This is now

- Quattorized + Lemonized
- Rely on Operator and SysAdmin teams
- Operated in same way as PC farms
- Getting more out of suppliersBIOS upgrade necessary for PXE enabling
*BTW: most applies to tapeservers as well*

- 2004/05/26

## What’s next?

- New hardware
- 360 TB “SATA in a box”, 2 different suppliers
- 140 TB FC attached external SATA disk arrays
- New software
- SLC3, RHEL 3
- New CASTOR stager
- New challenges
- Oracle SAN setup
- Alice data challenge

- 2004/05/26

## Conclusions

- A lot of work has been done to
- Stabilize Hardware and Software
- Automate + hand over basic operations
- Integrate into standard work flows
- Get more out of available hardware
**Achieved *****pro*****-active data management**

- 2004/05/26

## Useful links

*“**Standing on the shoulders of giants”*
- 	Tim Smith _[CHEP 2004](http://indico.cern.ch/contributionDisplay.py?contribId=374&sessionId=10&confId=0)_
- 		http://indico.cern.ch/contributionDisplay.py?contribId=374&sessionId=10&confId=0
- 	Helge Meinhard _[CHEP 2004](http://indico.cern.ch/contributionDisplay.py?contribId=325&sessionId=10&confId=0)_
- 		http://indico.cern.ch/contributionDisplay.py?contribId=325&sessionId=10&confId=0
- 	Peter Kelemen _[CERN IT “After C5”](http://cern.ch/Peter.Kelemen/talk/2004/C5/diskserver)_
- 		http://cern.ch/Peter.Kelemen/talk/2004/C5/diskserver
- 	Jan Iven _[HEPiX](http://hepwww.rl.ac.uk/hepix/nesc/iven.pdf)__[ 2004 Edinburgh](http://hepwww.rl.ac.uk/hepix/nesc/iven.pdf)_
- 		http://hepwww.rl.ac.uk/hepix/nesc/iven.pdf

- 2004/05/26