# Presentation: 810413

## Data Movement and Storage for Run II

- Eileen Berman
- Rob Kennedy
- Alexander Moibenko
- Gene Oleynik

## Mass Storage at Fermilab

- Provides rate-adapting distributed disk cache backed by (tape) mass storage
- Provide and maintain sufficient archival quality tape backed storage for experiments with high throughput access to reconstruct and analyze in real-time
- Development
- Substantial FNAL components
- DESY, LBL(SRM), GGF(GridFTP)

## Current Status

**Enstore**** – **Tape based Data Storage System
- Greater than 1.8 PB total data
- Between 10 TB  &  20 TB per day data movement
- Volumes
- 9940A > 6000
- 9940B ~ 9000
- LTO-1 > 2000
- LTO-2 > 350

- 3 separate systems (CDF, D0, General Purpose)
- 74 drives for Run II
- 88 nodes for Run II

## Current Status

**dCache**** – **disk cache
- Installed at FNAL: CDF, CMS, D0, LQCD, STK
- Stabilized at highest load in May 2004
- LAN interface – dcap
- Linux, OSF1, IRIX, SunOS
- WAN/Grid interface -
- Ftp, GridFTP
- SRM
- Grid storage element

## Current Status

**dCache** -
**CDF**
- 65 nodes with 3 read pools each (150 TB)
- 3 nodes with 3 write pools each (5 TB)
- 6 selection groups
**D0**
- 2 nodes with 2 read pools each (1.8 TB)
- 2 nodes with 2 write pools each (1.8 TB)

## Storage Resource Manager

- Uniform interface to storage systems on the Grid – abstracts basic file handling
- “SRM” at FNAL means:
- interface specification itself
- implementation w/dCache

## Projects

- Federation
- Dispersal
- Or multiple copies
- Automigration
- Security
- Lifetime work
- Tapeless Data Path
- Broaden admin base
- Resilient dCache
- SRM
- Administration Improvements

## Dispersal

**Goal**
- Reduce the risk of total data loss due to the data being in one physical location

- Proposal
- Disperse, not Duplicate the data, 1⁄2 in the new High Density Computing Facility
- Design shall allow for duplication across both sites as well
-  We are currently revisiting this and considering duplication as an alternative to dispersal

## Federation

- 3 instances of Enstore systems
***Goal***
- Increased administrative efficiency, given the evident evolution of these systems.
- Retain and enhance scalability
- Allow using of tapes drives across systems
- No reduction of performance during transition
- Technical details are being defined

## Automigration

**Goal**
- Conserve library slots, Data Center foot print
- Retire old technology
- Reuse media where possible
- Preserve data layout where possible
- Migration of tapes
- Greater than 4000 CDF 9940A tapes were migrated to 9940B tapes.
- Migration of predefined tapes in the background
- No interference with main operations
- Provide access to data on new tapes after the data has been migrated

## Security

- Write-Protect tabbing
- One copy permanent store
- Threats: Software error, or malicious employee
- Ongoing activity
- Security risk analysis
- Based on NIST model
- At request of FNAL Computer Security Staff
- Dispersal/Duplication – already discussed

## Lifetime Work

**Goal**
- System support and improvement to sustain performance in the face of change

- Specifics
- Maintaining Flexible Media posture – LTO-2
- Accounting databases
- Converted file and volume db to POSTGRES
- Operational monitoring

- Scaling operations – ~80 drives in service
- Consideration of new robot technology

## dCache Projects

- Tapeless Data Path
- Eliminates tape latency access to data
- Reduces access to tapes
- Broaden Admin Base
- Improve documentation
- Improve administration procedures
- Improve installation procedures
- Resilient dCache
- No tape backing - data replicated
- Suitable for an analysis environment
- Testing at CDF to use with CPU farms

## dCache Projects

- Next Gen Monitoring
- Queriable interface
- More flexibility in presentation
- Improved error tracking, especially by clients
- Investigations
- Alternative storage systems: Lustre, PVFS2, ...
- OSG and LCG issues, interoperability, ...

## SRM Projects

- FNAL “dCache” SRM
- parameterize for re-use with other storage systems
- Apply to Run II via SAM
- simplify use of various storage systems by adapting SAM to the use of SRM interface

## Administration Projects

- Improve uniformity of hardware and software loads (e.g. Rocks appliance)
- Replacement of aging PC's
- Approximately 30 mover PC's from 1999
- Improvements in problem and work flow tracking
- Evaluation of new technologies (drives, tape libraries)

## Administration

- Maintain data integrity
- Monitor system for write/read errors (random and systematic)
- Maintain fresh copies of data (copy overused tapes)
- Maintain operational stability
- Monitor 3 distinct Enstore systems and 4 dCache systems
- Investigate tape faults -> 10/week
- Increasingly diverse plant
- Manage and plan storage and access
- Keep adequate blanks for writing
- Programmatically compress data onto denser media
- Plan and implement for evolving storage  and access needs

## Resources

- People Available
- 3 FTE Enstore developers
- 3.5 FTE dCache/SRM developers
- 5 Administrators
- Cost
- ~ $225k/yr maintenance for automated tape libraries (ATL)and tape drives ( 24x7)
- $500k/yr for storage media
- $50k to replace aging components (PCs)
- Reference Costs
- $10k or $30k (depending on tape library) for a tape drive
- $75k/PB current cost of refurbished tape library
- $4k for Enstore PC and ~$1k/TB for dCache file server

## PNFS

- Underlying catalog service of Enstore and dCache
- Performance limited by its underlying RDBMS
- dCache's PNFS interface a bottleneck at high load
- Postgresql RDBMS – improved performance, admin tools
- Scaling issues - PNFS servers
- Investigating more performant dCache/PNFS options

## ATL Space

- At  rate, CDF to run out of storage in 1 yr, D0 in 3yr
- After shutdown CDF will increase rate
- Next 6 months need to increase capacity for CDF
- Compress tapes to new denser technology (  > $300K media)
- New ATL acquisition ( > $200k robotic library)

- Space and timing issues are the challenge
- Over next year strategic plan for future storage needs

## NFS v2

- Protocol is old
- Contains 32-bit file kludges
- No ACL support
- Alternatives need to be investigated

## Summary

- Enstore and dCache are used successfully in production by Run II experiments
- Mass storage functionality and throughput has met the needs of Run II
- Successful scaling projects have kept pace with changing requirements
- CDF tape migration
- Throughput enhancements via drive acquisitions
- Provision of Grid storage element functionality
- Current projects are preparing for future rate increase and necessary system expansion and enhancements