# Presentation: 266966

## Upper Levels of Data Movement

- SDSS Automation
- KTeV Migration and Automation
- dCache, FNAL + DESY collaborative effort
- Installations, Issues
- Resilient dCache
- SRM
- Initiatives and support program

## SDSS Automation

- Goal is to automate SDSS DLT tape mounts in the AML/2 robot.
- SDSS tapes are written outside enstore framework and in ‘foreign’ format, but metadata is available.
- SDSSCP product – reads SDSS tapes and transfers files to SDSS computers for analysis
- Final phases of testing have revealed several unexpected error paths that are being reworked
- Hardware ordered to support effort
- Procedures being developed to accept APO tapes and get them into AML/2 and known to enstore. Relabeling with proper barcode for current tapes.

## KTeV Migration and Automation

- Goal is to automate KTeV DLT tape analysis
- KTeV has 30-50 DLT drives on Kpasa... and they sometimes require ~30 independent parallel streams of data for their analysis
- Not cost effective to do this in AML/2 – must migrate to current media (9940B)
- Daft/Ftt record format – depends on byte count from drive.  This count is used to reformat KTeV tape data blocks.   KTeV depends on this format.
- We’ve written a migration script that reformats DLT tapes into the daft KTeV format and have migrated ~100 DLT tapes (out of ~5000) to 9940B

## KTeV, cont

- Plan is to build ~20 TB dCache system and have KTeV read/write data from dCache instead of their DLTs – only way to have ~30 streams active
- A KTeV dCache interface has been written and initially tested. Migrated tapes have been successfully read and produce identical results.
- CD-KTeV meeting required to finalize effort.  A lot of hardware is needed for project.

## dCache

- Installations on site:
- CDF – in very heavy use
- Biggest deployed dCache, largest bandwidth
- State of the art deployment
- Almost entirely read based
- CDF has plans for writing data to dCache to avoid tape bottlenecks (tapeless data path)
- D0 – evaluating system
- Primarily interested in writing

## dCache Installation, con’t

- CMS
- 2 dCaches installed at FNAL –
- Main production dCache coupled to public enstore and tape system - managed by CCF
- Another production dCache on their hotdog nodes run by the CD CMS group.  Not coupled to enstore.
- Model for Tier2 sites.

## dCache Installation, con’t

- LQCD
- Being tested and optimized for use by their worker nodes
- Moderate size
- 2 pool types – resilient & standard tape backend
- Main advantages of dCache as seen by LQCD:
- uniform file space across many disks
- Data throttling – recall that their clients are synchronized  and want simultaneous access to same data files

## dCache Installation, con’t

- Public system
- This is a small system, very inadequate to service its many clients. Requests can queue for days.
- Main users are minos, auger, cdms, theory, patriot.  KTeV migration will use the pubic system, EAG (Annis) expressed interest.
- Large offsite usage – ftp is primary protocol

## dCache Issues

- Small dCaches work well – larger ones require a lot of detailed knowledge and tuning to get them running efficiently.
- This means the underlying algorithms are still being understood.
- Many improvements over past year
- Cost – a single value that determines how/where transfers are performed – needs to become much more dynamic to handle changing client access patterns and loads

## dCache Issues, cont

- Few remaining problems with dCache connection with the storage system – timeouts present and when they expire, they always cause problems.  Retries where appropriate.
- Threaded applications are difficult to debug and monitor, especially across many nodes. Working on improving diagnostics and trouble shooting
- Difficult to recover from jvm out-of-memory in main threads. Splitting  where appropriate.
- Data integrity – improvements needed in speed & verification of checksums – bottleneck right now

## dCache Issues, cont

- Pool-to-pool copy – linked to cost problem mentioned previously. Some logic flaws are still present and they sometimes degrade rather than improve performance
- Prestager – load modeled in threads – needs to be rewritten
- Space reservation and file pinning
- Current concepts are being extended to form a more coherent service
- Goal is to make sure all SE (storage element) properties are fulfilled so dCache can be used interchangeably with other SE components.

## dCache Issues, cont

- FTP – clients see failures due to client side timeouts that expire because files not in cache.
- Firewall problems in wan transfers – limit port range
- FTP stream limits – up to 10/client
- Gridftp – new features being investigated
- For example, checksum, scaling features

## dCache Issues, cont

- Authentication/Authorization
- dccp –  Kerberos available today, providing gsi
- dCache “gridmap file” (dCache.kpwd) – works
- Should be based on VO, not independent file
- Some remote users want actual gridmap file to be used instead – providing this with site defaults
- PNFS – need to extend underlying database beyond present 2 GB limit

## Resilient dCache

- Formerly known as scratch dCache
- Goal is managed reliable storage without a tape backend (initial tests are with farm scratch space)
- Reliability is achieved through replication
- Expect pools to go in/out of service and files are replicated as this happens. Can also schedule pools offline to smooth replication process
- Working on active replica checksum comparison with replacement when necessary
- Working on user web interface

## Resilient dCache

- Can be installed by local site admin
-  tarball for dCache+pnfs
- Also need java, postgres, tomcat rpm
- Given to CDF, being deployed
- We know there are still problems to be solved with this project.

## SRM

- Goal is to provide middleware that manages site resources for the grid and provides uniform access for the many heterogeneous storage components
- Underlying transfer protocol is user definable and negotiable for transfers between sites
- Fully integrated into dCache storage element
- Working on protocol definition v2.1
- Major feature is space reservation and pinning
- MSSMSS file transfer features
- Allows CMS, for example, to specify dataset (1000’s of files) and SRM will reliably handle all file transfers between CERN/FNAL/Tier2 centers
- Will be used for lighting up network pipes for tests

## SRM Development work

- CDF SAM-dCache integration via SRM
- Addition of support for Volatile and Duarble file types based on resilient disk storage (BTeV)
- LCG SE requirements (auto creation of directories, other directory function, free space available...)
- Parameterization of time estimates and development of standard metrics
- Dynamic reservation of network resources (WAWG)
- Development of SRM v3.0 protocol
- Access permission management

## Initiatives/Support

- CDF offsite support  for Resilient dCache
- Agreement in principle reached – MOU being drafted for CDO consideration – based on LCG model
- Software installed by expert at site
- Expert at site contacts CDF point person for problems
- Only CDF point person can contact us for help

## Initiatives/Support, cont

- Brookhaven
- 2 (independent?) serious inquiries from BNL to create a full dCache system including authentication and tape storage
- Interested in CDF model of many pools and subcaches
- Plan to start helping with BNL setup later this week
- The hope is for new developer effort
- BNL will write HPSS (ftp) interface to dCache

## Initiatives/Support, cont

- Storcloud – DOE initiative (demo at SC2004)
- The goal is to build a virtual “storage on request” capability to support researchers and demonstrate high bandwidth applications
- Our goal is compatibility with this venture
- Initial contact has been made – waiting for real discussions to start.

## Initiatives/Support, cont

- BTeV
- Began initial discussions about robust and reliable disk based permanent storage
- 1 complete set at FNAL, 2 other complete sets distributed among 5 major universities in US
- Extension of resilient dCache concepts
- Multiple copies, active checking with duplicating and reduction to maintain robust store across BTeV
- Active file verification and correction

## Initiatives/Support, cont

- Vanderbilt – interested in installing dCache and Enstore for the university. Waiting for decision on their evaluation.
- BTeV ITR proposal, “Tools for Widely Distributed Permanent Disk Storage and Dynamic Reallocation of Grid Compute Resources” – Petravick (CoPI)  and Bakken on LOI