# Presentation: 527651

## dCache at PHENIX

Phenix Environment
dCache Benefits
Activities
HPSS Integration
Data Carousel Integration
Utilization Plans

- Phenix Environment
- dCache Benefits
- Activities
- HPSS Integration
- Data Carousel Integration
- Utilization Plans

- Scott O'Hare
- Dep't of Physics and Astronomy
- Stony Brook University
*631-344-6248*
*sohare@bnl.gov*

## ?

- General dCache Architecture

- PNFS filesystem

- Pool Manager

- PNFS Manager

- Doors
- (dCache protocol servers)

- Pools

- Client

## Current Testbed

- (via dCache “Spy” utility)

## Current Phenix Storage Environment 

- RHIC MSS = HPSS tape system
- HPSS accessed by multiple methods: Pftp, hsi, Data Carousel
- Local storage managed by users; replication and cleanup ad-hoc
- File Catalog used to locate files in local storage prior to accessing HPSS; validity checking on existing entries
- Data Carousel implements policy; updates Catalog after HPSS read

## HPSS is overloaded
Limits on pftp access enforced
Server Architecture and SFS database not scalable
Upgrades to disk cache, OS, server hardware
Batch optimization software improves read times by 3x
Run 4 data taking > 200 TB
Solaris servers manage about 75 GB Phenix disk
Substantial local storage on Linux farm

- Limits on pftp access enforced
- Server Architecture and SFS database not scalable
- Upgrades to disk cache, OS, server hardware
- Batch optimization software improves read times by 3x
- Run 4 data taking > 200 TB
- Solaris servers manage about 75 GB Phenix disk
- Substantial local storage on Linux farm

- Phenix Storage Environment (cont'd)

## Phenix Storage Resources

- Linux
- approx 355 machines, total storage 49.2 TB
- includes 60 new machines, total 26.3 TB
- Solaris
- 75.9 TB storage on 14 machines
- 27.5 TB new storage
- MSS (Mass Storage System)
- HPSS w/~10 TB internal Cache and 2 STK Powderhorn Tape Robots
- Capacity depends on #tapes dedicated to Phenix (effectively unlimited – performance depends on #drives)

## Potential dCache benefits

- Integrate disk space on multiple diverse machines into a unified storage facility
- Unify HPSS access mechanisms
- Improve file access performance
- Reduce load on HPSS servers
- recovery from failed disk/node
- Support gsi, K5 authentication
- Provide Grid access point with GridFTP, SRM implementations
- Replication and cleanup handled automatically

## Phenix dCache update 

- testbed upgraded to latest version (1.2.2-3); includes SRM (FNAL)
- RCF currently working SRM, Atlas dCache implementation; successes with SRM in initial testing, incl. Berkeley HRM
- Atlas dCache testbed to incorporate Phenix HPSS software
- RCF/Phenix receiving source for dCache, dCap, SRM (but not cells or pnfs). Successful compilations (& test run) of recent versions
- Phenix HPSS software supports GET, PUT operations for pftp, HSI. Perl script registers HPSS files with dCache
- Additional integration of dCache w/Carousel operations to provide migration path at Phenix
- Phenix production dCache system planned

## dCache distributions

- Original distribution was a “demo” package - dCache is not open-source
- 2nd “distribution” a grab-bag of assorted binaries thrown together for us. Difficult to set up, many functionality sysadmin unknowns
- Still lots to learn – ongoing assistance from Jon Bakken at FNAL, Patrick Fuhrmann at DESY
- First source package received April (thanks to Bruce Gibbard at RCF and Dave Morrison at Phenix). Not compiler-ready, but a valuable reference (managed to compile anyhow)
- We now have access to source for dcache, srm, and dcap. Not for PNFS or “cells” - legal issues . Also a “no modification” clause... Note: dCache SRM is Fermi code
- Latest releases compiling well (and run, too). Full binary   distribution w/ clear install instructions distributed at www.dcache.org. Frequent version upgrades.

## Assorted Activities

- Setups and Admin
- HPSS interface
- Dataflow and configuration analysis
- Doors
- dCache Kerb 5 authentication
- Solaris pools
- Pre-Load libraries
- source distributions, compilation
- SRM/HRM implementation & Testing
- production dCache system(s)
- . . .

## dCache

- Grid

- Remote User

- Pools

- Local User

- Access Modes

- SSH

- FW
- gateway

- Carousel,
- pftp, HSI

- RCF

## HPSS Interfacing

- HPSS storage looks like UNIX filesystem
- Convenient: PNFS also looks like UNIX fs
- Consistency issues must be managed
- HPSS Bitfile ID's give more rigorous file identification, but  no feasible API exists
- Access Methods:
- pftp: widely used
- HSI: full-feautured, uses Kerberos, command passing, error return
- Data Carousel: Incorporates optimization, priorites, syncs with File Catalog

## dCache HSM I/O

- Per-pool configurable mechanism calls a user-defined program/script when HSM I/O is required
- Arguments:
- GET or PUT command
- File identifiers (pnfs ID)
- Storage class info, including HSM ID
- Storage info derived from PNFS directory tags, also used to target pools for storage
- HSM metadata for file stored with pnfs ID database; metadata implements map to HSM storage location

## “PUT”

- “PUT”

- PNFS

- file

- POOL

- HPSS

- Pool
- Manager

- PUT

- HSM I/O
- script

- Pool Selection

- add HPSS storage info

- ACK

- Store in pool
- add “storage info”

## “GET”

- “GET”

- PNFS

- request file

- HPSS

- Pool
- Manager

- not
- cached

- HSM I/O
- script

- select pool

- cached

- GET

- ACK

- send file to user
- from cache

- stage file
- to pool

- HPSS storage
- info

## I/O script  notes

- HPSS storage metadata created by I/O script after successful PUT
- dCache storage info updated by dCache when file is stored in a pool (or replicated)
- I/O script “GET” is called only if requested file not cached
- GET retrieves HPSS storage metadata from PNFS file database entry using pnfsID key
- I/O script can choose from (a small number) of status codes to return to dCache, e.g. success, failure, suppress retries
- HPSS I/O can be enhanced to map dCache storage classes into HPSS CoS (class of service)

## Registration Utility for HPSS files

- Starting with a file in HPSS, a PNFS file entry is created and the corresponding pnfsID database entry loaded with the file's HPSS storage data
- “hp_register.pl”  script
- File behaves as though it was written to dCache, subsequently “flushed” to HPSS, and removed from pool.
- When “GET” is issued for file, it is seen to be uncached and is requested from HPSS.
- Requires a canonical mapping between a PNFS and HPSS directory
- “hp_register” issues “ls -lR” via to HPSS via HSI, creates corresponding file and subdirectory entries in PNFS

## dCache “Cleaner” Utility 

- dCache “Cleaner” Utility

- file is removed (rm) from the PNFS filesystem – disappears from PNFS view
- Cleaner (asynchronous process) locates all cached instances and deletes them
- File's PNFS ID and HPSS metadata are stored in $trash directory (API)
- “HPSS-cleaner” daemon to periodically scan $trash; if an entry is found, issues a file delete command to HPSS (removes $trash entry on success)

- PNFS

- “remove file”

- Cleaner

- HPSS
- Cleaner

- X

- X

- X

- trash

- “delete”

- HPSS

## Consistency Scenarios

- Restrictive:
- Dedicated HPSS directory for dCache exclusive access. Consistency is automatic.
- Permissive:
- Users can write into HPSS “dCache space”
- hp_register daemon provide HPSS view consistency to dCache
- User write permisson allows deletion of dCache files
- Fine-Grained:
- Write dCache-owned files to HPSS, with -rw--r--r permission.
- Use hp-register to provide dCache view of files added to HPSS by “non-dCache” means.
- (note: mods on the PNFS side not as critical)

## Using with ROOT

- Files can be created and read. All writes to a newly created file are transferred to dCache in a single burst when file is closed. Recent version permits overwrites.
- root [1] gSystem->Load(“libDCache.so”)
- root (2) TDCacheFile *tdc1 = 					TDCacheFile::Open("dcache:/pnfs/rcf.bnl.gov/users/sohare/td1.root")
- If the plug-in is used, TFile can be used normally:  prepending “dcache:” triggers use of dCap I/O
- root[7] gROOT->GetPluginManager()->AddHandler("TFile", "^dcache:", 			"TDCacheFile", "DCache", "TDCacheFile(const char *)") ;
- root(8) TFile *p_file = TFile::Open("dcache:/pnfs/rcf.bnl.gov/users/sohare/.. 		...hsm1/wrd0a.txt”)
- Newer version of ROOT may be required (4.0+)

## dCache Preload library

- dCache POSIX file I/O API, available for client app. programming
- setenv $LD_PRELOAD <path>/libpdcap.so
- Interpose dCap file I/O calls for standard UNIX where appropriate (i.e. accessing a dCache file)
- if PNFS mounted (e.g. remotely):
- cp /pnfs/rcf.bnl.gov/users/scott/testfile  ./testfile
- if PNFS unmounted, use dCap URL
- cp  dcap://rplay21:25125/pnfs/rcf.bnl.gov/users/scott/testfile \ ./testfile
- Recent versions work well with dCache operations, basic UNIX file I/O (cp, cat, ...), but cause very strange behavior with higher level programs (xterm, vi, emacs, ...)
- Will continue to monitor usability of new versions...

## Integrating Local Storage 

- Integrating large numbers of relatively small pools on Phenix/RCF Linux farms – a key dCache scenario
- Fermi CDF project running ~200 pools, but these are dCache-dedicated
- Extensive PNFS mounting on pool & client hosts
- Coexistence w/other Linux farm software
- “Resilient dCache” implementation
- dCache memory footprint
- CPU sharing
- NFS Compatibility an issue

## PHENIX dCache Plans

- Probably use “a few large pools” (Solaris) at outset – initial pool allocation approx. 5TB total (?)
- Est. 2-3 months to start, dependent upon RCF scheduling
- “Populate” dCache by registering PRDFs and DSTs
- Operate in “fine-grained” consistency mode (rw-r--r--)
- registration and cleaning daemons
- dCache to implement 4 levels of (Phenix) CoS
- High-priority read-only and write-only pools for Phenix production, low-priority RW pool(s) for general use
- Use with SRM, gridFTP doors in grid (and local access) environment; FW conduits for remote access

## HPSS/dCache Integration Software

- HPSS I/O script (“hpsscp.sh”) supports GET, PUT commands, access via pftp, HSI
- hpss-Cleaner process to forward dCache file removals to HPSS*
- hpss-register.pl script creates mapped dCache view of any HPSS directory tree w/o pre-staging,
- “daemonization” of hpss-register.pl could regularly scan hpss directories and create new pnfs (dCache) file entries, perform other consistency functions*
- hpsscp.sh mods (& dCache configs) to implement HPPS CoS (Class of service)*

## dCache/Carousel Integration Software 

- Enable Data Carousel as HPSS access mechanism for HPSS I/O script, automatically pre-searches File Catalog
- HPSS I/O script GET (non-Carousel) to search File Catalog, identify other local file locations before retrieving from HPSS
- PUT and GET command could register newly cached files with File Catalog. File Catalog verification would eliminate “uncached” file entries
- Give Carousel ability to read and write HPSS files

## ...continued testing

- notification mechanism for file retrievals;
- accurate state checking with Data Carousel
- Performance testing (multiple scenarios)
- Reliabilty testing (how easy to break?)
- read-only and write-only pools, “pool-to-pool protocol”
- option to replace pnfs native gdb w/ postgres
- . . .