# Presentation: 051274

## SRM: Expt Reqts

- Nick Brook

- Revisit LCG baseline services working group
- Priorities & timescales
- Use case (from LHCb)

## Baseline Services report

- The WG has agreed a common “LCG-SRM” set of functions, that the experiments need:
- LCG Service Challenge 3: v1.1
- LCG Service Challenge 4: LCG-SRM
- LCG SRM functionality:
- V1.1 + space management, pin/unpin, etc
- Not full set of V2.1
- V3 not required

- _Baseline Services report_

## Basic SRM functions

- (see link from baseline services group web pages - http://cern.ch/lcg/PEB/BS)

- File types:
- Volatile - temporary & sharable copy of a MSS resident file - if not pinned can be removed by garbage collector
- Durable - file cannot be removed automatically. If space needed file may be copied to MSS ...
- Permanent - system cannot remove file
- Expts only require - volatile & permanent file types

## Basic SRM functions

- Space reservation:
- SRM v1.1: space reservation done on file-by-file basis
- User doesn’t know in advance if SE will be able to store all files in request
- SRM v2.1: allows for a user to reserve space
- Reservation has a lifetime
- Data “PrepareToGet(Put)” requests fail if not enough space
- SRM v3.0: allows for “streaming”
- When space is exhausted new requests don’t fail but wait until space is released
- Expt happy with v2.1 space reservation functionality

## Basic SRM functions

- Permission functions:
- SRM v2.1: allows for a posix-like ACLs
- Can be associated with each directory or file
- Expts desire storage system to respect permissions based on VOMS roles & groups
- Expt have NO wish for file ownership by individual users

## Basic SRM functions

- Directory functions:
- Create/remove directories
- Delete files
- Rename directories or files (on a particular SE)
- Directory listing (not necessarily recursive listing)
- No need for “mv” (between SRM SE’s)

## Basic SRM functions

- Data transfer functions (misnomer - not actual data movement but to prepare access to data):
- stageIn, stageOut type functionality
- Pinning & unpinning functionality
- Request token to monitor status of request
- How many files ready
- How many files in progress
- How many files left to process
- Suspend/re-start/abort request

## Basic SRM functions

- Relative paths:
-  Relative paths in SURLS (with respect to a base VO home)
- srm://castorsrm.cern.ch/castor/cern.ch/grid/lhcb/DC04/prod0705/0705_123.dst
- Define SE
- Site definition for VO
- VO definition

## Basic SRM functions

- Query the protocols supported by site/SE:
- Function already in information system
- List of protocols supported by VO can be given in the application to SRM - return TURL with protocol applicable to site

## Prioritised List discussed at LCG SC3 workshop

- In descending order:
- 1.Pin/Unpin functionality
- 2.Relative paths in SURLs
- 3.Permission functions: All experiments would like the permissions to be based on roles & DN’s. SRM should be integrated with VOMS
- 4.Directory functions (with the exception of “mv”)
- 5.Global space reservation: ReserveSpace, ReleaseSpace and UpdateSpace, though CompactSpace is not needed
- 6.srmGetProtocols is seen as useful but not mandatory
- 7.AbortRequest, SuspendRequest and resumeRequest not seen essential
- First five seen as essential

## LCG Project Execution Board 

- 28th June 2005 meeting:
- Follows on from SC3 workshop in June
- Directory functions – OK
- Permission functions – OK
- Pin/Unpin – OK.
- i.e. assuming adequate resources / priority, could be implemented in all relevant SRMs on schedule for SC4 (delivery < end January 2006)
- Relative paths in SURLS:
- request is for something like $VO_HOME, it too can be provided in time for SC4
- Global space reservation:
- Requires more discussion with the developers. Unlikely to be delivered in time for SC4 but could perhaps be available mid-2006(?)

## LCG Timescales

- SC4 (Feb’2006) - “to have available all missing tools & existing components”
- Necessary to expose service well before start of SC4

## Step-by-step through stripping use case 

## Use case

- Stripping:
- centralised analysis
- Reduced reconstructed dataset about factor of ~10
- Performed 4 times a year: twice with recons & two other times
- Need to retrieve RAW & rDST data from MSS
- Output files distributed to all Tier-1 centres

## Need the reserve space management & reservation functionality - necessary to ensure the stripped DSTs have storage at LHCb Tier-1’s. Likely to be a slight overestimate &/or usage of “space update”

- 2.Production manager launches production jobs
- 3.Production job via SRM checks the VO specific namespace (without the need to know the site specific high level details) if the necessary directory structure exists at the production Tier-1 and the other Tier-1’s for the output files. If not will create the necessary directory hierarchy everywhere
- 4.Check to see if output file already exists at any Tier-1. If so exits with warning message to production manager. If file/directory exists at only 1 site it may be necessary for the production manager to delete/copy the offending file/directory elsewhere.
- 5.Job issues stage request via SRM for all needed input files
- 6.As files become available they are pinned by SRM with a validity time compatible with the expected duration of the job.
- 7.Job processes files as they become available, once processed the file will be unpinned
- 8.Once all (available) input files are processed the output file(s) are made permanent
- 9.Job checks the permission on the file to ensure against accidental deletion but to allow read permission for the entire VO
- 10.Once the production is finished the production manager releases reserved space at Tier-1’s.

## Need basic directory functionality - necessary to create the hierarchical structure to receive data & check files don’t exist etc

- 2.Production manager launches production jobs
- 3.Production job via SRM checks the VO specific namespace if the necessary directory structure exists at the production Tier-1 and the other Tier-1’s for the output files. If not will create the necessary directory hierarchy everywhere
- 4.Check to see if output file already exists at any Tier-1. If so exits with warning message to production manager. If file/directory exists at only 1 site it may be necessary for the production manager to delete/copy the offending file/directory elsewhere.
- 5.Job issues stage request via SRM for all needed input files
- 6.As files become available they are pinned by SRM with a validity time compatible with the expected duration of the job.
- 7.Job processes files as they become available, once processed the file will be unpinned
- 8.Once all (available) input files are processed the output file(s) are made permanent
- 9.Job checks the permission on the file to ensure against accidental deletion but to allow read permission for the entire VO
- 10.Once the production is finished the production manager releases reserved space at Tier-1’s.

## Current SRM v1.1 functionality - ability to optimise use of MSS system

- 2.Production manager launches production jobs
- 3.Production job via SRM checks the VO specific namespace if the necessary directory structure exists at the production Tier-1 and the other Tier-1’s for the output files. If not will create the necessary directory hierarchy everywhere
- 4.Check to see if output file already exists at any Tier-1. If so exits with warning message to production manager. If file/directory exists at only 1 site it may be necessary for the production manager to delete/copy the offending file/directory elsewhere.
- 5.Job issues stage request via SRM for all needed input files
- 6.As files become available they are pinned by SRM with a validity time compatible with the expected duration of the job.
- 7.Job processes files as they become available, once processed the file will be unpinned
- 8.Once all (available) input files are processed the output file(s) are made permanent
- 9.Job checks the permission on the file to ensure against accidental deletion but to allow read permission for the entire VO
- 10.Once the production is finished the production manager releases reserved space at Tier-1’s.

## Many jobs will be running in parallel - important for them not to interfere with each other. Essential to pin file once staged to ensure its availability (& unpin, of course!)

- 2.Production manager launches production jobs
- 3.Production job via SRM checks the VO specific namespace if the necessary directory structure exists at the production Tier-1 and the other Tier-1’s for the output files. If not will create the necessary directory hierarchy everywhere
- 4.Check to see if output file already exists at any Tier-1. If so exits with warning message to production manager. If file/directory exists at only 1 site it may be necessary for the production manager to delete/copy the offending file/directory elsewhere.
- 5.Job issues stage request via SRM for all needed input files
- 6.As files become available they are pinned by SRM with a validity time compatible with the expected duration of the job.
- 7.Job processes files as they become available, once processed the file will be unpinned
- 8.Once all (available) input files are processed the output file(s) are made permanent
- 9.Job checks the permission on the file to ensure against accidental deletion but to allow read permission for the entire VO
- 10.Once the production is finished the production manager releases reserved space at Tier-1’s.

## Current SRM v1.1 functionality - ability to store files in MSS system. Necessary before a reserved space is released!

- 2.Production manager launches production jobs
- 3.Production job via SRM checks the VO specific namespace if the necessary directory structure exists at the production Tier-1 and the other Tier-1’s for the output files. If not will create the necessary directory hierarchy everywhere
- 4.Check to see if output file already exists at any Tier-1. If so exits with warning message to production manager. If file/directory exists at only 1 site it may be necessary for the production manager to delete/copy the offending file/directory elsewhere.
- 5.Job issues stage request via SRM for all needed input files
- 6.As files become available they are pinned by SRM with a validity time compatible with the expected duration of the job.
- 7.Job processes files as they become available, once processed the file will be unpinned
- 8.Once all (available) input files are processed the output file(s) are made permanent
- 9.Job checks the permission on the file to ensure against accidental deletion but to allow read permission for the entire VO
- 10.Once the production is finished the production manager releases reserved space at Tier-1’s.

## Permissions functions - essential to ensure file is readable by whole VO and only production manager has write access. Specialised stripping would need to set group/sub-group privileges

- 2.Production manager launches production jobs
- 3.Production job via SRM checks the VO specific namespace if the necessary directory structure exists at the production Tier-1 and the other Tier-1’s for the output files. If not will create the necessary directory hierarchy everywhere
- 4.Check to see if output file already exists at any Tier-1. If so exits with warning message to production manager. If file/directory exists at only 1 site it may be necessary for the production manager to delete/copy the offending file/directory elsewhere.
- 5.Job issues stage request via SRM for all needed input files
- 6.As files become available they are pinned by SRM with a validity time compatible with the expected duration of the job.
- 7.Job processes files as they become available, once processed the file will be unpinned
- 8.Once all (available) input files are processed the output file(s) are made permanent
- 9.Job checks the permission on the file to ensure against accidental deletion but to allow read permission for the entire VO
- 10.Once the production is finished the production manager releases reserved space at Tier-1’s.

## Again use of space management functions - needed for releasing the space used for the output of stripping. 

- 2.Production manager launches production jobs
- 3.Production job via SRM checks the VO specific namespace if the necessary directory structure exists at the production Tier-1 and the other Tier-1’s for the output files. If not will create the necessary directory hierarchy everywhere
- 4.Check to see if output file already exists at any Tier-1. If so exits with warning message to production manager. If file/directory exists at only 1 site it may be necessary for the production manager to delete/copy the offending file/directory elsewhere.
- 5.Job issues stage request via SRM for all needed input files
- 6.As files become available they are pinned by SRM with a validity time compatible with the expected duration of the job.
- 7.Job processes files as they become available, once processed the file will be unpinned
- 8.Once all (available) input files are processed the output file(s) are made permanent
- 9.Job checks the permission on the file to ensure against accidental deletion but to allow read permission for the entire VO
- 10.Once the production is finished the production manager releases reserved space at Tier-1’s.

- Again use of space management functions - needed for releasing the space used for the output of stripping.

## }

- All expts see SRM as essential
- Fundamental building block of a datagrid
- LCG set of functionality - agreed by all expts
- Optimisation of MSS essential
- Communicate multiple requests
- Handle priorities
- Optimise tape access
- Expose to expt before SC4
- Timescales tight

- Storage system manager reqts but recognised by expts