# Presentation: 266958

**C    O    M    P    U    T    A    T    I    O    N    A    L        R    E    S    E    A    R    C    H        D    I    V    I    S    I    O    N**

**SRM Basic Spec****LBNL Proposal**

- Arie Shoshani, Alex Sim, Junmin Gu, Kurt StockingerScientific Data Management GroupLawrence Berkeley National Laboratory

## Summary

**LBNL proposal of SRM Basic specification based on v.1.1**
**Original v.1.1 methods/arguments in BLUE****Proposed modifications in **
**Green for essential additions**** ****with + to the v.1.1 **
**Skyblue for optional additions with ++ to the v.1.1 **
**Red for subtractions from the v.1.1**
**Purple for renaming and implied updates of the v.1.1 ****with =>**
**Notations:**{}   ... tuple()    ... list (ordered collection)[]    ... set (unordered collection)_     ... underline as mandatory (as opposed to optional)::=  ... definition operator|      ... choice operator (exclusive or)//     ... comment*italic* ... recommended** **

## Basically

**From v.2.1.1 or in addition to v.1.1**
**No space management**
**Automatic**
**minimum directory management**
**Ls, removeFiles**
**No mkdir, etc.**
**No permission management**
**Somewhat automatic**
**Some new functions**
**srmGetSystemInfo // ConfigInfo**
**Ping**

## Relationship to v.1.1 and v.2.1.1 (1)

| v.1.1 | This version | v.2.1.1 |
| --- | --- | --- |
| requestID = integer | requestID=string | requestToken
                      =string |
| requestStatus | Separated into RequestSummary and ReturnRequestStatus. FileStatuses from v.1.1 is separated out.  | Split into RequestSummary and RequestFileStatus as the response to the request.  |
| FileMetaData  | Enhanced FileMetaData  | MetaDataPathDetail  |

## Relationship to v.1.1 and v.2.1.1 (2)

| v.1.1 | This version | v.2.1.1 |
| --- | --- | --- |
| RequestFileStatus  | RequestFileStatus  | Many different request status. E.g. GetRequestFileStatus, PutRequestFileStatus, etc.  |
| StatusCode  | Enhanced and
concrete
Explanation added  | extensive  |
| automatic storagetype | FileStorageType desired to be added  | fileStorageType added |

## Relationship to v.1.1 and v.2.1.1 (3)

| v.1.1 | This version | v.2.1.1 |
| --- | --- | --- |
| Get  | Enhanced  | srmPrepareToGetAdded more parameters  |
| Put | Enhanced | srmPrepareToPutAdded more parameters  |
| Copy | Enhanced | srmCopyAdded more parameters  |

## Relationship to v.1.1 and v.2.1.1 (4)

| v.1.1 | This version | v.2.1.1 |
| --- | --- | --- |
| MkPermanent  | srmChangeFileStorageType  | srmChangeFileStorageType  |
| Pin  | Automatic when file is ready  | automatic  |
| UnPin  | srmRelease, or automatic when file lifetime is expired  | srmRelease or automatic  |
| Additional pinning to extend lifetime ? | srmExtendedFileLifeTime added  | srmExtendedFileLifeTime added  |

## Relationship to v.1.1 and v.2.1.1 (5)

| v.1.1 | This version | v.2.1.1 |
| --- | --- | --- |
| getRequestStatus  | srmGetRequestStatus to return RequestSummary  | Separated into request type. E.g. srmStatusOf{Get,Put,Copy}Request  |
| Together in getRequestStatus | srmGetRequestFilesStatus added to return detailed information on each file  | Separated into request type. E.g. srmStatusOf{Get,Put,Copy}Request  |
| getFileMetadata  | srmLS with ability to return single directory meta information | srmLS with extensive
Options Enhanced
directory management  |

## Relationship to v.1.1 and v.2.1.1 (6)

| v.1.1 | This version | v.2.1.1 |
| --- | --- | --- |
| getProtocols  | srmGetTransferProtocols  | none  |
| getEstGetTime  | Absorbed into srmGetRequestStatus 
Or srmStatusOf{Get,Put,Copy}Request | Absorbed into srmStatusOfGetRequest  |
| getEstPutTime  | Absorbed into srmGetRequestStatus
Or srmStatusOf{Get,Put,Copy}Request  | Absorbed into
srmStatusOfPutRequest  |

## Relationship to v.1.1 and v.2.1.1 (7)

| v.1.1 | This version | v.2.1.1 |
| --- | --- | --- |
| setFileStatus  | srmPutDone  | Enhanced into srmChangeFileStorageType
Or srmPutDone  |
| AdvisoryDelete  | srmRemoveFiles ? | srmRemoveFiles  |
| Automatic space management | Automatic Space Management  | Space Management  |
|  |  | Permission Management  |
|  | minimal | Directory management |

## GET

**In: SURLs, Protocols****	****AuthorizationID, ****	****DesiredFileLifeTime,****	****FileStorageType****Out: ****RequestStatusSummary****	**** RequestID**
**Basic assumptions: UserID**

**Get()**
**In: SURLs, Protocols****Out: ****RequestStatus**** **

**srmPrepareToGet()**
**In:****	****authorizationID,**_**arrayOfGetFileRequest,**_**arrayOfTransferProtocols, userRequestDescription, storageSystemInfo, **
**	****totalRetryTime**
**	**
**Out:****	****requestToken**_**,**_
**	****	**_**returnStatus,**_** **
**	****	****arrayOfFileStatus **

## RequestStatusSummary

**RequestID****Type (Get, Put, ... )**** ****State**** ****ErrorMessage ****FileStatuses****RetryDeltaTime****SubmitTime****StartTime****FinishTime****EstimatedTimeToStart**** **

**RequestID****RequestType****StatusCode ****// RequestErrorCode****StatusExplanation**** ****// ErrorDetails****SubmitTime****StartTime****FinishTime****EstimatedTimeToStart****TotalFilesInRequest****NumberOfQueuedFilesInRequest****NumberOfFinishedFilesInRequest****NumberOfProcessingFilesInRequest**

**requestToken,****requestType,**
**	****totalFilesInThisRequest,**
**	****numOfQueuedRequests,**
**	****numOfFinishedRequests,**
**	****numOfProgressingRequests,****isSuspended **

## PUT

**Put()**
**In: ****Src_file_names****     Dest_file_names,**** ****     Sizes****     WantPermanent**** ****    Protocols**** **
**Out: RequestStatus**

**In: ****{****Dest_SURL****	****Size ****}[]****      ****FileStorageType****      Protocols****      ****AuthorizationID****     ****OverwriteMode****     ****DesiredLifeTime**** **
**Out: ****RequestStatusSummary****        ****RequestID**

**srmPrepareToPut()**
**In:****	****authorizationID**_**arrayOfFileRequest,**_**	**
**	****arrayOfTransferProtocols,**
**	****userRequestDescription,**
**	****overwriteOption,**
**	****storageSystemInfo,**
**	****totalRetryTime**
**Out:****	****requestToken**_**,**_
**	****	**_**returnStatus,**_** **
**	****	****arrayOfPutFileStatus**

## COPY

**Copy()**
**In: Source_SURLs,****     Dest_SURLs**
**Out: RequestStatus**

**In: ****{**** Source_SURL,****	****Dest_SURL ****}[]****	****AuthorizationID**** ****	****DesiredLifeTime****	****FileStorageType****	****OverwriteMode****Out: ****RequestStatusSummary****        ****RequestID**

**srmCopy()**
**In: authorizationID,**
**	****arrayOfCopyFileRequest,**
**	****userRequestDescription,**
**	****overwriteOption,**
**	****removeSourceFiles****storageSystemInfo,**
**	****totalRetryTime**
**	**
**Out:****	****requestToken**_**,**_
**	****	**_**returnStatus,**_** **
**   ****	****           ****arrayOfCopyFileStatus **

## MkPermanent

**MkPermanent**
**IN: SURLs**
**OUT: ReturnStatus**

**srmChangeFileStorageType()**
**In:****	****authorizationID,**
**	****	****arrayOfPath,****	****	**
**	****	**_**desiredStorageType,**_
**	****	****spaceToken**
**Out:****	**_**returnStatus,**_
**	****	****arrayOfFileStatus **

## PIN

**Obsolete**
**Automatic when file is ready**
**In: TURLs**
**Out: RequestStatus**

## UnPin

**UnPin()**
**In: RequestID****	****TURLs**
**Out: ****RequestStatus****	**

**UnPin**** => Release**
**In: RequestID****	****TURLs ****| SURLs****	****AuthorizationID**
**Out: ****	****{ TURL | SURL****	****  ****RequestStatusCode****	****  Explanation } []**

**srmReleaseFile()**
**	****IN: requestToken****    authorizationID,**
**	****    ****siteURLs,**
**	****    ****keepSpace**
**Out:****	**_**returnStatus,**_** **
**                ****arrayOfFileStatus **

## getRequestStatus

**In: RequestID**
**Out: RequestStatus**** => RequestStatusSummary**
**For details: getRequestFilesStatus ****(previous getRequestStatus which used to include RequestFileStatus and FileMetaData)**

**getRequestStatus()**
**In: RequestID**
**Out: RequestStatus**

**srmStatus(Get,Put,Copy)Request**
**IN:  requestToken,**
**	****       ****authorizationID**
**	****       ****arrayOfFromSURLs,**
**Out:****	**_**returnStatus,**_** **
**	****           ****arrayOfFileSta **

## getFileMetaData

**getFileMetaData**
**In: SURLs****	**
**Out: FileMetaData**

**getFileMetaData**** => LS**
**In: SURLs****	****AuthorizationID****	****IsAllLevelRecursive****	****ListOffset****	****ListCount**
**                  ****infoOnly**
**Out: FileMetaData**

**srmLs()**
**IN: authorizationID,**
**	****      ****path,**
**	****      ****fileStorageType,**
**	****      ****fullDetailedList,**
**             ****allLevelRecursive,**
**             ****numOfLevels,**
**             ****offset,**
**             ****count**
**Out: ****	****MetaDataPathDetail[]**
**                  **_**returnStatus**_** **

## FileMetaData

**SURL****Size****Owner****Group****PermMode****ChecksumType****ChecksumValue****IsPermanent**** ****IsPinned**** ****IsCached**** **** **

**SURL****Size****Owner****Group****PermMode****ChecksumType****ChecksumValue****FileStorageType****StatusCode****StatusCode****LifetimeAssigned****LifetimeLeft****CreatedTime****LastModifiedTime****FileType****OriginalSURL****SubDirectoryPTR**** ****// dirEntries**

_**path**_**, **
**	****status, **
**	****size,    // 0 if dir**
**	****ownerPermission,**
**	****userPermission,**
**	****groupPermission,**
**	****otherPermission**
**	****createdAtTime,**
**	****lastModificationTime,**
**	****owner,**
**	****fileStorageType,**
**	****fileType, // Directory or FilelifetimeAssigned,**
**	****lifetimeLeft,**
**	****checkSumType,**
**	****checkSumValue,**
**	****originalSURL, MetaDataSubPathDetail[]**

## getProtocols

**In: none**
**Out: list of transfer protocols**
**None in v.2.1**
**Let’s keep this or put into the new srmSystemInfo or srmConfigInfo**

## getEst[Get/PUT]Time

**Make it Obsolete**
**getEstGetTime: **
**Out- RequestStatus ****In- SURLs, Protocols**
**getEstPutTime: **
**Out- RequestStatus ****In- Src_file_name, **
**	****dest_files_names, ****sizes, wantPermanent,**
**	****Protocols**

**getEstimationTime**
**In:**** ****Source_SURL****	****Dest_SURL****	****Size**
**	****	****AuthorizationID****	****TransferProtocols****	****FileStorageType**
**Out:**** ****RequestEstimation****	**

**None**
**Similar one is getStatusRequest() after request is submitted****	**

## AdvisoryDelete

**Obsolete**
**AdvisoryDelete: **
**Out- none****In- SURLs**

**srmRemoveFiles**
**In:**** ****SURLs****      authorizationID**
**Out:**** ReturnRequestStatus**

**srmRemoveFiles**
**IN:  requestToken,**
**	****       ****authorizationID,**
**	****       ****siteURLs**
**Out:****	**_**returnStatus,**_** **
**                  ****arrayOfFileStatus **

## GetRequestFilesStatus

**In: RequestID****Out: RequestFileStatus []**

**In: RequestID****	****ListOffset****	****ListCount**
**Out: RequestFileStatus []**

**srmStatusOf{Get, Put, Copy}Request**
**IN: **_**requestToken,**_
**	****      ****authorizationID**
**             ****arrayOfFromSURLs,**
**Out:****	**_**returnStatus,**_** **
**	****	****arrayOfFileStatus**

## RequestFileStatus

**SURL**** ****Size****State**** ****Fileld****TURL****Owner, ****Group, ****PermMode****CheckSumType, ****CheckSumValue****IsPinned, ****IsPermanent**** ****QueueOrder****EstSecondsToStart**** ****SourceFilename****DestFilename****IsCached**** **

**Source_SURL****Dest_SURL****Size****StatusCode ****Explanation ****TURL****CheckSumType, ****CheckSumValue****FileStorageType****EstWaitingTimeOnQueue****EstProcessingTime****SourceFilename****DestFilename****RemainingLifetime**

_**fromSURLInfo**_**,****	****fileSize,**
_**status**_**,**
**estimatedWaitTimeOnQueue,**
**estimatedProcessingTime,**
**transferURL**
**remainingPinTime **
**siteURL, **
_**toSURL**_**, **

## Ping

**In: none**
**Out: alive_response**

## GetConfig

**In: flag**
**Out: { value | ****	****list of supported functions}**** **
**	****e.g. GET, PUT, COPY, ****	****	****DYNAMIC_MKDIR, ****	****	****PUSH_MODE, PULL_MODE,****	****	****Default_space = 5GB/user, etc.**

## New types

**FileStorageType**
**VOLATILE, DURABLE, PERMANENT**
**FileType**
**FILE, DIRECTORY, LINK**
**OverwriteMode**
**NEVER, ALWAYS, WHEN_DIFFERENT**
**RequestType**
**GET, PUT, COPY**
**StatusCode **
**// RequestStatusCode, FileStatusCode, RequestErrorCode, FileErrorCode**
**Explanation: human-readable strings ****// ErrorDetails**
**RequestID**** : from int to string**

## Conclusion

**Based on SRM v.1.1 and many of v.2.1.1**
**Included much of Olof’s issues and suggestions to v.1.1**
**Operational specification would be bound to Web Services. **