**Grid File Access**

**Proposal**

David Foster, Don Petravick, Michael Ernst

  ------------ ------------------------------------------------------------
  *Date:*      **20th March 2003**

  *Version:*   **1.3**

  *Status:*    **Draft**

  *Editor:*    **David Foster**
  ------------ ------------------------------------------------------------

  -------------------------------- ------------ ------------------ -----------------------------------
  []{#__RefHeading___Toc35831489                                   
  .anchor}**Document Log**                                         

  **Issue**                        **Date**     **Author**         **Comment**

  0.1                              14/12/2002   DGF                First Version

  0.2                              02/28/2003   ME, DP             Second Version

  0.4                              03/03/2003   ME, DP             third version, accommodate DGF's
                                                                   input

  0.5                              03/06/2003   ME, DP             added specific S/W components

  0.6                              03/07/2003   ME, DO             changes in response to David's
                                                                   comments

  1.0                              03/14/2003   DGF                Preparation of 1.0

  1.1                              03/18/2003   DGF                Exec Summary and Conclusions

  1.2                              03/20/2003   DGF                Included comments from Jeff, Oxana,
                                                                   James Casey

  1.3                              03/21/2003   DGF                Included comments from John Gordon,
                                                                   J-P Baud
  -------------------------------- ------------ ------------------ -----------------------------------

  -------------------------------- ----------------------- -----------------------------------------
  []{#__RefHeading___Toc35831490                           
  .anchor}**Document Change                                
  Record**                                                 

  **Issue**                        **Item**                **Reason for Change**

                                                           

                                                           

                                                           
  -------------------------------- ----------------------- -----------------------------------------

# Mandate and Introduction

> *This working group was mandated by the Grid Deployment Board at its
> meeting of the 9^th^ December 2002 and subsequently reaffirmed at the
> meeting of the 6^th^ February 2003.*
>
> *The motivation is to understand the Grid File Access requirements and
> issues and present an understanding as to the state of the technology
> and the strategy going forward.*
>
> *In particular a specific recommendation for LCG-1 should be a product
> of this work.*
>
> ***This document does not cover all the discussion or background but
> is really focussed on the solution proposal and the implications.***

# Table of Contents

[**Document Log** ii](#__RefHeading___Toc35831489)

[**Document Change Record** ii](#__RefHeading___Toc35831490)

[1 Mandate and Introduction 1](#mandate-and-introduction)

[2 Table of Contents 2](#table-of-contents)

[3 Executive Summary 3](#executive-summary)

[4 Basic Requirements 4](#basic-requirements)

[5 Architectural Models 7](#architectural-models)

[6 Job Workflow 9](#job-workflow)

[7 The Proposal 11](#the-proposal)

[8 Middleware Discussion 15](#middleware-discussion)

[9 System Requirements 16](#system-requirements)

[10 Software Recommendations 17](#software-recommendations)

[Appendix 1 -- File naming conventions
18](#appendix-1-file-naming-conventions)

[Appendix 2 - Physics Data Objects
19](#appendix-2---physics-data-objects)

[References 21](#references)

##  {#section .unnumbered}

# Executive Summary

This document proposes the use of the SRM interface for mass storage
manipulation together with the different access protocols (e.g. RFIO,
dCap) for file access to a local storage system. It is assumed that the
local storage system will be grid accessible through the GridFTP
protocol so files can be copied into the storage system.

There are different levels of complexity that can be envisaged:

1.  Deploy RFIO for worker node to file system access. Provide GridFTP
    services to give the possibility of copying in/out data to the wide
    area.

2.  Add the possibility to manipulate mass storage through a SRM client
    library and an associated SRM service for the local mass storage
    system. There is a strong argument in favour of hiding mass storage
    from the worker nodes and applications. This will require future
    data management services to provide these transparent services.

3.  Allow for the possibility of multiple file access protocols. This
    would require however a mechanism for matching job capability to the
    access mechanism in the absence of a common abstract layer (Posix
    I/O?) hiding the details of the access mechanism from the
    application.

This leaves a lot of complex questions unanswered. Indeed many people
are working on issues such as the security implications of local and
wide area data access.

However, while waiting for this work to complete, what is presented is a
pragmatic approach to deployable functionality in order for experience
to be gained. It is proposed to implement Level 1 and Level 2. This
functionality will be tested by taking the EDG-WP5 implementation
provided as part of the EDG testbed 2.

[However, there are alternative software solutions if major problems are
found with this approach. Such solutions will probably require
additional integration work.]{.underline}

The rest of this document will discuss a number of the issues and
implications in some detail but it is clear that a complete grid storage
system architecture is required for the final design of LCG. This
information should provide input to a future architecture design.

# Basic Requirements

*An application running on a worker node must be able to perform basic
Open/Seek/Read/Write operations on a file on the local storage element.
The implementation should work with the file selection process below:*

In LCG-1 a cluster of worker nodes are supported by a storage element.
The storage element is a combination of storage related services and
physical storage. In this document this storage element functionality is
limited and is largely used as a temporary store. Although we recognize
there are permanent stores, we detailed the case of a temporary store
"near" a compute element servicing the I/O requirements for the jobs
running on the worker nodes.

Ideally some combination of middleware, e.g. the Resource Broker and the
Replica Manager, not the job itself, will cause replicas of a job's
input data to be placed on a cluster's storage elements in time for job
execution, and middleware will handle the validated output of a job
appropriately, for example, by moving it to a permanent store of some
type.

These Storage Elements have a Wide area network (or GRID) interface and
a local interface (typically LAN).

[Grid Interface]{.underline}

Looking out of the fabric and towards the grid, Grid replica services
must be able to place files on these storage elements and retrieve files
from these storage elements. In addition to file movement, the storage
element must also present management interfaces to grid middleware. An
example of a management interface: allowing grid middleware to confirm
that a replica it thinks ought to be on the storage element is actually
there, and to reserve space for the expected output of the job.

[LAN Interface]{.underline}

Looking into the fabric, a job running on a worker node must be able to
read files from the storage element, and write its output onto the
storage element. Read and write access requirements include the open(),
read(), write(), close(), creat(), and seek() functionalities.

[File Access]{.underline}

Current HEP grid sites provide storage elements using a variety of
technologies.

Two important categories of technologies are

-   Kernel like, possibly distributed file systems.

-   User-code distributed file systems.

NFS typifies a Kernel level file system. While an NFS-based storage
element needs additional services on the \"grid\" side, kernel
techniques provide standard access mechanisms for application code to
access file data as the files look like local files so standard I/O
mechanisms can be used. Not all HEP sites have found that these sorts of
systems scale to their needs. Consequently, a number of \"user code file
systems\" have been put into service. A partial list (below) gives a
notion of the complexity of the environment.

[System access method Example site]{.underline}

Castor\[12\] rfio (protocol) CERN

Dcache/Enstore dcap (protocol) Fermilab

Dcache /OSM dcap (protocol) DESY

HPSS rfio (protocol) IN2P3

HPSS hpss custom protocol Various

rootd Rootd (protocol) Various

POSIX file sys kernel file interface Various

Slashgrid\[3\] kernel file interface Various

Table 1 \-- Diversity of File Access Protocols

[File Naming]{.underline}

A single file may be identified by many different names. However, a
Globally Unique IDentifier (GUID) should be assigned to each new dataset
by the system.

Logical file names (LFNs) may be provided as aliases to the GUID.

The Replica Metadata Catalog (RMC) maps an LFN provided by an
application to a GUID, and the Replica Location Service (RLS) maps the
GUID to a Site File Name (SFN).

The SFN is of the form "machine:port/path understood by machine", which
gets translated by middleware into a Site URL (SURL) e.g. of the form
\"srm://machine:port/path understood by machine\".

The SURL is transformed by the Storage Resource Management system to a
Transport URL (TURL) which is the final specification of the location of
the file and the protocol to be used.

[Grid Implications]{.underline}

From the LAN side, there **is no standard mechanism or interface to
these I/O sub-systems**. Therefore, before a job is submitted to a given
worker node the I/O system it supports must be known and compatible with
the expectations of the job. **This is significant if we imagine a grid
with mixed I/O systems.**

1.  Sites must declare the I/O capabilities of their storage elements.
    This can be subject to a detailed design, one notion might be:
    Compute elements must declare what loadable APIs to local I/O are
    available for jobs. (e.g. root_io, file_io), and must set up the
    environment appropriately (e.g. LD_LIBRARY_PATH)

2.  Jobs could declare their ability to handle various I/O protocols.
    How this is done can be subject to a detailed design, an example
    notion is: Jobs could declare which protocols and API's they are
    able to use for local file access. (root_io, file_io). A reasonable
    thing is to suppose jobs with no declaration can just do
    "file::...."

3.  Schedulers and /or resource brokers (i.e. Workload Managers) will
    consider these declarations and place jobs appropriately.

4.  Files are fully specified using a URL-like syntax, we always convert
    to a SURL of the form \"srm://\...\" and then ask the relevant SRM
    for a TURL.

5.  Integration with scientific I/O packages should be supplemented by
    provision of a standard utility for copying files from the worker
    node to and from the storage element.

Table 3 \-- Requirements induced by diverse access methods

**Simplicity would dictate a single deployed I/O system initially.**

[Storage Management]{.underline}

Storage management services should **coordinates space allocation and
file transfer requests \[11\].** The requirements include:

-   Grid Middleware should be able to determine what datasets are on the
    local storage elements.

-   Creating or Staging datasets of many files to the local storage
    element should be supported by space reservation functionality that
    helps assure the success of the whole operation.

-   Data should move directly between storage elements. Entities like
    replica managers should be able to master
    storage-system-to-storage-system transfers.

-   Data movement between storage elements should be made reliable at
    the lowest level feasible.

**Currently very little exists in the way of data management services
and should be part of future storage architecture and developments.**

The Storage Resource Manager (SRM) **manages** the storage resources, it
is not involved in the actual data movement apart from possibly
initiating transfers. This is illustrated in figure 3. There the
application is performing all management (space allocation, SFN/TURL
conversion etc) operations through its SRM client while the actual data
transfer is utilizing the protocol specified in the TURL, i.e. the dCap
protocol in this particular example.

# Architectural Models

The storage resources for a grid job may consist of two types of
storage.

1.  Sandbox storage is temporary disk space, used as scratch, perhaps to
    hold executables, and is available for the duration of the job, and
    is presented as a file system to a job. However this approach is
    undesirable were there are many similar jobs with the same file
    requirements or where the files involved are very large.

2.  Storage on Storage Elements is the main way files are fed to and
    from jobs. Storage elements have a Grid side and a Local side.

[Local Access]{.underline}

Within a fabric, local storage resources are provided by a site to a job
using local methods. For example, the job may access a file through
physics I/O libraries, with site-local authentication and authorization
mechanisms. Examples that illustrate the local access concept are: a
storage element that is an NFS server, or a storage element that runs a
rootd server.

Local access requirements include:

-   allowing for the local side of the storage element to be behind the
    same firewall as the compute elements.

-   allows for local authentication and authorization mechanisms.

-   allows for local low-level administration mechanisms for the storage
    server, including cluster administration software, like Rocks.

[Grid Access]{.underline}

Grid software will use grid methods to place input data onto fabric\'s
storage elements to allow for computation on a site\'s worker nodes.
Grid data management software will copy or make permanent a job's output
data sets, and make them part of the virtual organizations\' data.
Therefore storage elements must have a GRID interface, which is
presented in a grid environment. Elements of a grid environment include:

-   Presence of firewalls and other protective measures.

-   The use of GRID authorization and authentication measures.

-   Given a scheduling decision to run a job on a site, placement of
    input data may proceed without knowing which specific computer on
    that fabric node will run the job.

-   We allow for storage elements to buffer the output of a job, freeing
    the processor for a new job.

There are many ways to architect systems with these properties. An
illustrative system, but by no means the only one, is pictured below:
The illustration depicts worker nodes on a local network isolated from
the wide area network.

![](media/image2.png){width="6.847222222222222in"
height="5.146527777777778in"}

Depicted is a local, isolated LAN holding the worker nodes, and a wide
area network. Two computing systems straddle the local network and the
wide area network.

A head node, which provides batch system access, and other services
outside the scope of this document, and may provide some sort of
firewall.

A scalable storage system implemented on a number of disk-heavy
commodity computers, which implements only grid services on the wide
area network, and file serving and other protocols on the isolated local
area network.

In a simpler case, the storage system could be a single node, with two
network interfaces. The node could run local firewall software allowing
it to only offer GridFTP and other Grid services to the wide area
network. On the local interface any number of appropriate services could
be offered. The local interface could be available for any sort of
administrative use or other function.

There are many variations on this theme, the primary requirements are
that the grid-side protocols needed by the storage element are available
to the wide area network, and that the throughput to the storage element
be sufficient to accommodate the movement of data sets to and from the
fabric.

# Job Workflow

As discussed in the basic requirements sections, the SRM can be used to
parameterize specific I/O mechanisms, deferring decisions on which
protocol to use until run time. This is an advantage, since this
accommodates the diverse storage systems currently in place.

Under these conditions:

-   Jobs must declare which protocols they can speak.

-   Fabric sites must declare the protocols their storage elements can
    speak.

**Replica Manager pre-staging**

Sometime before a job is released to run, a replica manager might move
input data to prepare for the job execution. The LCG grid storage
elements must have a grid interface that has a management component in
addition to the obvious data transfer component.

Storage elements have to present a \"grid side\" interface that is
appropriate for a replica manager to use. The elements of the interface
include

-   Allowing the replica manager to check that files it supposes to be
    on the storage element are indeed there.

-   Determining that there is available capacity, quota and suitable
    permissions for the new files to be placed on the storage element.

-   Taking steps to make sure replicas stay on the storage element for
    the duration of the job.

-   Possibly optimize the files on the storage system, for example stage
    the input files in the case of a Hierarchical Storage Element.

-   Replicate data onto the local store.

**Output data set handling**

The output of a job has to be handled. Some elements of handling are:

1.  Possible movement of the data from scratch areas (on the worker node
    or storage element) to a permanent storage system (or equivalent
    operations for systems that can perform this operation internally)

2.  Retention of the data on the storage element.

3.  Deletion of the data.

The SRM component proposed for file management provides for a number of
primitives that are useful for these operations.

-   Structured query of the storage element's name space (v2.1)

-   Space reservations, increasing the probability that write operations
    will succeed.

-   Pinning files in the store.

-   Pre-stage files, for storage systems having a hierarchy.

-   Have the storage element copy files directly to/from other storage
    systems in a reliable fashion, with interface semantics appropriate
    for replica managers.

-   Retaining files on the local storage element until they can be
    copied off the storage element or otherwise made permanent.
    (\"Durable spaces\")

# The Proposal

The SRM protocol is used for URL translation, name translation, and
protocol selection.

The GRIDFTP protocol V1.0 is used for **grid file transfers** and as a
common **local transfer protocol.**

These are supported by a variety of packages, summarized in the table
below:

  ---------------- --------------- -------------- -------------- --------------
  Package          SRM V1.1        GridFTP V1.0   Temporary      Permanent
                                                  Store?         Store?

  EDG WP-5         V2.1but         yes, via       yes            Yes either as
                   supports        Globus                        disk or with
                   required                                      backing MSS
                   functionality                                 
                   so can easily                                 
                   back-fit V1.1                                 
                   interface.                                    

  DRM(LBL)         Yes             yes            yes            no

  Dcache           Yes             yes            yes            no
  (stand-alone)                                                  

  Dcache/Enstore   Yes             yes            no             yes

  CASTOR           Yes             yes            yes            yes
  ---------------- --------------- -------------- -------------- --------------

This approach accommodates all the types of systems listed in the basic
requirements, and allows for the run-time negotiation of transfer
protocols.

The diagram below depicts how the proposed protocol works for job I/O in
the LCG system for reading data. \[The placement and distribution of
replicas on the storage element is discussed in section 6, Job
work-flows\]

![](media/image3.png){width="5.767361111111111in" height="4.325in"}
Figure 2 -- Local and Grid Interactions with a Storage Element

**In step 1, Grid Middleware seeks to place a job's input files on a
storage element before a job has been queued to the local batch
system.**

The required steps are detailed below:

1.1 The replica managers query the storage element to confirm the
existence of replicas its database indicates are on the storage element.
To confirm the existence of the file, the SRM interface is used. This is
because the reference is the SFN or \"srm:\" SURL here, not a URL to the
resource itself.

Replica managers keep files on the temporary store by using the **SRM
interface to pin them**. Pinned files may disappear because of
operational accident (e.g disk crash, or "rm" out of band). Replica
managers indicate disinterest in a file by undoing the SRM pin operation
(SrmRelease), marking files as eligible for deletion by the SRM when
space is needed.

1.2 The replica manager reserves space on the storage element for
replicas it will place there using the **SRM reserve space** operation..
If the temporary store is full, the temporary store deletes un-pinned
files using some implementation-specific algorithm, e.g. least recently
used.

1.3 The replica manager commands the storage element to copy the
replicas into the reserved space by using the **srmCopy service**. This
is supported in Version 1.1 of SRM.

1.4 The SRM server causes the storage element to use GridFTP (or other
transfer protocol) to place the replicas onto itself. The transfers are
"reliable" in the sense that the SRM will re-try errors. This is
SrmCopy() in Version 1.1 of the protocol.

1.5 The replica manager pins the copies on the storage system using
SrmPin.

1.6 **The replica manager notifies the job manager that pre-staging for
this job is complete**.

![](media/image4.png){width="6.344444444444444in"
height="4.752777777777778in"}Figure 3 -- Software layering diagram of an
application and a storage element, Notice the flat, un-layered
relationship between the client software in the application.

**In step 2, the job interacts with the storage element's resource
manager to obtain a transfer URL.**

As indicated in figure 3, the replica manager client, SRM client and
scientific I/O libraries are independent, \-- not layered on themselves.

2.0 The Physics application requests the Experiment's Framework to open
a file.

2.1 The Experiment's Framework interacts with the replica manager to
obtain SFN's for its LFN's. (the job does not know about GUIDS). This
interaction is not layered through SRM software.

2.2 The Experiment's Framework computes an SURL based on the SFN (this
composing a string of the form "srm://host:port/SFN/").

2.3 The Experiment's Framework calls the SRM to get a TURL from the
SURL. (SRM V1.1) and a list of I/O protocols the scientific I/O package
can speak (n.b. Root can be queried for its I/O capabilities at run
time).

**In step 3. the application opens and reads it's input.**

3.1 The job calls a scientific I/O library in an open operation. It
gives the library the TURL passed back by the SRM. The job read its data
and closes the file through the scientific I/O library.

**In step 4, Grid Middleware moves the permanent output of the job to a
permanent store somewhere on the grid, using SRM copy, the transfer is
storage-system-to-storage-system.**

The figure illustrates a subset of the possible capabilities. Other
capabilities may include allowing for grid- wide access to data by the
job directly, if that is desired or necessary.

This proposal accommodates and subsumes the simple cases of kernel file
systems. Here the TURL is computed from to the SURL in a straightforward
way. An example of a source URL and its equivalent TURL are (under
suitable assumptions about uniform mounting of file systems):

SURL: srm://srm:srm-port/file-name

TURL: [file:///file-name](../../../../../../../../file-name)

**Other considerations**

In the practical experience of a number of labs, straightforward use of
a distributed file system has not been a success in data intensive
computing environments. This is because mutually uncoordinated I/O
requests can cause high levels of packet loss in a naive Ethernet SAN.
Fermilab, for example, gives its experiments a \"farms file copy\"
utility, which marshals access to farm head nodes, shaping the traffic
on the Ethernet LAN, and preventing the thrashing of the disks on file
severs. Such tools are not standardized among HEP computational sites.
The SRM protocol can be used in a uniform way as a traffic-shaping tool
by metering the TURLs, if required.

Storage system developers have also noted that the translation step
between SURL and TURL provides an opportunity to scale systems since the
SRM can load-balance across many machines and ports. An example of how
this may be used in the LCG-1 would be to support replicas of pile-up
data.

Very significantly, the proposal of SRM as a protocol is not coupled to
prescription of a single implementation. There are a number of groups
working on independent, inter-operating implementations of SRM. These
include groups that are implementing SRM features on large-scale central
data-centre infrastructure for example CERN (castor), DESY/Fermilab
(Dcache), Jefferson Lab (Jasmine) and LBL (HPSS). LBL have a Disk
Resource manager that is stand alone and lighter weight.

The DataGrid Storage Element is actively following SRM, and had
semantically equivalent functions in December, 2002. Fermilab is
considering implementing the SRM interface on its \"dFarm\" distributed
scratch space system used on its farms, and is interested in producing
Java Classes that might be useful to embed in other storage systems.
There is interest from the Condor group at University of Wisconsin.

Therefore there is acceptance at Tier 0 and Tier 1 centres, and there is
likely to be a "market place\" of suitable, open source packages for
Tier 2 and lower centers.

# Middleware Discussion

The interaction of the proposed storage element solution with the
underlying grid middleware is relatively easy to understand and plan.

The external interface to the storage element itself must be a
compatible implementation of a wide area grid data transfer protocol.
**The current logical choice is GridFTP**, which is a sufficiently
defined protocol that several groups have been able to make compatible
implementations. As proposed however, the software handling replicas
does not need to use grid FTP client software. The SRM_copy() feature of
the SRM protocol provides for a reliable file transfer directly between
storage systems, so that neither replica engines nor applications that
stage their own data handle file data directly.

In addition to the transfer tools there are elements for authentication
and authorization. The implementation of security on the storage
elements is not more complicated than security for executing process
requests. For data registered to the grid being transferred between
known storage elements, it may be possible to simplify the
authentication by using trust relationships.

# System Requirements

The system requirements for only outgoing TCP/IP access to compute nodes
proposed by the LCG Grid Deployment Board working groups and the gateway
and firewall architectures used currently in many computing centers
place significant restrictions on the methods used for grid data access.

However:

-   Each participating site will need a storage element as defined
    earlier in this document.

    -   The storage element will need at least one network interface
        with a valid external IP address. Depending on the configuration
        of the local cluster it may also have a second network interface
        connected to an internal local area network.

    -   The storage element must run an agreed set of grid services for
        incoming and outgoing data transfer. If the storage element is
        behind a local firewall, allowances may need to be made to
        enable services running on the storage element.

    -   The storage element must be connected to a storage pool of
        agreed size. The type of storage pool to deploy is a decision
        for the site manager. A multi-disk RAID device directly
        connected to the storage element or an aggregation of local
        disks on the compute nodes using tools like dCache, dFarm, or
        pvfs should be immaterial to the storage element functionality.

    -   The storage pool chosen must be able to determine the space
        available when polled, the length of time a given file has
        resided in the storage element, and the access properties of a
        given file.

    -   The storage element must have the same authentication,
        authorization, and VO management outlined by the grid deployment
        board working group.

-   LCG Applications running with the more flexible data access
    architecture will require some specific functionality.

    -   Applications running on the LCG will need to publish the local
        file access protocols that they support and fabric nodes will
        need to publish the local file access protocols used, so that
        the resource broker can assign requests properly. **This
        requirement can be removed if a single access method can be
        agreed upon.**

    -   Applications could be set up to request files of the storage
        element, which would then request to have them transferred if
        they did not reside locally. This simple implementation would
        work well with rudimentary resource brokers and would not
        require applications to publish the required input files before
        submission. More advanced resource brokers will be able to
        pre-stage files before submitting processes to improve
        efficiency. Applications in LCG-1 will publish the required
        information through the JDL.

#  Software Recommendations

Possible components to be deployed that match the proposed general
solution

  ----------------- ---------------- ------------- --------------- ---------------
  **Component**     **Package**      **Version**   **Group**       **Delivered**

  SRM Client\*      SRMcp            V1.1          FNAL            April 2003

  GridFTP Protocol  GridFTP          V1.0          Globus          Available

  Temporary Storage dCache           V1.5.1        FNAL/DESY       May 2003
  Sys.                                                             

  Temporary Storage DRM              V1.2          LBL             Available
  Sys.                                                             

  Temporary Storage CASTOR           V1.5.2.1      CERN            Available
  Sys.                                                             

  Permanent Storage CASTOR           V1.5.2.1      CERN            Available
  Sys.                                                             

  Permanent Storage dCache/Enstore                 FNAL            May 2003
  Sys.                                                             

  dCap Protocol     dCap             V2            DESY/FNAL       Available

  dCap Library      dCap             V2_24         DESY/FNAL       Available

  rfio Protocol     rfio                           CERN            Available
  ----------------- ---------------- ------------- --------------- ---------------

For LCG-1 clients may access SRM by executing UNIX programs.
Alternatively a Java class library is available. A C/C++ class library
is not currently available, though not impossible.

How the **[EDG-WP5]{.underline}** matches the overall proposal..

  ---------------------- ------------------- ------------- -----------------
  **Component**          **Package**         **Version**   **Delivered**

  SRM Client\*                                             April 2003

  GridFTP Protocol                                         April 2003

  Temporary Storage Sys.                                   April 2003

  Permanent Storage Sys.                                   April 2003

  rfio Protocol                                            April 2003
  ---------------------- ------------------- ------------- -----------------

# Appendix 1 -- File naming conventions {#appendix-1-file-naming-conventions .unnumbered}

The following is extracted from the SRM document (V2.0) and provides the
naming conventions agreed between PPDG and EDG WP2:

LFN- is a logical file name that is globally unique for a given dataset.
Thus, the dataset name is usually the first part of the LFN. It is the
choice of the dataset designer how to assign these names. If the files
are organized in directories then the directory names are part of the
LFN. The dataset name and directory names are separated by "/". An
example of an LFN is: "CERN-dataset-7/run17/part1/file-123".

SFN- is a file name assigned by a site to a file. Normally, the site
file name will consist of a "machine:port/directory/LFN", but the site
can choose to use another string instead of the LFN. An example of an
SFN is: "sleepy.lbl.gov:4000/tmp/foo-3000". In this example we used the
simple file name "foo-3000" to simplify the example.

SURL -- is a "site URL" which consists of "protocol://SFN". The protocol
for communicating with an SRM is simply "srm". An example of an SURL for
a file managed by SRM is: "srm://sleepy.lbl.gov:4000/tmp/foo-3000".

TFN -- is the "transfer" file name of the actual physical location of a
file that needs to be transferred. It has a format similar to an SFN.

TURL -- is the "transfer URL" that an SRM returns to a client for the
client to "get" or "put" a file in that location. It consists of
"protocol://TFN", where the protocol must be a specific transfer
protocol selected by SRM from the list of protocols provided by the
client (see recommendation 9). If the physical storage location matches
the one provided by the SURL, then only the protocol is replaced in the
TURL. For the above SURL example, and assuming the protocol is
"gridftp", the TURL will be:
"gridftp://sleepy.lbl.gov:4000/tmp/foo-3000". However, the physical file
location can be anywhere at that site, giving the freedom for the site
manager to change the physical locations of files without having to
change the SURL or update the replica catalog. If for the above example
the physical location of the file is on another machine (e.g.
"dm.lbl.gov", another path (e.g. "/home /level1"), and even another
file-name (e.g. "abc-3000") then the TURL will be:
"gridftp://dm.lbl.gov:4000/home /level1/abc-3000".

StFN -- is the "storage" file name that a client may request SRM to use
when it stores the file. This is useful for telling SRMs where to
archive a file. Normally, SRMs that archive files, such as an HRM, may
choose to honor that request. But, the SRM may choose to give it another
name, and return that to the client.

For LCG (POOL and EDG-RLS), we have additional semantics defined:

GUID -- The Global Unique ID is the truly enforced internal identifier
that the system assigns itself upon creation of the entry. The UUID
mechanism is used to create these identifiers. GUIDs are immutable, and
an SFN may have one and only one GUID.

LFN -- The additional semantic consideration for the LFN is that
although it should be globally unique, this cannot be enforced by the
system since this name is controlled by the users and applications. From
the user viewpoint, LFNs are mutable, may be changed and can be regarded
as aliases or symlinks to the GUID. A GUID may have many LFNs. LFNs
should enable the users to define human-readable aliases and enable
applications to impose their own semantics on the identifier. So where
the SRM document is not specific, we are: the mapping between LFN and
SFN is an N : 1 : M mapping where the 1 in the middle is the file\'s
GUID. So there may be many replicas of an SFN with the same GUID (1:M)
and there may be many aliases of a GUID (N:1). For the user the GUID is
hidden and we have the N:M mapping as advocated by the SRM and RLS
documents.

# Appendix 2 - Physics Data Objects {#appendix-2---physics-data-objects .unnumbered}

The interaction that is more complicated and will require an active
development program is the interaction of the resource brokers and grid
schedulers with the data storage. The LHC experiments, in general, have
been attempting to improve data accessibility by utilizing databases to
manage the storage. One common method for organizing the data in the
database is by completeness.

At the bottom level is the simplest information required to perform a
selection of the event described. Sometimes called a "tag", this data is
predicted to be between a few hundred and a few thousand bytes and
contains a pointer back to the full event. The next level of the
database contains the information required to perform a standard
analysis, often referred to as the analysis object data or AOD. It is
predicted to be between tens to hundreds of kilobytes. Above that there
is usually a level of event summary data, sometimes called an ESD. It is
usually predicted to be approximately half a megabyte, and is needed to
calculate complicated variables or recalculate elements in the AOD. The
final level is the raw event data, from which all other levels can be
calculated. The size estimates vary between experiments for the raw
event data.

A very simple application might access the data at only one level of the
database. For instance, a user might wish to make an event selection
based on information in the tags. This could be a fast process, because
even selecting from a sample of 10\^9 events might only be a few hundred
gigabytes to read over. It is easy to imagine that the application
request would communicate with the experiment database before submission
to the grid. The database would identify all the tag data files required
to complete the request and communicate this to the grid scheduler,
which would in turn assign the request to available computing resources
with the requested data or pre-stage the requested data to available
resources and submit the job. In a more complicated example, the user
might then wish to use the selected sample of events chosen from the
tags described above and perform an analysis using variables from the
AOD. One advantage of the database is that the tags have a pointer back
to event data and the AOD can be accessed transparently, if the database
files are available. Alternatively, the user might wish to
re-reconstruct selected events or recalculate variables on the AOD with
a new algorithm, which might require access to ESD database files for a
very small subset of events. The initial conditions that the resource
broker and grid scheduler used to optimize the placement of the tag
selection job do not completely describe the data requirements because
the job itself can point to additional needed database files.

In the past, these types of analysis jobs would have been accomplished
with multiple process requests: one to select the events and another
running on a different set of database files to calculate the additional
quantities or re-reconstruct the selected events. With the
implementation of a database it is possible to conceive of complicated
applications with a single request. However, it is difficult to maintain
the data accessibility over a wide area.

It is not obvious how an application running on computing resources
connects to the global database though the storage element can identify
which database files it needs next, if the complete set of required data
cannot be identified before a request is submitted. It is not possible
or reasonable to ship all the levels of the database, especially if only
a small subset is actually needed. Nevertheless, it is not clear how the
resource brokers and grid schedulers can optimise these types of
requests and choose appropriate computing resources. To optimise this
type of functionality it may require that the applications themselves be
re-locatable after they have been started, if the runtime data
requirements no longer match the initial assumptions. This would be a
significant application development effort. It will also require a
careful assessment of the capabilities of the computing system to know
how often requests of this type could be accommodated and have
middleware that would continually update the predictions of when a
request would complete.

Simple application requests can be handled with technologies and
techniques available today. To handle more complicated application
requests will require research into the interaction of the schedulers,
the applications, and the storage elements.

Pool provides technology neutral navigational access to distributed data
without exposing details of the particular storage technology.
Evidently, POOL would need to be SRM-aware.

Assuming

1.  POOL opens a container file when the first object in that file is
    requested,

2.  The file should be present on the local storage element when it is
    opened,

Pool would use the same primitives a that a replica manager would use to
place a file on the storage element:

1.  Query to see if the file is in the local store

2.  If not, reserve space for it, and bring the file into the local
    store using srm_copy()

3.  Pin the file in the local store.

4.  Release the pin when done with the file.

The "middleware selection" report of GDB working group 1 mentions
monitoring in the context of understanding, debugging and trouble
shooting. The SRM API has rich source of parameter information and it
seems likely that it could be interfaced to a monitoring system for
trouble shooting. Some effort should be applied to defining a common SRM
monitoring schema for all SRM implementations, which could be expanded
upon by individual implementations. This is currently a research topic
in the SRM community.

The "middleware selection" report of GDB working group 1 does not
specify that information exposed via the monitoring channel (Which we
understand is likely R-GMA) is used by schedulers and other grid
software.

# References {#references .unnumbered}

> \[2\] Coda File System http://www.coda.cs.cmu.edu/
>
> \[3\] SlashGrid http://www.gridpp.ac.uk/slashgrid/
>
> \[4\] AVFS http://sourceforge.net/projects/avf
>
> \[5\] UserVFS http://uservfs.sourceforge.net/
>
> \[6\] The Pluggable File System http://www.cs.wisc.edu/condor/pfs/
>
> \[7\] UVFS http://www.sciencething.org/geekthings/index.html
>
> \[8\] FUSE http://sourceforge.net/projects/avf
>
> \[9\] LUFS http://lufs.sourceforge.net/
>
> \[10\] newuserfs <http://sourcefrog.net/projects/newuserfs>
>
> \[11\] <http://sdm.lbl.gov/srm-wg>
>
> \[12\] <http://castor.web.cern.ch/castor/DOCUMENTATION/MAN/#rfio>