# Presentation: 235623

## Current status

- Station does talk to dCache via SRM.
- Prestaging, correct sam location -> gsiftp/dcap translation
- Sam download is in fact available (more testing though needed)
- Station does use SRM over unix FS to stage files. Provides analysis with valid access urls.
- Distributed analysis via SAM on SRM is a done deal.
- Station release has been deployed at cdfsam09 (and pilot tested)
- Station does write to permanent storages ( tape backed dCache.). The outcome of that is not usable due SAM location registration issues.
- Location replication agreement should be understood first.

## Experiment data import. Basic concept

- Replace FSS with sam station to symmetrically import data INTO FNAL central storage (dCache)
- SAM station usage pattern is to distribute data. However , SAM station can equally collect data at the central storage provided offisite analysis properly registers metadata and sets up (registers in the DB , persistence ! ) inital (source,local) replica location. In return we get recoverability of and full control over data inflow.  If a user wants something stored , he must register his data at his site and wait until central "importer" script wakes up and starts streaming his request into the central storage of choice. Hence, if the central capacity to accommodate all incoming data is scarce, site production will be pushed back by reduced availability of the their own local storage. In a more general sense,  user will contact the destination station to start the migration of his data there. The principal point is that at any point data propagation in and out of site can be  monitored and accounted through our metadata catalog.
- Major benefit (the experiment):
- Automation of and full control over incoming traffic !!!!
- No more fcp, and ad-hoc hardware deployments.
- Minor benefit (SAM support): No more FSS

## Site

**SRM**

**SAM station**

**SAM station**

**MC**

**MC**

**MC**

**MC Importer**

**Metadata**

**add md/register location**

**upload**

**dCache/enstore**

**Site**

**FNAL**

**MC validato**r

## In a detail.

- Jobs registers new file location at a storage
- Uploads a file ( note a sequence )
- Verifies upload result using SAM enabled analysis or simple “sam get dataset”.
- A file automatically falls under the local SAM station management

## ... continued

- Data is “marked” in the metadata catalog
- ”mark” is used to build a dim query to run a project on the central concentrator station (at FNAL).
- The project is set such that its data is uploaded by the central concentrator station into the taped backed dCache

## Benefits.

- Data flow management is transparent to the underlying hardware.
- Data is been pulled in a automatic way that guarantees availability of the storage space and network bandwidth as well as smoothes out spikes in production activities.
- Service failures are recoverable.
- Overall simplification (hence better supportability) of the DH infrastructure.

## Open questions

- Space allocation for the primary MC data.
- Even though files will be imported into the station cache, initial upload is not managed.
- Still, the biggest question is : Autodestination
- Station does not talk to autodest server when generating dest locations

## Work plan

- Modify sam_upload tool to use srmcp and autodest selection. Register MD and location. (2 weeks )
- Later transition to sam store that will utilize station existing feature to “reserve or wait” feature.
- Station may utilize source location to build destination (location replication feature). (3 weeks )
- The exact implementation of that feature is not yet clear. Some sort of common agreement needed.

## Continued...

- Deploy UNIX SRM service at a site  with fixes ( 2 weeks)
- UNIX SRM has some reliability issues that will need to be addressed ( @ Timur )
- Deploy and configure SRM enabled station there (1 day ).
- Deploy and configure SRM enabled station at FNAL (1 day ).
- Understand how to validate existing metadata (2 weeks ).
- Add new attribute ? Bold solution !
- Come up with an importer (a cron job ?) ( 1 week )
- Watch things singing and dancing .... (forever)

## Overall

- Worst case total : Around 10 man weeks.