# Presentation: 147440

## ATLASMagda Distributed Data Manager

**Torre Wenaus**
**BNL**
- PPDG Robust File Replication Meeting
- Jefferson Lab
- January 10, 2002

## MAnager for Grid-based DAta
Focused on the principal PPDG year 1 deliverable
Deploy to users a production distributed data management service
Designed for rapid development of components to support users quickly, with components later replaced by Grid Toolkit elements
Deploy as an evolving production tool and as a testing ground for Grid Toolkit components
Designed for ‘managed production’ and  ‘chaotic end-user’ usage
Adopted by ATLAS for 2002 ATLAS Data Challenges
Developers - T. Wenaus and soon W. Deng (pdoc) and new hire

- Focused on the principal PPDG year 1 deliverable
- Deploy to users a production distributed data management service
- Designed for rapid development of components to support users quickly, with components later replaced by Grid Toolkit elements
- Deploy as an evolving production tool and as a testing ground for Grid Toolkit components
- Designed for ‘managed production’ *and*  ‘chaotic end-user’ usage
- Adopted by ATLAS for 2002 ATLAS Data Challenges
- Developers - T. Wenaus and soon W. Deng (pdoc) and new hire

**Magda**

- Info:              http://www.usatlas.bnl.gov/magda/info
- The system: http://www.usatlas.bnl.gov/magda/dyShowMain.pl

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Architecture & Schema

- MySQL database at the core of the system
- DB interaction via perl, C++, java, cgi (perl) scripts
- C++ and Java APIs autogenerated off the MySQL DB schema
- User interaction via web interface and command line
- Principal components:
**File catalog** covering any file types
**Data repositories** organized into ***sites***, each with its ***locations***
- Computers with repository access: a ***host*** can access a set of *sites*
- Logical files can optionally be organized into ***collections***
**Replication** operations organized into ***tasks***

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Magda Architecture

**Magda Architecture**

- Location

- Location

- Location

- Site

- Host 2

- Location

- Location

- Cache

- Disk Site

- Location

- Location

- Location

- Mass StoreSite

- Source to cache
- stagein

- Source to dest
- transfer

- MySQL

- Synch  via DB

- Host 1

- Collection of logical
- files to replicate

- Spider

- Spider

- scp, gsiftp

- Register replicas

- Catalog updates

**WAN**

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Files and Collections

- Files & replicas
- Logical name is arbitrary string, usually but not necessarily the filename
- In some cases with partial path (eg. for code, path in CVS repository)
- Logical name plus virtual organization (=atlas.org) defines unique logical file
- File instances include a replica number
- Zero for the **master** instance; N=locationID for other instances
- Notion of **master instance** is essential for cases where replication must be done off of a specific (trusted or assured current) instance
- Not currently supported by Globus replica catalog
- Several types of file collections
- Logical collections: arbitrary user-defined set of logical files
- Location collections: all files at a given location
- Key collections: files associated with a key or SQL query

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Distributed Catalog

- Catalog of ATLAS data at CERN, BNL (also LBNL, ANL, BU, UTA)
- Supported data stores: CERN Castor, CERN stage, BNL HPSS (rftp service), disk, code repositories, ...
- Current content: physics TDR data, test beam data, ntuples, ...
- About 200k files currently cataloged representing >6TB data
- Has run without problems with ~1.5M files cataloged
- ‘Spider’ crawls data stores to populate and validate catalogs
- Catalog entries can also be added or modified directly
- Single MySQL DB serves entire system in present implementation
- ‘MySQL accelerator’ provides good catalog loading performance over WAN; 2k files in <1sec. Sends bunched actions and initiates remotely with cgi
- Globus replica catalog ‘loader’ written for evaluation; not used yet

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Slide 7

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Slide 8

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Slide 9

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Other Metadata

- Simple user-defined metadata support: ‘keys’ (strings) can be assigned to logical files
- Will integrate with external application metadata catalogs for ‘metadata about the data’ (eg. physics generator, run type, ...)
- In ATLAS, a MySQL/phpMyAdmin based tool being developed by Grenoble for DC1
- Parenthetically... not part of the PPDG work but benefiting from it...
- New Magda derivative begun: **Hemp**, Hybrid Event store Metadata Prototype, for the RDBMS part of a ROOT/RDBMS event store
- Close ties to data signature work (‘history info’) as well as file management

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## File Replication

- Replication operations organized as user-defined tasks specifying source collection and host, transfer tool, pull/push, destination host and location, and intermediate caches
- User-specified logical file collections are replicated
- e.g. a set of files with a particular physics channel key
- Designed to support multiple file transfer tools, user-selectable, which are useful in different contexts (eg. scp for transfers ‘outside the grid’)
- In use between CERN, BNL, and among US ATLAS testbed sites
- CERN stage, Castor, HPSS  cache  **scp**  cache  BNL HPSS
- BNL HPSS or disk  cache  **gsiftp ** testbed disk
- ~270GB replicated to date
- GDMP integration just underway

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Replication Steps

- Replication steps for each logical file, coordinated via state info in DB:
- Mark as ‘processing’ in DB collection
- Find the least-cost replica instance accessible at source host (ie. disk instance preferred over MSS); stage into cache if necessary
- On stage complete, mark as available for transfer
- Independent transfer script (running on source or destination side) transfers files as they become available, and marks as available on destination side
- If final destination is MSS, transferred files are deposited in a cache, and an independent destination-side script archives them
- Caches have ‘maximum size’ to throttle to available space
- If any stage breaks, others wait until file flow resumes and then proceed
- File validation is by checking file size
- Failed transfers are re-queued

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Slide 13

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## GDMP & Magda

- Integration as a data mover underway
- Characteristics of present implementation limit scope of its application in Magda
- One root disk directory per site
- Subscription updates bring in all new data for a site
- File collections not used
- LFN fixed as ‘dir/filename’ (RC constraint)
- Doesn’t catalog or directly manage files in MSS
- Write access to tmp, etc disk areas required for all GDMP users
- System state info (in files) only available locally
- Will try it initially for managed-production transfers between large centers

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Command Line Tools

- Deployed command line tools for data access:
- magda_findfile
- Search catalog for logical files and their instances
- magda_getfile
- Retrieve file via catalog lookup and (as necessary) staging from MSS or (still to come) remote replication into disk cache
- Creates local soft link to disk instance, or a local copy
- Usage count maintained in catalog to manage deletion
- magda_releasefile
- Removes local soft link, decrements usage count in catalog, deletes instance (optionally) if usage count goes to zero
- magda_putfile
- Archive files (eg. in Castor or HPSS) and register them in catalog

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Data Grid Reference Architecture

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## DGRA Components & Magda

| Component | What’s in Magda | Application of Grid Toolkit items |
| --- | --- | --- |
| Planner | Task definitions | DAGman, SQL service |
| Executor | Scripts coordinated via MySQL for multi-stage replication and driven by task definition | DAGman, remote execution |
| Catalog services | MySQL with schema interfaces in perl, C++, Java | SQL service |
| Info services | ditto | ditto |
| Policy | Built in ‘least cost’ behaviour; command options | ? |
| Security | MySQL security, htpasswd | GSI as used in grid tools |
| Monitoring | Web presentation of MySQL-maintained info | Relevant monitoring tools that emerge |
| Replica management | MySQL  | RC, RM, GDMP |
| Reliable transfer service | Transfer script driven by task def’n | gsiftp, GDMP, reliable file transfer |
| Storage resources | Scripts using command line tools for disk, HPSS, Castor, stagein | HSM |

**Application of GT: ****green = currently used or in progress****, brown = currently in mind to be looked at and tried out**

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002

## Near Term Activity

- Application in DC0 (deployed)
- File management in production; replication to BNL; CERN, BNL data access
- Interface with Grenoble application metadata catalog
- GDMP integration - ready for DC1 (in progress)
- Application in DC1 (beginning ~March)
- As DC0, but add replication and end-user data access at testbed sites
- Interface with hybrid ROOT/RDBMS event store
- Evaluation/use of other grid tools: Globus RC, SQL service, HSM, ...
- Athena (ATLAS offline framework) integration

- Torre Wenaus, BNL          PPDG ATLAS

- JLAB, Jan 2002