# Presentation: 108865

## Integration ofLarge File Store and HSM

*Michael Ernst*
*Fermilab*
*May 29, 2003*

## The Place ...

- The Machine ...

- The Detectors ...

## A Stack of Accelerators

## Slide 4

## particle passes through the detector, collides with atoms and kicks out electrons.       The electrons are attracted to the nearest positive wire.       The electric pulse on the wire is amplified and sent to a computer.       From the position of the wire and the arrival time of the signal, the computer reconstructs the position of the collision(s). 

**particle** passes through the detector, collides with atoms and **kicks out electrons**.       The **electrons** are attracted to the nearest positive **wire**.       The **electric pulse** on the wire is amplified and sent **to a computer**.       From the **position** of the wire and the **arrival time** of the signal, the computer **reconstructs** the **position** of the collision(s).

## The CMS Experiment

- CMS has adopted a distributed computing model to perform data analysis,
- event simulation, and event reconstruction in which two-thirds of the total
- computing resources are located at regional centers.

- The unprecedented size of the LHC collaborations and complexity of the

## Shared Resources (ATLAS and CMS) 

| Resource | CERN (Tier0+1) | Tier1+2 (not at CERN)
 |
| --- | --- | --- |
| Tape Storage [PB] | 11.5 | 17 |
| Disk Space [PB] | 2.3 | 8.1 |
| CPU Capacity 
[M SI95] | 2.4 | 5 |

## Size of US CMS Tier1 Regional Center 

| Tape Storage
(ESD/AOD/Raw/TAG/Calib.) | 1.02 PB |
| --- | --- |
| Disk for
AOD/TAG/user/cache/... | 0.8 PB |
| CPU for simulation, data,
reprocessing, analysis, data
serving | 167k SI95’ |

## Data Handling Problems and Opportunities

- Data volumes are huge, size of individual data product 1kB-1MB
- Workload are dominated by reading
- Stepwise refinements of algorithms and event selection functions leads to running over the same input dataset
- Because of the large data reduction factor in CMS physics analysis, the input datasets in such analysis efforts will represent very sparse subsets of the set of events over a period of detector running (with sparseness increases with progressing analysis effort).
- System that uses fixed partitioning of data product values over large files stages all data needed by staging all large files is too inefficient
- Leads to creation of new files (copy sparse subset of data product values into new files – could be as bad as doing it manually)

## CMS is working on ...

- CMS is currently in the process of developing the Data Model
- Data Storage and Data Access are the most demanding problems
- The choice of OS and Persistency solution can strongly influence the hardware needs (and the human resources required to support ...)
- Moving away from a Persistency Model based on OODB
- Problem mapping Objects to Files
- “New Model” should be developed with focus on optimization of the underlying storage architecture and storage technology
- Classic Filesystems at the limit of their scaling capabilities

## Requirements

- Offline (Batch) Analysis of MC Data
- Total Volume 10TB now; growing to ~ 100TB in 2005
- Subset O(1< sz [TB] <10) analyzed at given time
- > 500 simultaneous streams @ O(0.1< bw [MB/s] <1)/stream
- Interactive Analysis
- Space/Bandwidth Ratio:             2 TB / 200 – 300 MB/s
- 100 Physicists working simultaneously
- Analysis performed on Desktop w/100BaseT connection

## Interactive

- Production &
- Analysis Batch

- ~10TB
- (dCache)

- HSM

**AFS**

- user home areas
- executables

- - shared work space

## Production &

- Analysis Batch

- Interactive

- ~100TB
- (dCache)
- Farms
- Desktops

- HSM

- NAS Server

- user home areas
- executables
- shared workspace

## User access to FNAL Facilities

- Network

**Objects**

- HSM

- Large Filestore
- (Caching)

## System Architecture

**PRODUCTION**
** ****CLUSTER**
**(150 - 1500 Nodes)**

**US-CMS**
**TESTBED**

**HSM**
**(17 DRIVES, shared)**

**ESNET (OC12)**

**(DF to StarLight)**

**CISCO**
**6509**

**DCACHE (> 7 TB)**

**USER ANALYSIS**

**R&D**

** **

**NAS**

## Unified Storage Interface

**CMS-**
**specific**

- HSM

## Integration of Large Fileserver with HSM

- Required Fileserver Components
- Internal Locking, handle file states, state changes and conflicts (migration and client access)
- Separation of File Data Management and File Namespace Management (Metadata Handling)
- Ability to handle generic file metadata for any file object
- Store bitfile-id and other data in persistent relation to the file object
- Support for versioning of migrated data
- Space Management Components
- Agent looking for candidates to be migrated (control the migration/recall operations)
- Admin Interface to control the migration behavior (rule based; low/high watermark based system)

## Integration of large Fileserver with HSM

- Interface Scenarios
**Scenario 1**
- File Copy style application internal to Fileserver interfacing with HSM
- POSIX compliant I/O calls
- TCP Data Connections between Fileserver and HSM
- UDP/TCP/RPC based control connections (depending on HSM)
- Likely that requirements demand a POSIX compliant OS
- Fileserver Core needs to support following Components/Services
- IO to/from the file data to be migrated/recalled/removed
- Transparent to end-user
- File metadata is not affected by migration operation (unchanged)
- Access to Metadata related to File Object
- Strict relation between File Object and HSM Metadata is                                                             managed by the Fileserver Components
- Network programming interface (i.e. socket interface, ONC RPC)
- HSM requirements
- Control and Data Communication through standard network protocols (i.e. TCP/IP)

## Integration of large Fileserver with HSM

- Interface Scenarios
**Scenario 2**
- Application accessing the HSM runs outside the fileserver
- All Communication is done through the network.
- Data and Metadata access through the supported Network FS Protocols (i.e. NFS)
- Requirements on Fileserver
- Export of a separate namespace which allows metadata access (virtual                                                            files) and access to 'true' file object metadata (i.e. the real filesize) in contrast to the unchanged file metadata the user sees on the client system
- Ability to trigger store/restore/remove operations (i.e. through                                                              special files, file contents) and pass the result/completion code back to                                                            the fileserver
- Requirements on HSM
- Programming Interface to build custom application for HSM access  or
- Toolkit supporting the requested functionality (might require 'glue' code (i.e. shell-script, TCL, Perl etc.)