# Presentation: 800197

## Tier-1 Status and Upgrade Plans

-                               Michael Ernst

## Tier-1 Activities (as part of the UF Activities)

- Most of the US-CMS activities over the next 12 months are being driven by preparations for DC04 and the Physics TDR that follows
- Running event production for the challenge (DPE Production)
- Developing the Grid and Facilities infrastructure needed to complete the challenge
- Includes increasing facility capabilities through procurement
- Grid capabilities through Grid2003 and LCG-1 efforts
- Increasing the capability of User Analysis Facility
- Facilities and environment improvements

## Tier-1 Facilities Capacity

- Offsite Contribution to CMS Computing Capacity in 2003
- Total:                         2000 kSI2000
- US Tier-1 Facility:      200 kSI2000 (10%)
- Currently installed (AMD based systems only):   74 kSI2000

## kSI2000*Months

**kSI2000*Months**

## Current Resources

- Production Grid
- 38 Worker Nodes (76 CPUs)
- Dual AMD 1900 1U Nodes (RH 7.3)
- 2 Servers
- 4*700MHz ea., 200GB Disk ea.
- Development Grid
- 6 Worker Nodes (12 CPUs)
- 1 Server
- User Analysis Farm
- 18 Worker Nodes (36 CPUs)
- 10 Dual AMD 1900 1U Nodes (RH 7.3)
-   8 Dual PIII Nodes (RH 6.2)
- 1 Server
- Misc.
- 10 Dual AMD Nodes for Storage Evaluation
- 32 PIII Nodes (to be retired)
- Storage
- 7 Disk Servers, 1.5 TB ea. (5 used as dCache Pool Nodes)

## Network attached Storage

- Node

- Node

- Node

- Node

- Node

- Node

- Node

- Node

- Node

- Node

- Front-end Node

- Front-end Node

- dCache

- ENSTORE

- Web servers

- Db servers

- Dynamic partitioning and config. the farm

- CISCO
-  6509

- User Interactive

- Production

**Notes:**
dCache: making a multi-terabyte server farm look like one coherent and homogeneous storage system.
Rate adaptation  between the application and the tertiary storage resources. 
Optimized usage of expensive tape robot systems and drives by coordinated read and write requests. Use dccp command instead of encp!
No explicit staging is necessary to access the data (but prestaging possible and in some cases desirable).
The data access method is unique independent of where the data resides.
High performance and fault tolerant transport protocol between applications and data servers
Fault tolerant, no specialized servers which can cause severe downtime when crashing. 
Can be accessed directly from your application (e.g. root TDCacheFile class).
Can be used as scalable file store without HSM.
Deployed as such in the tier 2 sites. 

## ...

- NAS Head

- Cisco
-  3750

-   dCache
- Pool Node

- 1

- 16

- Production Cell

- FC Switch (64 Ports)

- FNAL
-   Site
-    Net

- link trunking
- (n* 1GigE)

- Infortrend
-     A16F

- ~ 4 TB/ea

-    dCache Pool Nodes

- User Area

- Enstore

- Production Cell

- Analysis Cell

## Upgrade

- 76 Worker Nodes (Dual 1 U Servers)
- 120 CPUs for Production (99 kSI2000)
- 32 CPUs for Analysis (26 kSI2000)
- SMC X5DPA w/Intel 7501, 533 MHz Frontside Bus
- 2 * 2.4 GHz XEON Processors
- 2GB Memory
- 2 Disk Drives (40 + 120 GB)
- 10/100/1000 Ethernet on-board
- 17 Server Nodes
- Intel SE7501 Dual XEON E-ATX Server Board
- PCI-X 64 bit/100 MHz
- 2 * 2.8 GHz XEON Processors
- 2GB Memory

## Data Transfer Service and Cataloguing 

- Data Transfer Service and Cataloguing

**Fermilab (Tier1)**
**Caltech (Tier2)**
**San Diego (Tier2)**
**Florida (Tier2)**
**CERN (Tier0)**

**UCSD**

**Florida**

**Caltech**

**Fermilab**

**CERN**

- SRB

- GridFTP/(SRM)

**dCache/Enstore**

**Castor**

## dCache Components

- Pool Manager

- I/O Door Nodes

- dCap

- (K)Ftp

**(Krb5,ssl)**

- Http

- File Name Space

- Database

- NFS Server

- pnfs

- File Name Space

- Provider

*OSM*

*Enstore*

*TSM *

- Admin Doors

*dCache Components*

*dCache Components*

## Storage

- We have been looking for a highly available, reasonably performing storage solution to provide analysis space for users
- Users now make requests that cannot be easily accommodated by our existing facilities
- Have looked at a number of NAS and SAN systems
- Investigated three possible storage solutions
- A NAS system for common space and user space and 3ware RAID arrays for large scale storage
- A NAS system for common space and user space and separate storage modules attached to servers
- A large SAN system that combines the common and user space with the large scale storage
- This is a very flexible and robust solution

## Upgrade - Storage

- 5 Infortrend A16F Serial ATA Disk Arrays (~20TB)
- 2 * Gbps FC Interfaces
- 16 * 300GB SATA Drives (1.5 Gbps compatible w/hot swap)
- RAID level 0, 1, 1 (0+1), 3, 5, 10, 30, 50 (multiple RAID selections)
- 2 redundant hot-swappable Power Supplies
- 1 LSI LOGIC E5600 (~3.5 TB)
- 2 RAID Controllers
- 8 * 2Gbps FC Host connections (800MB/s max.)
- 4 * 2Gbps FC Drive connections (800MB/s max.)
- Number of Drives supported     14 – 224 (32.7 TB max.)
- max. 795MB/s (or 53.200 I/O ops/s sustained)
- NAS Heads based on IBRIX Cluster File System running on Dell 2650 Servers exporting FSs via NFS
- Beta software installed at FNAL
- Expect production release by December

## Upgrade - Networking

- 5 Cisco 3750 Modular Switches
- 24 Ethernet 10/100/1000 ports and 4 SFP uplinks
- Stackable (up to 9 units, 32-Gbps high-speed stacking bus)
- Single image, full IOS
- QLOGIC SANbox 2-64 FC Switch
- 64  2Gbps FC Ports (expandable by 8 port blades)
- Up to 256 Gbps (full duplex) Fabric Aggr. Bandwidth (412MB/s P-to-P)

## Projects @ Tier-1 Facility

- Installation and Configuration Management
- ROCKS
- VOLDEMORT (passing config., password file, push RPMs, scripts, files)
- YUM
- Storage Technology Evaluation (jointly with FNAL CD)
- Primarily NAS
- Wide Area Network Lab
- Ultra Scale Networking (n * Gbps)
- Grid Storage Element / Replica Management Service Development
- Jointly with SRM Collaboration and LCG

## The Functional View

- Wide Area
- Access

## Replica ManagementArchitecture

## Replica Management- A Proposal

## Generalizing the Concept: Need to Standardize APIs

## FNAL => CERN: 1 TB in 20 hours

- Network Throughput on Server Interface at Fermilab

-  Network Throughput at FNAL Border Router (sorted by experiments/groups)

*FNAL => CERN: 1 TB in 20 hours*

## People keeping the wheels spinning

- Joe Kaiser (SCS)
- Lisa Giacchetti (SCS)
- Bonnie Alcorn & ISA Group
- Hans Wenzel (UAF, User Coordination)
- David Fagan
- Nickolai Kouropatkine (iVDGL, Running CMS Production)
- Anzar Afaq (PPDG, Development & Running Production)
- ... and many more