# Presentation: 013096

## Status of Data Handling For Analysis

- _Outline:_
-  The SAM Project
-  Current Operations
-  Future plans
-  Some Tips
-  Shifters needed
-  Won't mention reprocessing, specific remote issues

- Adam Lyon (FNAL/CD/DØ)
- February 2004 Collaboration Meeting

## What is SAM?              More than just a file server!

- Data handling system for Run II DØ
- SAM manages file storage
- Data files are stored in tape systems at FNAL and elsewhere (most use ENSTORE at FNAL)
- Files are cached around the world for fast access
- SAM manages file delivery
- Users at FNAL and remote sites retrieve files out of file storage. SAM handles caching for efficiency
- You don't care about file locations
- SAM manages file cataloging
- SAM DB holds meta-data for each file. You don't need to know the file names to get data
- SAM manages analysis bookkeeping
- SAM remembers what files you ran over, what files you processed, what applications you ran, when you ran them and where
- Designed for PETABYTE (1015) sized experiment datasets (that's us)!

**Notes:**
SAM stands for "Sequential data access via meta-data"

## Who is working on Data Handling?

-  SAM Project Co-leaders
- Wyatt Merritt (DØ), Rick St. Denis (CDF)
-  SAM Project Technical Leaders
- Sinisa Veseli (DØ), Rob Kennedy (CDF)
-  SAM Contributors:
- CCF: Andrew Baranovski, Gabriele Garzoglio, Igor Terekhov
- CEPA: Carmenita Moore, Steve White (0.5 FTE)
- CDF: Randy Herber, Art Kreymer, Stefan Stonjek (GS)
- DØCA: Lauri Loebel Carpenter, Robert Illingworth, Adam Lyon
-  Others:	D0-primary (Joe Boyd, ...);	DB Support (Julie Trumbo, Steve Kovich, ...)

**Notes:**
CCF stands for Computing and Communications Fabric (SAM part is under "Middleware Applications")
CEPA stands for Computing and Engineering for Physics Applications

## SAM-GRID Projects    What are all those people doing?

**Active Subprojects:** C++ API, DBServer, JIM, H Stream Reco for CDF, Caching, Chains&Links, CDF DFC, Test Harness, Linux deploy of DBServers, Config Man
**Planned Subprojects:** Request system, Autodest, Further monitoring (MIS)
**Related Subprojects:** d0tools, SBIR II, Condor mods, workflow packages for CDF & D0, Authorization & Accounting
**Recently completed Subprojects:**  Python API, V5.1 Schema Design, Batch Adapter, D0 Online dcache TDP, 1st Gen Monitoring Tools, Data Dimensions Grammar
- Lots of people are working very hard.
- SAM being used by CDF and MINOS. CMS examining components.

## Current SAM Configuration (user facilities) at FNAL

- central-analysis
- The d0mino station; very large central cache [10 TB]; efficient for pick events; will disappear
- cab
- Main analysis farm; 167 nodes; station runs on d0mino; will eventually merge with cabsrv1
- cabsrv1
- "New Experimental Cab"; 160 nodes; station runs on linux server; newer machines (faster CPU, more memory, more cache disk, GB ethernet, "better" PBS); will grow
- clued0
- Desktop cluster; 60 nodes; will grow
- New Linux Fileservers
- 50 TB for central cache! Are present, but not being used yet.

## Experimentation

- There are many station parameters to tune
- Maximum parallel transfers
- Maximum concurrent enstore requests
- Configuration of cache disks
- ...
- We're moving away from d0mino to Linux
- How robust are these linux machines?
- How many "pmasters" can they run?
- How many concurrent file transfers can they handle?
- Running test harness on a small cluster to explore SAM parameter space

## Sam Statistics          Files delivered by month

- 1999

- 2000

- 2001

- 2002

- 2003

**Run II Begins**

## SAM Statistics        How much is SAM being used?

***9000 Projects!***

***233 Different Users!***

**Data from early January 6 until February 24**

## SAM Statistics          How much data has been analyzed?

***~500K Files!***

***~1%***

## Problems

- Hardware Problems
- CRC Errors (linux disks only)
- NOACCESS Tapes
- Cannot access SAM DB (d0ofprd1 == d0ora2)
- Failover problems
- RAID Array Nightmare
- Network quarantine
- Nightmare weeks ~ 15% efficiency loss, but we doubled computing capacity too!
- CAB/Clued0 machines problems

- Software Problems
- Rare bugs in *sam_station *and *sam_user_api*
- Problems with PBS
- Nameserver saturation (fixed, will be better in new RECO)
- Missed UDP packets (fixed in latest release)
- Pilot Error
- Calibration group filling up DB "forward archive"
- Vendor bungled DB RAID Array upgrade

- SAM relies on a working hardware and software
- ... but SAM can recover from unforeseen problems (losing all projects is extremely rare)

## Reducing problems

- "Watcher scripts" look for nodes going down and automatically alerts SAM stations accordingly   						(Robert Illingworth)
- CAB, CABSRV1 and Clued0 are watched
- Stops SAM from retrieving files from a down node
- Prevents data delivery errors
- Restarts stagers on nodes that came back to life
- Prevents "Cannot deliver files to node" errors
- Greatly reduces shifter load
- samTV displays health of FNAL SAM stations(See _http://www-clued0.fnal.gov/~sam/samTV/current_ (_[click](http://www-clued0.fnal.gov/~sam/samTV/current)_))(_[cdf](http://ncdf151.fnal.gov:8520/samTV/current/samTV.html)_)

## SAM Statistics          How much data has been analyzed?

*Raw*

*Thumbnails + ...*

***256 TB!***

***8.3 Billion Events!***

**Data from early January 6 until February 24**

## ENSTORE Statistics            We do not lose data!

**0.6 Petabytes** in tape storage!

- Only 5 files unrecoverable (5 GB total;  8ppm loss)  !!!
- One of them was RAW file

## SAM Statistics                  How often do we use tapes?

- Local cache:
- Used for repeat file accesses
- Can see the larger local cache on *cabsrv1*
- All files on *Clued0 *are routed through head node with large cache
- Prevents DAB and Trailer network overload
- D0mino (central) cache:
- Looks like we're not using it enough
- Can we decrease our tape usage?

**Notes:**
Note that d0mino use of its cache disk is not counted as a file transfer, so it does not show up at all on the plot (should be local)

## Future central caching schemes

- The new linux file servers will have huge SAM caches
- How do we fill them?
- Manual population
- Physics Groups decide how to populate central cache
- Requires management (see how well we've done so far?)
- Popular files may not be cached
- Automatic population
- Route all files through head node(s) with big caches (long cache lifetime)
- Popular files automatically live a long time in the cache
- Files that turn unpopular automatically leave
- Head nodes are bottlenecks; first access to file may be slow but subsequent accesses should be fast
- Need to experiment to determine best scheme

## SAM Statistics          How long do we wait for data?

- Time between *Request Next File* and*Open File*
- For CAB and CABSRV1
- 50% of enstore transfers occur within 10 minutes.
- 75% within 20 minutes
- 95% within 1 hour
- For CENTRAL-ANALYSIS and CLUED0
- 95% of enstore transfers within 10 minutes

| Station | CAB | CABSRV1 | CLUED0 | CA |
| --- | --- | --- | --- | --- |
| % no wait | 30% | 40% | 38% | 18% |

**Notes:**
Influenced by:
Transfer throttle on stager
Transfer throttle on station
Network speed & load
Enstore queue length, tape transfer time, network access to enstore
Influenced by:
Transfer throttle on stager
Transfer throttle on station
Network speed & load
Enstore queue length, tape transfer time, network access to enstore

## Box and Whisker Plots

## SAM Statistics                  Processing time per file

## Status                Where are we now and what's coming soon

- Aside from rare glitches, SAM is stable and survives heavy pounding
- Storing files into SAM seems to be more problematic than reading from SAM
- Need monitoring (none right now)
- Better handling of meta-data
- Better documentation
- Better documentation needed
- Working to move "pmaster" onto worker nodes
- More efficient SAM DB Schema coming soon
- DB Servers Rewrite nearly complete - more efficient and easier to add functionality

## SAM Tips and Reminders

- Support model
- Send SAM problems to d0sam-admin@fnal.gov
- d0sam-users@fnal.gov are for announcements
- Shifters are "first responders" on 16 hrs / day
- SAM on-call experts backup the shifters
- Helpdesk (x2345) /d0-primary for hardware problems Helpdesk open only during FNAL business hoursAlways cc:d0-primary@fnal.gov
- Don't copy directly from /pnfs - won't work!
- Create catch-up datasets **after** your project is done
- SAM delivers files to all projects that ask for them (files are not "load balanced" among projects)

## Shifters                                    Will work for data

- You can help by becoming a SAM shifter
- SAM shifters now get full shift credit
- First line of defense in SAM monitoring
- SAM "Helpdesk"
- Shifts aren't as scary as they look...
- There are plenty of experts (some are on call) to help you
- While the documentation is somewhat disorganized, there's a wealth of information
- SAM shifters can't break the detector ($$$)
- Send mail to Daria Zieminska if you are interested (daria@indiana.edu)

## EXTRA SLIDES FOLLOW

## Top Users (Jan 6, 2004 - Feb 24, 2004)

- Top users by # of projects

- Top users by consumed files

## SAM Grows

- Started at DØ
- CDF is beginning to use it (still testing)
- SAM will completely replace their data handling system
- MINOS
- Will use SAM for their data handling
- Have interesting use cases (must synchronize use of multiple files).
- CMS
- Examining SAM components

## SAM Statistics                      Process wait times

- File Source

## Some SAM buzzwords

- Dataset Definition
- A set of requirements to obtain a particular set of files
*e.g.* data_tier thumbnail and run_number 181933
- Datasets can change over time
- More files that satisfy the dataset may be added to SAM
- Snapshot
- The files that satisfy a dataset at a particular time (e.g. when you start an analysis job)
- Snapshots are static
- Project
- The running of an executable over files in SAM
- Consists of the dataset definition, the snapshot from that dataset definition, and application information
- Bookkeeping data is kept - how many files did you successfully process, where did your job run, how long did it take

## SAM Statistics                  Processing time per file

## SAM Statistics                 What are people doing?