# Presentation: 406528

## James Annis
Gabriele Garzoglio
Peretz Partensky
Chris Stoughton

- Gabriele Garzoglio
- Peretz Partensky
- Chris Stoughton

-  The Experimental Astrophysics Group, Fermilab

**The** **Terabyte Analysis Machine** **Project**

-  data intensive computing in astronomy

## TAM Design

- The TAM is a compact analysis cluster that is designed to
- Bring compute power to bear on large datasets.
- Make high I/O rate scans through large data sets
- Compute power brought to large data sets 		(Hardware)
- 10 processor Linux cluster
- High memory (1 Gig RAM/node) and local disk (140 Gig/node)
- SAN to Terabyte of global disk (FC network and HW RAID)
- Global File System (9x faster read than NFS; HW limited)
- High I/O rate scans 					(Software)
- The SDSS science database SX
- The Distance Machine framework

**Notes:**
There exists a plot showing the speed of GFS relative to NFS

## TAM Hardware

- Fast Ethernet Switch

- Gigabit Uplink To Slow Access Data

- Fibre Channel Switch

- 1 Terabyte of Global Disk

- 5 Compute Nodes

- 0.5 Terabyte of Local Disk

- April 2001

## The Terabyte Analysis Machine

- System integrator
- Linux NetworX
- ACE cluster control box
- Compute Nodes
- Linux NetworX
- Dual 600 MHz Pentium III
- ASUS motherboard
- 1 Gig RAM
- 2x36 Gig EIDE disks
- Qlogic 2100 HBA
- Ethernet
- Cisco Catalyst 2948G

- Fibre Channel
- Gadzoox Capellix 3000
- Global Disk
- DotHill SanNet 4200
- Dual Fibre Channel controllers
- 10x73 Gig Seagate Cheetah SCSI disk
- Software
- Linux 2.2.19
- Qlogic drivers
- GFS V4.0
- Condor

## GFS: The Global File System

- Sistina Software (ex-University of Minnesota)
- Open source (GPL;  now Sistina Public License)
- Linux and FreeBSD
- 64-bit files and file system
- Distributed, server-less metadata
- Data synchronization via global, disk based locks
- Journaling and node cast-out
- Three major pieces:
- The network storage pool driver (on nodes)
- The file system (on disk)
- The locking modules (on disk or ip server)

## GFS Performance

- Test setup
- 5 nodes
- 1 5-disk RAID
- Results
- RAID limited at 95 Mbytes/sec
- at >15 threads, disk head move limited
- Linear rate increase before hardware limits
- Circa 9x faster than NFS

## Fibre Channel

- Fibre Channel hardware has performed flawlessly, no maintence
- Qlogic hardware bus adaptors (single channel)
- Gadzoox Capellix Fibre Channel Switch
- One port per node
- Two ports per raid box
- Dot Hill hardware raid system, with dual FC controllers
- Hardware bus adaptor ~$800/node
- Switches ~$1000/port
- HBA shows up as a scsi device (/dev/sda) on Linux
- HBA has driver code and firmware code
- Must down load driver code and compile into kernel
- We haven't explored fibre channel's ability to connect machines over km’s.

## Global File System I

- We have never lost data to a GFS problem
- Untuned GFS cleary outperformed untuned NFS
- Linux kernel buffering an issue (“but I edited that file...”)
- Sistina mailing list very responsive
- Must patch Linux kernel (2.2.19 or 2.4.6)
- Objectivity doesn’t behave on GFS. One must duplicate federation files for each machine that wants access.
- GFS itself is on the complicated side, and is unfamiliar to sys-admins.
- We haven't explored GFS machines as file servers.

## Global File System II

- Biggest issues are node death, power on, and IP locking
- GFS is a journaling FS; who replay journal?
- STOMITH: shoot the other machine in the head
- Needs to be able to power cycle nodes so that other nodes can replay journal
- Power on:
- /etc/rc.d/init.d scriptable
- We have never survived a power loss without human intervention at power up.
- IP locking
- DMEP is the diskless protocol, very new; 2 vendor support
- Memexpd is an ip lock server
- Allow any hardware for disk
- Single point of failure and potential throughput bottleneck

## Disk Area Arrangement

- Data area on /GFS
- Home area on /GFS
- Standard products and UPS available from NFS
- Normal user areas (desktops) available from NFS
- /GFS area not NFS shared to desktops
- Large amounts of local disk (ide disk) available on each node, not accesible from other nodes

## The Data Volume

- 15 Terabytes of corrected frames (2-d map)
- 4 Terabytes of atlas images, binned sky and masks
- 1 Terabyte of complex object catalogs
- 120+ attributes
- Radial profiles
- Links to atlas image, spectra
- 0.1 Terabyte of spectra catalogs (3-d map)

## Summary

- Astronomy faces a data avalanche
- We are exploring datawolves, clusters aimed at large datasets.
- TAM is an University class* analysis datawolf.
- Hardware
- Server-less data sharing amongst cluster nodes
- Software
- The Distance Machine Framework
- ANN algorithm library linked to SX
- SX re-indexing and re-clustering optimally for ANN
- Corba data serving

- *In Fermilab terms, “trailer class”

- A z=0.2 cluster of galaxies

## Links

- _[http://www-tam.](http://www-tam.fnal.gov/)__[fnal](http://www-tam.fnal.gov/)__[.](http://www-tam.fnal.gov/)__[gov](http://www-tam.fnal.gov/)_
- _[http://projects.](http://projects.fnal.gov/act/kdi.html)__[fnal](http://projects.fnal.gov/act/kdi.html)__[.](http://projects.fnal.gov/act/kdi.html)__[gov](http://projects.fnal.gov/act/kdi.html)__[/act/](http://projects.fnal.gov/act/kdi.html)__[kdi](http://projects.fnal.gov/act/kdi.html)__[.html](http://projects.fnal.gov/act/kdi.html)_
- _[http://](http://sdss.fnal.gov:8000/~annis/astroCompute.html)__[sdss](http://sdss.fnal.gov:8000/~annis/astroCompute.html)__[.](http://sdss.fnal.gov:8000/~annis/astroCompute.html)__[fnal](http://sdss.fnal.gov:8000/~annis/astroCompute.html)__[.](http://sdss.fnal.gov:8000/~annis/astroCompute.html)__[gov](http://sdss.fnal.gov:8000/~annis/astroCompute.html)__[:8000/~annis/](http://sdss.fnal.gov:8000/~annis/astroCompute.html)__[astroCompute](http://sdss.fnal.gov:8000/~annis/astroCompute.html)__[.html](http://sdss.fnal.gov:8000/~annis/astroCompute.html)_

## The Sloan Digital Sky Survey

- Dedicated telescope
- Extremely wide field optics
- Open-air, no dome design
- The imaging camera
- 138 Megapixels
-  5 band passes
-  2.5 degrees field of view
- The twin spectrographs
- 660 fibers
- 3 degree field of view
-  high resolution (0.3 nm)
- 400-900 nm

## SDSS Science Aims

- SDSS Spring 2000

- Virgo Consortium Hubble Volume

- We will map the local universe (left) better than the numerical simulations (right) can predict.