# Presentation: 178688

## Huge-Memory Systems for Data-Intensive Science

- Richard P. Mount
-  SLAC
- CHEP,  September 29, 2004

## Outline

- The Science Case
- DOE Office of Science Data Management Workshop
- Technical Issues
- Characterizing scientific data
- Technology issues in data access
- Proposal and Currently Funded Plans
- The “solution” and the strategy
- Development Machine
- Leadership Class Machine

## Slide 3

## Slide 4

## Slide 5

## Slide 6

## Slide 7

## Slide 8

## Slide 9

## Slide 10

## Slide 11

## Slide 12

## Characterizing Scientific Data

-    My petabyte is harder to analyze than your petabyte
- Images (or meshes) are bulky but simply structured and usually have simple access patterns
- Features are perhaps 1000 times less bulky, but often have complex structures and hard-to-predict access patterns

## Characterizing Scientific Data

- This proposal aims at revolutionizing the query and analysis of scientific databases with complex structure.
- Generally this applies to feature databases (terabytes–petabytes) rather than bulk data (petabytes–exabytes)

## Technology Issues in Data Access

- Latency
- Speed/Bandwidth
- (Cost)
- (Reliabilty)

## Latency and Speed – Random Access

## Latency and Speed – Random Access

## Storage Issues

- Disks:
- Random access performance is lousy, unless objects are megabytes or more
- independent of cost
- deteriorating with time at the rate at which disk capacity increases
- (Define random-access performance as time taken to randomly access entire contents of a disk)

## The “Solution”

- Disk storage is lousy and getting worse
- Use memory instead of disk  (“Let them eat cake”)
- Obvious problem:
- Factor ****** **100 in cost
- Optimization:
- Brace ourselves to spend (some) more money
- Architecturally decouple data-cache memory from high-performance, close-to-the-processor memory
- Lessen performance-driven replication of disk-resident data

## The ProposalA Leadership Class Facility for Data-Intensive ScienceandCurrently Funded Plan

## The Strategy

- There is significant commercial interest in an architecture including data-cache memory
**But**: from interest to delivery will take 3-4 years
**And**: applications will take time to adapt not just codes, but their whole approach to computing, to exploit the new architecture
**Hence**: two phases
- Development phase (years 1,2,3)
- Commodity hardware taken to its limits
- BaBar as principal user, adapting existing data-access software to exploit the configuration
- BaBar/SLAC contribution to hardware and manpower
- Publicize results
- Encourage other users
- Begin collaboration with industry to design the leadership-class machine
- “Leadership-Class” Facility (years 3,4,5)
- New architecture
- Strong industrial collaboration
- Facility open to all

## Development MachineDesign Principles

- Attractive to scientists
- Big enough data-cache capacity to promise revolutionary benefits
- 1000 or more processors
- Processor to (any) data-cache memory latency < 100 s
- Aggregate bandwidth to data-cache memory > 10 times that to a similar sized disk cache
- Data-cache memory should be 3% to 10% of the working set (approximately 10 to 30 terabytes for BaBar)
- Cost effective, but acceptably reliable
- Constructed from carefully selected commodity components

## Development MachineDesign Choices

- Intel/AMD server mainboards with 4 or more ECC dimm slots per processor
- 2 Gbyte dimms ($635 each)
- 4 Gbyte dimms ($7,000 each) too expensive this year
- 64-bit operating system and processor
- Favors Solaris and AMD Opteron
- Large (500+ port) switch fabric
- Large IP switches are most cost-effective
- Use of ($10M+) BaBar disk/tape infrastructure, augmented for any non-BaBar use

## Development MachineDeployment – Proposed Year 1

- Memory Interconnect Switch Fabric

- 650 Nodes, each  2 CPU, 16 GB memory

- Storage Interconnect Switch Fabric

- > 100 Disk Servers
- Provided by BaBar

- Cisco/Extreme/Foundry

- Cisco/Extreme/Foundry

## Development MachineDeployment – Currently Funded

- Development MachineDeployment – Currently Funded

- Cisco Switch

- Data-Servers   64-128 Nodes, each  Sun V20z, 2 Opteron CPU, 16 GB memory
- Up to 2TB total Memory
- Solaris

- Cisco Switch

- Clients   up to 2000 Nodes, each  2 CPU, 2 GB memory
- Linux

## BaBar/HEP Object-Serving Software

- AMS and XrootD (Andy Hanushevsky/SLAC)
- Optimized for read-only access
- Make 100s of servers transparent to user code
- Load balancing
- Automatic staging from tape
- Failure recovery
- Can allow BaBar to start getting benefit from a new data-access architecture within months without changes to user code
- Minimizes impact of hundreds of separate address spaces in the data-cache memory

## “Leadership-Class” FacilityDesign Principles

- All data-cache memory should be directly addressable by all processors
- Optimize for read-only access to data-cache memory
- Choose commercial processor nodes optimized for throughput
- Use the (then) standard high-performance memory within nodes
- Data-cache memory design optimized for reliable bulk storage
- 5s latency is low enough
- No reason to be on  the processor motherboard
- Operating system should allow transparent access to data-cache memory, but should also distinguish between high-performance memory and data-cache memory

## “Leadership-Class” FacilityDesign Directions

- ~256 terabytes of data-cache memory and ~100 teraops/s by 2008
- Expandable by factor 2 in each of 2009,10,11
- Well-aligned with mainstream technologies but:
- Operating system enhancements
- Memory controller enhancements (read-only and coarse-grained locking where appropriate)
- Industry partnership essential
- Excellent network access essential
- (SLAC is frequently the largest single user of *both* ESNet and Internet 2)
- Detailed design proposal to DOE in 2006

## “Leadership-Class” Facility

- Memory Interconnect Fabric

- Disk and Mass-Storage Hierarchy

- SMP systems
- 100 teraops/s> 10,000 threads

- ~ 256 terabytes data-cache memory

## Summary

- Data-intensive science increasingly requires low-latency access to terabytes or petabytes
- Memory is one key:
- Commodity DRAM today (increasing total cost by ~2x)
- Storage-class memory (whatever that will be) in the future
- Revolutions in scientific data analysis will be another key
- Current HEP approaches to data analysis assume that random access is prohibitively expensive
- As a result, permitting random access brings much-less-than-revolutionary immediate benefit
- Use the impressive motive force of a major HEP collaboration with huge data-analysis needs to drive the development of techniques for revolutionary exploitation of an above-threshold machine.