# Presentation: 817372

## FAST-OS: Petascale Single System Image

- Jeffrey S. Vetter (PI)
- Nikhil Bhatia, Collin McCurdy, Phil Roth, Weikuan Yu
**Future Technologies Group **** ****Computer Science and Mathematics Division**

## PetaScale single system image

- What?
**Make a collection of standard hardware look like one big machine, in as many ways as feasible**
- Why?
**Provide a single solution to address all forms of clustering**
**Simultaneously address availability, scalability, manageability, ****and usability**
- Components
**Virtual cluster using contemporary hypervisor technology**
**Performance and scalability of parallel shared root file system**
**Paging behaviors of applications, i.e., reducing TLB misses**
**Reducing “noise” from operating systems at scale**

## Virtual clusters using hypervisors

**Physical nodes run Hypervisor in the privileged mode.**
**Virtual machines run on top of the hypervisor.**
**Virtual machines are compute nodes hosting user-level distributed memory (e.g., MPI) tasks.**
**Virtualization enables dynamic cluster management via live migration of compute nodes.**
**Capabilities for dynamic load balancing and cluster management ensuring high availability.**

**VM**

**VM**

**VM**

**VM**

**VM**

**VM**

**VM**

**VM**

**VM**

**Task **
**0**

**Task **
**1**

**Hypervisor (Xen)**

**Hypervisor (Xen)**

**Hypervisor (Xen)**

**Physical Node 0**

**Physical Node 1**

**Physical Node 15**

**Task **
**3**

**Task **
**4**

**Task **
**5**

**Task **
**7**

**Task **
**60**

**Task **
**61**

**Task **
**63**

## Virtual cluster management

**Single system view of ****the cluster**
**Cluster performance diagnosis**
**Dynamic node addition and removal ensuring high availability **
**Dynamic load balancing across the entire cluster**
**Innovative load balancing schemes based on distributed algorithms**

**Intel x86**

**Xen**

**Virtual Machine**

**Virtual Application**

**Xen**

**Intel x86**

## Virunga—virtual cluster manager

**Virunga Client**

**Performance**
**Data**
**Presenter**

**Performance**
**Data Parser**

**System Administration Manager**

**Monitoring station (Login Noded)**

## Paging behavior of DOE applications

**With performance counters and simulation:**
**Show that the HPCC benchmarks, meant to characterize the memory behavior of HPC applications, do not exhibit the same behavior in the presence of paging hardware as scientific applications of interest to the Office of Science**
**Offer insight into why that is the case**
**Use memory system simulation to determine whether large page performance will improve with the next generation of Opteron processors **

- Goal:  Understand behavior of applications in large-scale systems composed of commodity processors

## Experimental results (from performance counters)

- Experimental results (from performance counters)

**Performance: trends for large vs small nearly opposite**
**TLBs: significantly different miss rates**

- HPCC Benchmarks

- Applications

**1.0**

**0**

**Cycle**

**MPIRANDOM**

**STREAM**

**RANDOM**

**FFT**

**DGEMM**

**PTRANS**

**HPL**

**MPIFFT**

**Tb_misses**

**MPIRANDOM**

**STREAM**

**RANDOM**

**FFT**

**DGEMM**

**PTRANS**

**HPL**

**MPIFFT**

**HYCOM**

**AMBER**

**1.0**

**0**

**CAM**

**POP**

**GTC**

**GYRO**

**LMP**

**Cycle**

**Tb_misses**

**CAM**

**POP**

**HYCOM**

**GTC**

**GYRO**

**LMP**

**AMBER**

**L2tlb_hits**

**MPIRANDOM**

**STREAM**

**RANDOM**

**FFT**

**DGEMM**

**PTRANS**

**HPL**

**MPIFFT**

**L2tlb_hits**

**CAM**

**POP**

**HYCOM**

**GTC**

**GYRO**

**LMP**

**AMBER**

## Reuse distances (simulated)

- HPCC Benchmarks

- Applications

- Reuse distances (simulated)

**Patterns are clearly significantly different...**

**pctg**

**Large**
**Small**

**RANDOM**

**50**
**40**
**30**
**20**
**10**
**0**

**70**
**60**
**50**
**40**
**30**
**20**
**10**
**0**

**pctg**

**Large**
**Small**

**STREAM**

**pctg**

**Large**
**Small**

**PTRANS**

**60**
**50**
**40**
**30**
**20**
**10**
**0**

**pctg**

**Large**
**Small**

**HPL**

**100**
**90**
**80**
**70**
**60**
**50**
**40**
**30**
**20**
**10**
**0**

## Paging conclusions

**HPCC benchmarks are *****not***** representative of paging behavior of typical DOE applications.**
**Applications access many more arrays concurrently.**
**Simulation results (not shown) indicate the following:**
**New paging hardware in next-generation Opteron processors *****will***** improve large page performance.**
**Performance near that with paging turned off.**
***However*****, simulations also indicate that the mere presence of a TLB is likely degrading performance, whether paging hardware is on or off.**
**More research into implications of paging, and of commodity processors in general, on performance of scientific applications is required.**

## Parallel root file system

- Goals of study
**Use a parallel file system for implementation ****of shared root environment**
**Evaluate performance of parallel root file system **
**Evaluate the benefits of high-speed interconnects**
**Understand root I/O access pattern and potential scaling limits**
- Current status
**RootFS implemented using NFS, PVFS-2, Lustre, and GFS**
**RootFS distributed using ramdisk via etherboot**
**Modified mkinitrd program locally**
**Modified to init scripts to mount root at boot time **
**Evaluation with parallel benchmarks (IOR, b_eff_io, NPB I/O)**
**Evaluation with emulated loads of I/O accesses for RootFS**
**Evaluation of high-speed interconnects for Lustre-based RootFS**

## Example results: Parallel benchmarksIOR read/write throughput

## Performance with synthetic I/O accesses

- Startup with different images

- Time cvs co mpich2

- Time diff mpich2

- Time tar jcf linux-2.6.17.13

- Time tar jxf linux-2.6.17.13.tar.bz2

**16K**
**16M**
**1G**

**Lustre-TCP-Time**
**Lustre-TCP-CPU**
**Lustre-IB-Time**
**Lustre-IB-CPU**

**Lustre-TCP-Time**
**Lustre-TCP-CPU**
**Lustre-IB-Time**
**Lustre-IB-CPU**

## Contacts

- Jeffrey S. Vetter
- Principle Investigator
- Future Technologies Group Computer Science and Mathematics Division (865) 356-1649
- vetter@ornl.gov
- Nikhil Bhatia	Phil Roth
- (865) 241-1535	(865) 241-1543
- bhatia@ornl.gov	rothpc@ornl.gov
- Collin McCurdy	Weikuan Yu
- (865) 241-6433	(865) 574-7990
- cmccurdy@ornl.gov	wyu@ornl.gov

-    Vetter_petassi_SC07