# Presentation: 189612

## Welcome to the PVFS BOF!

- Rob Ross, Rob Latham, Neill Miller
- Argonne National Laboratory
- Walt Ligon, Phil Carns
- Clemson University

## An interesting year for PFSs

- At least three Linux parallel file systems out there and in use now
- PVFS
- GPFS
- Lustre
- Many “cluster file systems” also available
- Verdict is still out on what of these are useful for what applications
- Now we’re going to complicate things by adding another option :)

## Goals for PVFS and PVFS2

- Focusing on parallel,scientific applications
- Expect use of MPI-IOand high-level libraries
- Providing our solution to a wide audience
- Ease of install, configuration, administration
- Handling and surviving faults is a new area of effort for us

## Outline

-  Shorter discussion of PVFS (PVFS1)
- Status
- Future
- Longer discussion of PVFS2
- Goals
- Architecture
- Status
- Future
- Leave a lot of time for questions!

## PVFS1 Status

- Version 1.6.1 released in the last week
- Experimental support for symlinks (finally!)
- Bug fixes (of course)
- Performance optimizations in stat path
- Faster ls
- Thanks to Axciom guys for this and other patches

## PVFS1 Future

- Community support has been amazing
- We will continue to support PVFS1
- Bug fixes
- Integrating patches
- Dealing with RedHat’s changes
- We won’t be making any more radical changes to PVFS1
- It is already stable
- Maintain it as a viable option for production

## Fault Tolerance!

- Yes, we do actually care about this
- No, it’s not easy to just do RAID between servers
- If you want to talk about this, please ask me offline
- Instead we’ve been working with Dell to implement failover solutions
- Requires more expensive hardware
- _Maintains high performance_

## Dell Approach

- Failover protection of PVFS meta server, and/or
- Failover protection of I/O nodes
- Using RH AS 2.1 Cluster Manager framework
- Floating IP address
- Mechanism for failure detection (heartbeats)
- Shared storage and quorum
- Mechanism for I/O fencing (power switches, watchdog timers)
- Hooks for custom service restart scripts

**Notes:**
Info on Cluster Manager:
Nodes in the HA setup share 3 partitions: one for the application use (start, stop scripts, working directory, etc.) and 2 (for redundancy) raw quorum partitions that store static service definitions and dynamic service state (e.g. where service is currently running) 

Shared partition has to be mounted write-through to preserve data integrity during failures

For complete protection, you need an IO fencing mechanism (to ensure that if a server is thought to be dead, it actually reboots to prevent “waking up” after some time and writing to the storage that is already controlled by the other node). CM supports watchdog timers and network power switches.

We used an ethernet network for heartbeats, but multiple heartbeat channels can be used for redundancy (e.g. serial connection in addition to ethernet)

During a failover, Cluster Manager will:
Mount partitions at correct mount points
Configure IP address for the service
Call the service start script

Polling time is configurable at the granularity of 1 second (default is 2s).

## Metadata Server Protection

**Dell PowerVault 220**

**Communication Network**

**Clients**

**Dell PowerEdge 2650**

**Dell PowerEdge 2650**

**Heartbeat Channel**

**Active**
**Meta Server**

**Shared**
**Storage**

**Standby**
**Meta Server**

**. . .**

**IO Nodes**

**Dell PE 2650**

**HA Service**

**Meta Server IP**

**Notes:**
PowerVault 220 enclosure is setup in clustering mode
PERC adapter is also setup in clustering mode (SCSI id of one of the controllers has to be changed because the two controller are on the same channel when in clustering mode)
Dell servers have redundant power supplies and dual NIC interfaces that can all be used to increase availability
Note about using PERC: we’ve used megaraid 2.x driver which is the only one that supports clustering. However, this configuration is not qualified by RH (bugs in the megaraid driver)

## Metadata and I/O Failover

- FC

**Communication Network**

**Clients**

**Dell PE 2650**

**Dell PE 2650**

**Heartbeat**
**Channel**

**Active**
**Meta Server**

**Shared FC**
**Storage**

**Standby**
**Meta Server**

**. . .**

**Dell|EMC CX600**

**Dell PE 2650**

**Dell PE 2650**

**Dell PE 2650**

**Active IO Nodes**

**HA Service**

**HA Service**

**Notes:**
RHEL 3 supports up to 16 servers (per service) that share storage, which enables us to do IO node failover. All servers can be active, or one can be reserved for standby. Meta server can be a separate service using the same storage to reduce cost.
This setup can utilize caching on the more expensive FC enclosures to improve performance
Note that in this setup all IO nodes are active, I.e. each is a standby for every other IO server

## The Last Failover Slide

- These solutions are more expensive than local disks
- Many users have backup solutions that allow the PFS to be a scratch space
- We’ll leave it to users (or procurers?) to decide what is necessary at a site
- We’ll be continuing to work with Dell on this to document the process and configuration

## PVFS in the Field

- Many sites have PVFS in deployment
- Largest known deployment is the CalTech Teragrid installation
- 70+ Terabyte file system
- Also used in industry (e.g. Axciom)
- Lots of papers lately too!
- CCGrid, Cluster2003
- Researchers modifying and augmenting PVFS, comparing to PVFS
- You can buy PVFS systems from vendors
- Dell, HP, Linux Networx, others

## End of PVFS1 Discussion

- Questions on PVFS1?