# Presentation: 484240

## HP’s OpenSSI Linux Cluster Project
www.openssi.org

Bruce J. Walker
HP FellowOffice of Strategy and Technology

- HP’s OpenSSI Linux Cluster Project
- www.openssi.org
- Bruce J. Walker
- HP FellowOffice of Strategy and Technology

- hp-ssi.ppt

- 6/11/03

## Overview of HP  OpenSSI Cluster

**Single HA root filesystem**
**Consistent OS kernel on each node**
**Join cluster early in boot**
**Strong Membership**
**Single view of filesystems, devices, processes, ipc objects**
**Single management domain**
**Load balancing of connections and processes**
**Dynamic service provisioning**

- hp-ssi.ppt

- 6/11/03

**Notes:**
An SSI cluster is different than many other types of cluster as a result of
having a single root.  That single root may be accessed in parallel from 
multiple nodes but the nodes must automatically join the cluster early in 
boot to keep that root coherent.  Unlike a NUMA system, the SSI cluster
works by having an instance on the OS kernel resident in the memory of
each node.  These instances work together to present the SSI view
 of the resources.  The can work together by having a strong sense of
membership - each nodes knows the state of all other nodes, and by
having communication channels from the kernel on each node to each
other node.

## Goals

- Become the Industry Standard in Linux Clustering
- Scale up and Scale down
- Modular technology that applies to all forms of clustering
- 	- single technology base for HPC and Commercial

- hp-ssi.ppt

- 6/11/03

## Clustering Goals

- To various degrees, all of:
- 	High Availability
- 	Scalability
- 	Manageability
- 	Usability

- hp-ssi.ppt

- 6/11/03

## HP OpenSSI Linux Clusters

- SMP

- HUGE

- Ideal/Perfect Cluster in all dimensions

- ReallyBIG

- SMP

- HP SSI Linux Clusters

- log scale

- HP OpenSSI Linux Clusters

- Typical HA Cluster

- hp-ssi.ppt

- 6/11/03

**Notes:**
The ideal solution to each cluster area involves having a strong story in all four dimensions – availability, manageability, usability and scalability;
The HP SSI project is ambitiously addressing all four dimensions simultaneously in order to be a common platform foundation for all cluster variants.
Fundamental Single System Image is a key component

## HP OpenSSI Linux ClustersApproach

- 2 or more “service” nodes + optional “compute” nodes
- Computing can be done on service nodes
- Service nodes provide availability and 2 forms of load balancing
- Compute nodes for larger scale;
- Vproc for clusterwide process space, process launch and process movement
- Lustre for scale up filesystem story (including root)
- CFS or GFS for scale down
- Diskless node option
- No daemons need on compute nodes for job launch or stdio
- Integration with other open source components:
- Lustre, LVS, PBS, Maui, Ganglia, SuperMon, SLURM, ...

- hp-ssi.ppt

- 6/11/03

## Availability

**No Single (or even multiple) Point(s) of Failure**
**Automatic Failover/restart of services in the event of hardware or software failure**
**Filesystem failover integrated and automatic**
**Application Availability is simpler in an SSI Cluster environment;  statefull restart easily done;**
**Application transparent checkpoint/restart**
**Online software upgrade (ongoing) **
**Architected to avoid scheduled downtime**
**Node eviction via transparent process migration**

- hp-ssi.ppt

- 6/11/03

**Notes:**
SSI clusters include all the features of failover clusters so applications and
sources failover or restart in the event of failures.  
Availability can be better than it is in failover clusters for a couple of 
reasons: first application programming environment is single system rather
than distributed (and can thus help avoid programmer errors); second,
programming for failover is simpler because there is a single failover 
script and because all applications can run anywhere at any time and 
resources are failed over by the system and thus don’t require 
application-specific failover programming

## HP OpenSSI Linux Clusters -Manageability

- HP OpenSSI Linux Clusters -Manageability

**Single Installation; single root filesystem**
**Joining the cluster is automatic as part of booting and doesn’t have to managed**
**Trivial online addition of new nodes**
**Use standard single node tools (SSI Admin) **
**Visibility of all resources of all nodes from any node**
**Applications, utilities, programmers, users and administrators often needn’t  be aware of the SSI Cluster**
**Simpler HA (high availability) management**

- hp-ssi.ppt

- 6/11/03

**Notes:**
Unlike failover clusters, nodes of an SSI cluster are always part of the cluster, simplifying administration in booting and installation.   A key advantage SSI clusters have over large scale SMP environments is that as you grow your environment you can add nodes with the latest technology.
Some cluster technologies talk about Cluster management tools - another set of administration to learn.  SSI clusters can be managed with very small additions to standard management facilities.  Unlike other cluster technologies,  SSI clusters provide the opportunity to be unaware of the cluster. 

## HP OpenSSI Linux ClusterEase of Use

**Can run anything anywhere with no setup;**
**Can see everything from any node;**
**service failover/restart is trivial;**
**automatic or manual load balancing;**
**powerful environment for application service provisioning, monitoring and re-arranging as needed**

- hp-ssi.ppt

- 6/11/03

## Scalability

- Two aspects – overhead and load balancing

- hp-ssi.ppt

- 6/11/03

## Scalability  - Overhead

-  **No daemons needed per node **
**(single HA clusterwide init)**
**Node monitoring adjustable and could co-ordinate noise**
**Minimal cross-node co-ordination**
**Efficient kernel-to-kernel rexec() for process launch**
**Parallel multicast boot**

- hp-ssi.ppt

- 6/11/03

## Scalability – Load Balancing 

- Scalability – Load Balancing

**Load balancing of connections and processes**
**SSI makes distributing function very easy**
**SSI allows any process on any processor**
**All resources transparently visible  from all nodes:**
** ****filesystems, IPC, processes, devices*, networking***
**OS version in local memory on each node**
**Migrated processes use local resources and not home-node resources**

- hp-ssi.ppt

- 6/11/03

**Notes:**
What design characteristics of the SSI Cluster are going to allow scalability of applications?  The first attribute is that any process can run on any node.  This is a key attribute, along with transparent access to all resource are the features that facilitates scalability in SMP environment.  Unlike the SMP environment, however, there are independent  memories on each node and independent copies of the OS kernel.  What this means is that memory contention is reduced, as is SMP lock contention.  For the right set of applications, the best scalability is accomplished by SSI clustering together medium-scale SMP machines (4-way SMP).  In a scalability test done with 30 4-way SMP nodes running NonStop Clusters on Unixware, the scalability was above 95%, while on the same test a large-scale SMP (64 processors, about 1/2 that of the cluster) was only getting 61% scalability.  OS kernel lock contention and memory contention are serious as one tries to scale up.   SSI clustering is the answer in many cases, not only because of the parallelism but also because one can cluster price/performance commodity hardware and can incrementally add on the latest, hottest machines without replacing the existing cluster (no fork-lift upgrades).

## Component Contributions to HP OpenSSI Cluster Project

**HP OpenSSI Cluster Project**

- CLMS

**GFS**

**DLM**

**LVS**

**Mosix**

**Beowulf**

**DRBD**

**EVMS**

**Lustre**

**Vproc**

**Appl. Avail.**

- IPC

**CFS**

**Load Leveling**

**       **

**      **

- HP contributed

- Open source and integrated

- To be integrated

**OCFS**

- hp-ssi.ppt

- 6/11/03

**Notes:**
I wouldn’t suggest you try to explain each of these but just get the message across that we donated several different technologies (ones in red), we integrated with a bunch of other open source projects/technologies (green) and still have other to integrate with (blue).
In case someone asks:
CLMS – cluster membership and management subsystem
CFS -  cluster filesystem from NSC 
IPC – clusterwide interprocess communication (semaphores, message queues, pipes, sockets, etc.)
Vproc – virtual process subsystem that gives a clusterwide process space and clusterwide process management (and process movement)
Load leveling – both putting processes on less loaded nodes and moving running processes from overloaded nodes to less loaded nodes
Appl. Avail. – Application and resource availability – monitoring and restarting daemons and services and applications

Lustre – highly parallel and scalable cluster filesystem project that HP is heavily involved with (project lead)
openGFS – open source version of the Sistina GFS cluster filesystem
openDLM – distributed lock manager project originally open sourced by IBM and then abandoned
LVS – Linux Virtual Server project – used to provide a single, available cluster IP address for the cluster and to load balance incoming connections to the cluster;  some manageability extensions are provided in OpenSSI for LVS
Mosix – we leveraged the process load balancing algorithms and adapted them to OpenSSI
Beowulf – we integrated an MPI implementation (LAMPI) and others have run with MPICH

OCFS -  Oralce Cluster Filesystem
DRBD – Distributed replicated block device – with this and CFS you can have a highly available cluster filesystem without having multiported disk hardware
EVMS – Enterprise Volume manager – cluster aware volume manager

## SSI Cluster Architecture/ Components

**5. Process**
**    ****Loadleveling**

**16. Appl Availability;**
**      ****HA daemons**

***1. Membership***

***2. Internode Communication/***
***      ******HA interconnect***

**3. Filesystem**

**CFS**

**Physical filesystems**

**9. Devices/**
**shared storage**
**devfs**

**7. Networking/**
**     ****LVS**

**4. Process Mgmt**
**       ****VPROC**

**6. IPC***  *

**15. Sysadmin;**** **

- Kernel
- Interface

**11. CLVM/**
**      ****EVMS (TBD)**

**GFS**

**8. DLM**

**10. Kernel data**
**replication service**

**18. Timesync**

**13. Packaging and Install**

**14. Init; booting; **
**      ****run levels**

**12. DRBD **

**19. MPI, etc. **

**Lustre**

**17. Application Service**
**      ****Provisioning**

- hp-ssi.ppt

- 6/11/03

**Notes:**
This is a first cut at listing all the component areas for SSI clusters.  Many of
the component areas will be touched on in subsequent slides.
Those below the line  are in the kernel. The filesystem component can be satisfied either by layering a cluster 
filesystem (CFS) over a standard physical filesystem (like ext3) or by having a parallel 
physical filesystem like GFS or Lustre.

## Vproc Features

**Clusterwide unique pids (decentralized)**
**Peer-to-peer – no master node**
**process and process group tracking under arbitrary failure and recovery**
**no polling**
**reliable signal delivery under arbitrary failure**
**process always executes system calls locally**
**no “do-do” at “home node”; never more than 1 task struct per process**
**for HA and performance, processes can completely move**
** ****therefore can service a node without application interruption**
**process always only has 1 process id**
**transparent process migration**
**clusterwide /proc, **
**clusterwide job control**
**single init**
**Unmodified “ps” shows all processes on all nodes**
**transparent clusterwide debugging (ptrace or /proc)**
**integrated with load leveling (manual and automatic)**
- exec time and migration based automatic load leveling
- fastnode command and option on rexec, rfork, migrate
**architecture to allow competing remote process implementations**

- hp-ssi.ppt

- 6/11/03

##  HP SSI Linux Clusters - Status

**Version 1.0 just released – **
**	****Binary, Source and CVS options**
**	****Functionally complete RH9 and RHel3**
**	****Debian release also available **
**	****IA-32 and Itanium Platforms **
**	****Runs HPTC apps as well as Oracle RAC**
**	****Available at OpenSSI.org **

- hp-ssi.ppt

- 6/11/03

**Notes:**
The OpenSSI Cluster project (www.opensource.hp.com or
sourceforge.net/projects/ssic-linux) has source code available to build
clusters for development.  From the above list it is clear than most of
the basic functionality is there but some work is needed before one could
make a product.

## Ongoing Enhancements

- 2.6-based version
- Improved scalability (two node classes)
- Integrate power/console management
- SMP Clusters (multiple nodes within a large SMP)
- Vproc without single root (flexibility)
- Node eviction

- hp-ssi.ppt

- 6/11/03

## HP OpenSSI Linux Clusters - Conclusions

- HP has recognized that Linux clusters are important part of the future.
- HP has recognized that combining scalability with availability and  manageability/ease-of-use is key to clustering
- HP is leveraging its merger with Compaq (Tandem/Digital) to bring the very best of clustering to a Linux base

- hp-ssi.ppt

- 6/11/03