# Presentation: 235618

## The New MCS Large Computing ResourceChiba CityA Linux Cluster in 314 parts

- Rémy Evard
- evard@mcs.anl.gov
- 30 September 1999

## Chiba City

**8 Computing Towns**
- 256  Dual Pentium III systems

**1 Visualization Town**
-   32 Pentium III systems
-  with Matrox G400 cards

**1 Storage Town**
-   8 Xeon systems
- with 300G disk each

**Cluster Management**
-   12 PIII Mayor Systems
- 4 PIII Front End Systems
-  2 Xeon File Servers
- 3.4 TB disk

**High Performance Net**
-   64-bit Myrinet

**Management Net**
- Gigabit and Fast Ethernet
- Gigabit External Link

- The Argonne Scalable Cluster

**27 Sep 1999**

## Why Linux Clusters?

_**Cost**_
- Cheap computing
- As pioneered by the Beowulf project.
- Maximize Performance
- Alpha-based clusters.
- Replace your SP for just the cost of maintenance!

_**Capability**_
- Open Source
- The basis of possible community efforts.
- Scalability
- Cheaper cost of components means larger systems for the same $.
- Flexibility
- “Build it yourself” means build it how you like, change it how you like.
- Applicability to many problem domains.

## Alarming Trends of Systems Software Availability from Computing Vendors

**Applications**

**High Perf Tools**

**High Perf Env**

**System Software**

**Operating System**

**Hardware**

**Solution**
**provided by:**

**Applications**

**High Perf Tools**

**High Perf Env**

**System Software**

**Operating System**

**Hardware**

**Customer**

**Vendor**

**1990**

**2000**

***2010***

**Solutions**
**Created by...**

- Focusing on
- Scientific
- Computing

- Leveraging
- Business
- Solutions

- Some
- Government
- Research

- GigaFLOPs

- TeraFLOPs

- PetaFLOPs

**Necessary**
**computing**
**power:**

**Notes:**
Talking points for this slide:
  The ASCI barriers story
  The reasons that vendors have decreased support and their realities
  The need for Petaflops

## So... why not Linux Clusters?

- A very bad multi-system I/O situation.
- Immature schedulers and systems management.
- “Do it yourself” means ... “do it yourself”.
- Everyone else is doing it.

## Cluster Projects at MCS

- Windows NT SuperCluster (1996-1998) (8 heterogeneous nodes)  (r.i.p.)
- Windows NT MPI porting project (8 nodes)
- Microsoft supported (PIIs,SMP, 256 MB, 8 GB disk, Giganet)
- Collage and ActiveMural (22 nodes)
- Linux Cluster to Drive (PIIs, 256 MB, 4 GB disk, display Adapters)
- Chiba City Test Clusters (4-16 nodes)
- various development environments
- Chiba City Main System (300+ nodes)
- 256 core nodes (dual PIIIs, 512MB RAM, 9 GBdisk, Myrinet, Gig-E)
- 32 visualization nodes (PIIIs, 512MB RAM, 9 GBdisk, Myrinet, Matrox G200s)
- 8 storage nodes (Xeon, 300 GB Disk/node)
- 18 management nodes (PIIs, multiple networks, 18 GB disk)
- The DSL Data Grid Node (20 nodes) 4 TB aggregate
- PIII, 512 MB RAM, 200 GB disk, Fast and Gig-E
- Alpha Cluster for Computational Biology
- 18 XP1000’s (633MHz, Ev6), 512 MB RAM, 10 GB disk , ServerNetII, Management Node

## Software Research on Clusters at ANL

- Scalable Systems Management
- Chiba City Management Model (w/LANL, LBNL)
- MPI and Communications Software
- GigaNet, Myrinet, ServerNetII
- Data Management and Grid Services
- Globus Services on Linux (w/LBNL, ISI)
- Visualization  and Collaboration Tools
- Parallel OpenGL server (w/Princeton, UIUC)
- vTK and CAVE Software for Linux Clusters
- Scalable Media Server (FL Voyager Server on Linux Cluster)
- Scalable Display Environment and Tools
- Virtual Frame Buffer	Software (w/Princeton)
- VNC (ATT) modifications for ActiveMural
- Parallel I/O
- MPI-IO and Parallel Filesystems Developments (w/Clemson, PVFS)

- Argonne Software is OpenSource

## “One of these things is not like the others...”

- Politics

- Sausages

- Big Computer Purchasing

- Logic
- &
- Beauty

## Our Cluster is from:

- IBM
- Servers, mayors, visualization systems
- OEMing:
- VA Linux Systems
- Compute nodes
- Cisco
- Ethernet
- Myricom
- Myrinet
- Others
- Racks, cables, power boxes, serial control...
- (A big thanks to Ed Jedlicka.)

## Chiba City System Details

- Purpose:
- Scalable CS research
- Prototype application support
- System - 314 computers:
- 256 computing nodes, PIII 500MHz, 512M, 9G local disk
- 32 visualization nodes, PIII 500MHz, 512M, Matrox G200
- 8 storage nodes, 500 MHz Xeon, 512M, 300GB disk: 2.4TB total
- 10 town mayors, 1 city mayor, other management systems: PIII 500 MHz, 512M, 3TB disk

- Communications:
- 64-bit Myrinet computing net
- Switched fast/gigabit ethernet management net
- Serial control network
- Software Environment:
- Linux (based on RH 6.0), plus “install your own” OS support
- Compilers: GNU g++, PGI, etc
- Libraries and Tools: PETSc, MPICH, Globus, ROMIO, SUMMA3d, Jumpshot, Visualization, PVFS, HPSS, ADSM, PBS + Maui Scheduler

## Chiba City

**8 Computing Towns**
- 256  Dual Pentium III systems

**1 Visualization Town**
-   32 Pentium III systems
-  with Matrox G400 cards

**1 Storage Town**
-   8 Xeon systems
- with 300G disk each

**Cluster Management**
-   12 PIII Mayor Systems
- 4 PIII Front End Systems
-  2 Xeon File Servers
- 3.4 TB disk

**High Performance Net**
-   64-bit Myrinet

**Management Net**
- Gigabit and Fast Ethernet
- Gigabit External Link

- The Argonne Scalable Cluster

**27 Sep 1999**

## Raw Numbers

- Computers
- 256 systems for computing.
- 288 systems for user processes.
- 570 CPUs total.
- 314 systems total.
- 256 GF theoretical peak.
- Disk
- 5.8 TB of disk, excluding node disks.
- 8.4 TB of disk, including node disks.
- Memory
- 128G of RAM on computing nodes.
- 153G of RAM total.

## Planned Usage

- Generally
- 50% for batch-scheduled application code
- 50% for dedicated cluster and scalability research
- Not infrequently
- 100% used for special projects

## A Few of the Requirements

- Support CS development:
- Users with root on their nodes.
- Nodes with custom drivers, kernels, and OSs.
- The ability to schedule those customizations.
- Support multiple users and multiple projects:
- The usual production supercomputer environment: batch scheduling, application development environment, reliability...
- Manageability for 1000s of nodes:
- Think in terms of 1,000,000 nodes.
- Only have to touch a node when we first install it.
- Completely recover from whatever a user did to it.
- ...

## CS Development & Scalability Usage

- Early environment:
- Nodes dedicated to one user or group manually for days or weeks.
- Work using Chiba system image, with root access if necessary.
- No global file system.
- Archive to serial HSM.
- Nodes can be rebuilt from scratch on demand.
- Eventually:
- Can install own custom OS - modified Linux, or NT... Whatever.
- Can schedule the OS installation.
- Can schedule chunks of the cluster for long term work.

## Application Usage

- Generally:
- Use “login” nodes for compiles and tests.
- Submit jobs to scheduler.
- Interactive access to nodes quite possible.
- Early environment:
- Scheduler will be suboptimal.
- Data migration support will be suboptimal.
- No global file system.  No parallel file system.
- No allocations.
- Eventually:
- Better schedule, data migration, and archive interfaces.
- Global file system?  Parallel file system?
- Allocations.

## Systems Management

- Goals:
- Scalability up to thousands of nodes.
- Portability of tools to many cluster types and site requirements.
- Support of CS research.
- Users with root, users doing horrible things to the OS.
- Support of many apps.
- Scheduled, production env.
- Initial goal: “at least as good as the SP”.

- R&D:
- User-driven OS selection:
- Users with custom OSs, or root requirements
- Automated OS installation
- OS image scheduling
- Zero-touch recovery
- Scalable environment
- Complete remote mgmt.
- Hierarchical file and command distribution.
- Town-based configuration and monitoring.
- Error Management

## A Chiba Town

- A “town”is the basic cluster building unit.
- 32 compute nodes
- Compute nodes run user jobs.
-  In a special purpose town, such as the ‘storage town’, they run dedicated services.
- 1 mayor node
- The mayor node manages the other nodes by:
- OS distribution to each node.
- Monitoring the status of each node.
- Console centralization for town.
- File distribution within town.
- The mayor node is not available to user processes.

**mayor**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

## Cluster Management Systems

_**City Mayor**_

- The city mayor is used to manage each of the town mayors, to serve consoles for the mayors, and to monitor basic system parameters.

_**Scheduler**_

- The scheduler is the node that runs the main scheduling daemons.

_**Home File Server**_

- The home file server stores the home directories of the cluster users, and exports them via NFS to the front end machines.

_**System File Server**_

- The system file server stores the basic Linux distribution, 3rd party software installations, and so on.  It exports this via NFS to the front end machines and the mayors.

## Ethernet Network

- The Ethernet:
- Primarily for builds and monitoring.
- Completely switched.
- Fast switched ethernet within towns.
- Mayors, front ends, and control systems are on gigabit ethernet.
- Interconnect is gigabit ethernet.
- IP Topology:
- 1 flat IP subnet.

**Eth**
**Switch**

- ...

**Gig Eth**
**Switch**

- ...

- ANL network

- Control Systems

- Front Ends

- Test Clusters

- Gigabit Ethernet

- Fast Ethernet

## System: Console Management

- Consoles are not required for the cluster nodes, but can be very handy, so we are working to support their use.
- Console purposes:
- Backup diagnostic path when networking on the node is down.
- Building and booting control.
- BIOS configuration, given the right kind of BIOS.
- Console usage:
- Scripting and monitoring of each node by its mayor.
- Manual access to console of any node (even with the network down).
- Console Hardware:
- Cyclades PCI card
- Serial expansion box, providing up to 64 serial connections.
- Console Software:
- Server runs on mayor, providing network access to the serial connections.
- Mayor consoles connected to City Mayor.
- Client console output, LILO, Boot, login, console.
- Pre-LILO dependent on BIOS
- Open Questions:
- How can the BIOS and Management features of the NetFinity and Intellistation impact or improve this plan?
- Can the BIOS of these systems somehow be accessed remotely?

**mayor**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**node**

**serial**
**concentrator**

- serial lines
- from each node

- network access
- to the console
- serial line

**city**
**mayor**

**serial**
**concentrator**

- ...

**mayor**

**mayor**

- scripting,
- monitoring

- scripting,
- monitoring

**node**

## Node Image Management

- Power on.

- Wait at LILO
- for Mayor.

- Boot from
- local disk.

- Boot from
- mayor.

- Implant boot.

- Node has correct OS.

- Node needs new OS.

- Network install of OS image.

- Error condition.

- Halt

- The mayor decides, based on a database of nodes and possible images, what image should be installed on a node.  It installs the image via a controlled boot process.
- All that is necessary to recover a node is to power cycle it.

## Scheduled Goals, part I

- Early October		Hardware Installation.
- 				Verification tests.
- 				1 town up for testing.
- 				Viz town up for SC99 prep.
- 				Data Grid up for SC99 prep.
- Pre-SC99		Panic.
- Post-SC99		All systems up.
- 				Myrinet installation.

## Optimistic Schedule, part II

- December		Alpha test.  Goal:  Work kinks out of system.
- 				128 nodes for application-level testing.
- 				128 for developer-level testing.
- 				Frequent reconfigs to test things and do experiments.
- 				Users can work with Systems to fix things.
- January			Beta test.  Goal: Semi-production mode
- 				128 nodes with scheduler, data migration, ...
- 				128 nodes for developer testing
- 				Users in “friendly user” mode.
- Spring:			Production mode.
- 				Users reviewed as per SP.
- 				OS image scheduling and CS research scheduling.

## Futures

- As funding questions get resolved, we will be looking at the next system.  The goal: 1000 nodes.
- Most likely, if that happens, this system will be devoted entirely to production computation.
- In the meantime, we believe the state of system software on Linux - schedulers, file systems, archiving, management - will seriously improve.

## MCS Cluster Raising  -  Help!

- Thursday, October 7th  &  Friday, October 8th
- Big community installation effort.
- We need help from the rest of MCS:
- Teams of 3-4 folks each.
- Each team builds and tests 1 town.
- 2 teams going at a time.
- Each “shift” lasts the morning or an afternoon.
- Grab some folks and sign up for a chunk of time on one of those days.
- If you can square dance, that would help too.

## The New MCS Large Computing ResourceChiba CityA Linux Cluster in 314 parts

- Rémy Evard
- evard@mcs.anl.gov
- 30 September 1999