# Chiba Cluster Install/Boot

Chiba Grey Doc #9

Rémy Evard, John-Paul Navarro, Daniel Nurmi

{evard, navarro, nurmi}@mcs.anl.gov

Mathematics and Computer Science Division

Argonne National Laboratory

24 August 1999

# Background

Hopefully by the end of this paper the reader will understand how we
have designed the method by which we are intending to install and boot
the Chiba City compute cluster. There are many aspects of the process
that were identified early on that affect how the final process has been
designed. The most crucial aspect involves the multiple OS flavors that
we plan to support across the cluster. This means that if a user so
desires, they can specify that for some allotted amount of time their
nodes can be running whatever OS they would like to run. For this
reason, our install process is in some sense complicated in that it must
understand that we as systems administrators can never assume that we
have control over what is happening on an individual node. The install
process is also designed such that even if a user wants to run a
completely experimental OS which is unsupported by our known OS, they
can do so. This functionality is not, however, over engineered. Even if
a prospective user of our system would only want one OS across their
system, we believe that our design is still very easy to
install/manage/upgrade and it provides excellent failsafe measures.
Another aspect that we have kept in mind is the scalability issue. We
need to have a cluster that can grow to tens of thousands of nodes. We
also need the system to run on an eight node system. This aspect comes
into play when we decide where data is stored and how it is accessed.
The final core issue that we have adhered to is keeping it simple. We
would like for anybody with PCs from joe random vendor to be able to
install a high performance, fully managed system in a day. This issue
has almost been the most difficult to stick with, as custom hardware
makes life seem easier at times.

# Assumptions

This document assumes that the reader is familiar with the Chiba City
system in that terms such as 'town' and 'mayor' will be thrown around
with reckless abandon. Reading the Chiba City Hardware Design Document
will perhaps help.

# Core Concepts

There are a few ideas that need to be explained before the core process
can be described.

**Image state:** We refer to a prospective OS file to be an image state.
An image state in our system can take two forms. One form is that of a
large disk image file. Disk image files can be created by using a tool
such as 'dd' on a linux system. The advantage of this sort of disk image
file is that along with OS data, the one file contains master boot
record information, disk partition information, file system format
information, etc. Some of this information could conceivably be
nonaccessible in any other format. For instance, if a user wants to run
an odd OS on a certain number of nodes, they need only to install it on
one node. Out process can then make an image file of the node's disk and
copy it back onto any number of machine's disks. Our management system
never needs to know anything about the OS that it is putting on to any
machine. The disadvantage of this method is that the disk files are
large and unchangable in and of themselves. If a change needs to be made
to a node's image state, the change must be made on a running system and
then a new image must be created that has the changes incorporated into
it.

**System state:** A system state is a more abstract idea. Our
boot/install process identifies four system states that any node in the
cluster can be in. This is explained later in the document.

**Management system:** Throughout the document, references are made to
the management system. This is a suite of software that runs on a mayor
system which controls the boot process of it's nodes. It is responsible
for deciding when a node needs to change from the state is is currently
in.

**LILO:** Lilo is the standard boot loader for linux systems and it's
prompt can be detected and controlled through a serial line. From a lilo
prompt accessed over a serial line, a mayor can control what a machine
will do next.

Keeping with the hierarchical design, our build process is distributed
in the sense that each mayor is running a management system for it's
nodes. The management system is flexible enough to be used by any mayor
to control any type of node. The city mayor is actually responsible for
managing each mayor in the same way that the mayors are responsible for
managing their nodes. When an admin or automated process decides that a
node needs to change image state, the mayor's management system is
triggered and the process begins. Our system state switching process has
a fairly simple core structure. Most of this structure relies on the
image state of an individual machine because that is really what defines
the personality of a cluster. The management system really only needs to
know what the current image state of each node is and what the image
state should be. With these two pieces of information, the management
system can decide what kind of action it needs to take. The two
important states are the ***boot*** system state and the ***build***
system state. The following state diagram describes the various system
states any machine in the cluster can be in.

![](media/image1.wmf)

### State 0: Preboot

The preboot system state is the state a machine is in when it has no
power. At this time, the image state of the node is not known to the
management system. Any machine should only be in this state at the "node
has never before been booted" stage.

### State 1: LILO

Once power is applied to the machine, it will enter this system state. A
node will also default to state 1 if it ever reboots. State 1 is the
system state at which decisions need to be made. When the management
system detects a machine in this system state it asks the question,
"What image state *is* the node in and what image state *should* the
node be in?". All image state information is stored in a central
database accessible by the management system. The answer will describe
what happens next. The following table describes this decision making
process.

  ----------------------------------- -----------------------------------
  **Answer\***                        **Action**

  "Node is in UNKNOWN image state and Stall, alert admin to the fact that
  should be in UNKNOWN image state."  a node is trying to boot but the
                                      system has no knowledge of what the
                                      machine should be.

  "Node is in IMAGE_0 image state and Stall, alert admin to the fact that
  should be in UNKNOWN image state."  a node is trying to boot but the
                                      system has no knowledge of what the
                                      machine should be.

  "Node is in UNKNOWN image state and Instruct node to proceed to state
  should be in IMAGE_0 image state."  3.

  "Node is in IMAGE_0 image state and Instruct node to proceed to state
  should be in IMAGE_0 image state."  2.

  "Node is in IMAGE_0 image state and Instruct node to proceed to state
  should be in IMAGE_1 image state."  3.
  ----------------------------------- -----------------------------------

\*The question asked by the mayor of the database was, "What image state
is node in and what image state should node be in?"

As actual answer would look something like "Node is in LINUX_0 image
state and should be in LINUX_0 image state" in which case the management
system decides that since the current node image state is consistent
with what the node image state should be, the machine should be allowed
to enter it's normal boot cycle (state 2). If the answer looks like
"Node is in LINUX_0 image state and should be in BEOS_1 image state.",
then the management system realizes that the node image state needs to
be changed and it instructs the node to enter it's build state (state
3).

### State 2: Boot

The boot state is the simplest state in the system. The management
system only knows that the node is in boot state and it should be
running.

### State 3: Build

The build state can become rather complicated. This is the state at
which an image state is assigned to the node. When a node enters the
build state, the management system knows that the node is trying to make
it's expected state match it's current state in the database. At the end
of the build process, the node will contact the database machine and
report that the 'current' image state of the node is now equal to the
'should be' state of the node.

### State Errors

Methods for detecting whether or not a state error has occurred differ
depending on what state a node is presently in. State 0 only has one
real error which is, "powered on node, never reached state 1". The
"never reached" phrase implies that a timeout is set on a per node basis
within the database. The management system will then start timing the
node as soon as it decides a node should boot. If the node reaches state
1, the state change is recorded and the management system stops
counting. If the boot manager's counting exceeds the timeout before it
gets confirmation that the node is at state 1, it means an error has
occurred. State 1 is the default state. Any errors that happen here will
be picked up as other state errors. State 2 can have many errors but
they all have the same implications. The only way to detect a state
error when a node is in state two is to have knowledge of the image
state of the node. For example the management system cannot use ping to
check for happy nodes because this is assuming that we know that the
image state of the node responds to pings. For this reason, the
management system will have knowledge of some image states (those that
are controlled by us) and it can have a method for checking on node
happiness. When a certain time period has been exceeded between checks,
the management system can know that there is a state error. State 3 has
a similar state error scenario to state 2. When the management system
send a node into state 3, it will again start a count and compare it to
a timeout value gotten from the database. If the 'current' image state
of the node is different than the 'should be' image state of the node
after the timeout is exceeded, a state error has occurred.

If an state error occurs in any of the states, the management system
will simply change the 'should be' field of the database to UNKNOWN and
reboot or power cycle the node. This will send it into state 1 where the
management system will detect a serious logical flaw. How is it that a
machine 'should be' in an unknown state? (see state 1 table above)

# Core Tools

Chiconsole management software, ChiDB, rootfs, bootdisk, buildtools

To actually implement the above procedure, there are a number of tools
that are being designed. These are briefly described here so that we can
go through the actual process in a step by step manner in another
section. These tools are all part of the management system that has been
defined above.

**Console Management:** The console management system is responsible for
allowing mayors to get information to and from nodes that they manage.
This software includes a trigger/response system that watches for
triggers sent to serial lines and responds to them. It is important that
we use a serial line because it allows us to assign host identity based
on physical location.

**Boot Manager:** There must be an entity that decides and watches the
state of each node. This is the boot manager's responsibility. The boot
manager not only triggers node image and system state changes, it also
is the entity that times how long a node has been in it's current system
state and decides whether or not a state error has occurred. The final
duty of the boot manager is to report state errors to the admin as it is
the system that understands what happened when a state error is
detected.

**Database:** The system database is a simple database of single
key/value pair records. Each node has one record which includes any
information that could conceivably change. For booting/installing, the
image and system state information is really what we need to access on a
per node basis.

**Bootfs:** The bootfs is an expanded linux file system that resides on
each mayor. The mayor is exporting this file system over NFS to any
number of nodes.

**Bootdisk:** Each node has a bootdisk in it's floppy drive. The disk
has a simple kernel that has all necessary hardware drivers and network
drivers installed in it. It really needs only to be able to access a
node's local hard disk and mount a root file system over NFS. The floppy
also has it's lilo sending/receiving data through the serial line.

# Core Procedure

This section intends to go through the actual boot and base image
install of a single node within Chiba City. The assumption is that the
node hardware has never been powered on before and therefore nothing is
known about the machine and it's disk is completely blank. The node is
connected to and ethernet network as well as a serial network managed
directly by it's mayor. The bootdisk is inserted into it's floppy drive.
Both the 'current' and 'should be' image state fields in the database
are set to 'UNKNOWN'. The 'current' and 'should be' system state fields
are both set to 'STATE0'

1.  Node is in system state 0 (powered off).

2.  Admin sets the 'should be' system state field of the database to
    'STATE1' and the 'should be' image state field to a known working
    image 'BASE'.

3.  The boot manager detects that the node should be in system state
    'STATE1' and triggers a power on of the node. The boot manager also
    gets a timeout value and starts counting.

4.  After bios checks, the node boots it's bootdisk and sends it's lilo
    prompt over the serial line. The node is now in system state 1.

5.  The serial watcher detects that a node has booted. The serial
    watcher sets the node's 'current' system state field in the database
    to 'STATE1'. The serial watcher asks the question, "What image state
    is node in and what image state should node be in?" and receives the
    answer, "Node is in UNKNOWN image state and should be in BASE image
    state.".

6.  The boot manager sees that both 'current' and 'should be' system
    state fields are set to 'STATE1' and stops counting.

7.  Serial watcher then decides that since the image state is not
    consistant, the node needs to build. Serial watcher sets the 'should
    be' system state field to 'STATE3'. The serial watcher then sends a
    serial command to the node's waiting LILO prompt telling it to mount
    it's root filesystem over NFS to it's mayor and use it's real IP
    address when doing so.

8.  Detecting and inconsistancy in 'current' and 'should be' system
    states, the boot manager gets a timeout value and starts counting.

9.  Node loads the disk kernel and mounts it's root file system over NFS
    from it's mayor. After setting the 'current' system state field to
    'STATE3', the node is in system state 3.

10. Install script is started on local node. The install script queries
    the database to get 'should be' image state information. Once
    gotten, the script will start to copy the image onto the local disk.

11. Once the install script finishes copying the data onto local disk,
    it sets the 'current' field for it's node in the data base to a
    value equal to the node's 'should be' field. In this case, this is
    'BASE'. After setting the node's 'should be' system state to
    'STATE1', the script will tell the node to reboot.

12. Boot manager detects system state inconsistancy and starts counting.

13. After rebooting, the node is now back in state 1. The LILO prompt is
    again sent over the serial line.

14. The serial watcher detects the LILO prompt and queries the database
    for node image state information. Serial watcher sets node's
    'current' system state to 'STATE1'. This time, it notices that the
    'current' image state field and the 'should be' image state field
    are both set to BASE so it decides to move the node into system
    state 2. The 'should be' system state field is set to 'STATE2'.

15. Boot manager detects inconsistancy in system state fields and starts
    counting.

16. The serial watcher sends a command to the node's waiting LILO prompt
    telling it that it needs to attempt to boot from local disk.

17. Whatever is on local disk is loaded. The node is now in state 2.
    Depending on what image state is loaded, the boot manager may try to
    contact the node's local OS. It can then decide whether or not a
    state error has occurred. If not, the 'current' system state field
    is set to 'STATE2'.