Date: 6/6/2006

From: John Urish

RE: Software RAID 1 for Linux

This document describes a procedure to convert an existing Scientific
Linux Fermi installation to a bootable software RAID 1 mirror.

The knowledge was gathered from various web resources and the book
"Managing RAID on LINUX" by Derek Vadala, published by OReilly. Don
Holmgren and Dan Yocum provided invaluable advice.

There are five parts to the process; Configuration of the disks,
creating the array, cloning the current disk, modifying the boot loading
process and making both disks bootable.

Your boot system must have a kernel that has been compiled with RAID
extensions and a system that contains the "mdadm" tool or "Raidtools".
"mdadm is preferable, it has a better feature set and does not require a
configuration file to be written before creating the array.

The second disk should be of approximately the same size and performance
as the existing disk. It cannot be smaller. If the second disk is
larger, the extra space will not be used by the RAID array. You may
create a non-RAID partition with the additional space.

Install the second disk on a separate bus if using IDE. If both disks
are on the same IDE bus and one fails, it may hang the bus. That will
negate the purpose of creating the RAID1 array.

The example in this document uses IDE drives hda and hdc on Primary IDE
and Secondary IDE respectively. Both disks are 40 GB.

Disk /dev/hda: 40.0 GB, 40020664320 bytes

255 heads, 63 sectors/track, 4865 cylinders

Units = cylinders of 16065 \* 512 = 8225280 bytes

Disk /dev/hdc: 40.0 GB, 40020664320 bytes

255 heads, 63 sectors/track, 4865 cylinders

Units = cylinders of 16065 \* 512 = 8225280 bytes

The original disk (hda) contains a default SLF 4.2 install. The disk was
formatted with Disk Druid in five partitions (one partition is the
extended partition). The sizes are 1=10GB, 2=10GB, 3=2GB(Swap),
4&5=16GB. Partitions four and five are the extended partition and the
partition contained in it. All partitions are created as type 83(Linux)
with the ext3 file system.

NOTE: It is recommended that all procedures be done in single-user mode
while logged on as root.

**WARNING: This procedure involves changing the partition table on the
original disk. This is a dangerous thing to do! ALWAYS have a verified
good backup before you start.**

**In especially critical cases, *do not change the partition type* of
the original disk, add a third disk and remove the original intact. Join
the third disk to the array as the last step of Part Four instead of
joining the original disk to the array.**

**Part One: Disk Configuration**

We will change the partition type of the existing disk to hex "fd" (the
RAID partition type) and create partitions on the second disk.

NOTE: You may do this step while booted from the original disk. Changes
will not take effect until reboot.

\[root@minos-spare \~\]# fdisk /dev/hda

The number of cylinders for this disk is set to 4865.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1\) software that runs at boot time (e.g., old versions of LILO)

2\) booting and partitioning software from other OSs

(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): t

Partition number (1-5): 1

Hex code (type L to list codes): fd

Changed system type of partition 1 to fd (Linux raid autodetect)

Do this for partitions 1,2,3 and 5. Partition 4 is the extended
partition and is not modified. *Don't forget to write the changes before
exiting fdisk!*

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or
resource busy.

The kernel still uses the old table.

The new table will be used at the next reboot.

Syncing disks.

The new second disk must be partitioned similarly to the original.
Because the second disk, in this case, is identical to the original, the
partition boundaries will be identical. You can specify the size rather
than cylinders in fdisk. Remember the partitions must be [at
least]{.underline} the same size as the original partitions!

\[root@minos-spare \~\]# fdisk /dev/hdc

The number of cylinders for this disk is set to 4865.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1\) software that runs at boot time (e.g., old versions of LILO)

2\) booting and partitioning software from other OSs

(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-4865, default 1):

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-77545, default 4865): 1275

Command (m for help): t

Partition number (1-5): 1

Hex code (type L to list codes): fd

Changed system type of partition 1 to fd (Linux raid autodetect)

Do this for partitions 1(10GB), 2(10GB), 3(2GB) and 5(rest of disk in
this case). *Don't forget to write the changes before exiting fdisk!*

Command (m for help): w

The partition table has been altered!

Below are the final partition tables for both disks:

Disk /dev/hda: 40.0 GB, 40020664320 bytes

255 heads, 63 sectors/track, 4865 cylinders

Units = cylinders of 16065 \* 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/hda1 \* 1 1275 10241406 fd Linux raid autodetect

/dev/hda2 1276 2550 10241437+ fd Linux raid autodetect

/dev/hda3 2551 2805 2048287+ fd Linux raid autodetect

/dev/hda4 2806 4865 16546950 5 Extended

/dev/hda5 2806 4865 16546918+ fd Linux raid autodetect

Disk /dev/hdc: 40.0 GB, 40020664320 bytes

255 heads, 63 sectors/track, 4865 cylinders

Units = cylinders of 16065 \* 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/hda1 \* 1 1275 10241406 fd Linux raid autodetect

/dev/hda2 1276 2550 10241437+ fd Linux raid autodetect

/dev/hda3 2551 2805 2048287+ fd Linux raid autodetect

/dev/hda4 2806 4865 16546950 5 Extended

/dev/hda5 2806 4865 16546918+ fd Linux raid autodetect

The file system will be created in Part Three. If you are creating a new
RAID from unformatted disks, wait until after the array is created to
make the file system with mkfs.

NOTE: Reboot and run "fdisk --l" to make sure disk changes take effect.

**Part Two: Create the RAID Array**

Create the raid array in degraded mode. The actual array creation now
takes place. This example uses "mdadm".

The options used are --C=create mode, -v=verbose, -l*n*=RAID Level(one
in this case), and the number of disks in the array --n*n*. The array
name is specified as md*n* and it uses hdc*n* and "missing".

\[root@minos-spare \~\]# mdadm --Cv --l1 --n2 /dev/md1 /dev/hdc1 missing

\[root@minos-spare \~\]# mdadm --Cv --l1 --n2 /dev/md2 /dev/hdc2 missing

\[root@minos-spare \~\]# mdadm --Cv --l1 --n2 /dev/md3 /dev/hdc3 missing

\[root@minos-spare \~\]# mdadm --Cv --l1 --n2 /dev/md5 /dev/hdc5 missing

**Part Three: Clone the Current Disk**

The contents of the current partitions will be copied to the RAID
partitions. This is done while preserving the current dates, permissions
and file hierarchy. Each partition is copied in turn.

Create a file system as appropriate on each of the new arrays. Use mkfs
for this. Use mkswap to initialize the swap partition. In this example
swap is on md3.

It's important to cd to the root of the partition when doing the
transfer. In the example below, the partition is hda2 which is mounted
as /var. The --xdev option prevents the find command from passing files
to cpio that are not on the current file system.

mkdir /mnt/md2mnt

mount /dev/md2 /mnt/md2mnt

cd /var

find . --xdev \| cpio --pm /mnt/md2mnt

umount md2mnt

rmdir /mnt/md2mnt

Check permissions for the /tmp directory. They should be "drwxrwxrwt".

**Part Four: Modify the Boot Configuration**

Now make the changes needed to boot properly from the RAID array. The
/etc/fstab, initrd and grub.conf files will have been created for
booting without RAID. We will make the necessary changes to boot from
the RAID array. The files on both the new and current disks will be
modified. That will allow the system to boot from the RAID array at the
next reboot.

NOTE: If you are creating the RAID array from two new disks and keeping
the original intact, do not copy these files to the original disk. After
making the file modifications to the RAID array, join the third disk to
the array and remove the original disk from the system. Before rebooting
use the Grub procedure in Part Five on the second and third disk.

Change /etc/fstab to reflect the new RAID partitions.

\# This file is edited by fstab-sync - see \'man fstab-sync\' for
details

/dev/md1 / ext3 defaults 1 1

/dev/md2 /var ext3 defaults 1 1

none /dev/pts devpts gid=5,mode=620 0 0

none /dev/shm tmpfs defaults 0 0

none /proc proc defaults 0 0

none /sys sysfs defaults 0 0

/dev/md3 swap swap defaults 0 0

/dev/md5 /home ext3 defaults 1 1

/dev/hdd /media/cdrom auto

/dev/fd0 /media/floppy auto

Edit /boot/grub/grub.conf. Change the kernel line to have root pointing
to the RAID partition on which the kernel resides. In this case md1.

\# grub.conf generated by anaconda

\#

\# Note that you do not have to rerun grub after making changes to this
file

\# NOTICE: You do not have a /boot partition. This means that

\# all kernel and initrd paths are relative to /, eg.

\# root (hd0,0)

\# kernel /boot/vmlinuz-version ro root=/dev/hda1

\# initrd /boot/initrd-version.img

#boot=/dev/hda

default=0

timeout=5

splashimage=(hd0,0)/boot/grub/splash.xpm.gz

hiddenmenu

title Scientific Linux 42 (Fermi) (2.6.9-22.0.2.EL)

root (hd0,0)

kernel /boot/vmlinuz-2.6.9-22.0.2.EL ro root=/dev/md1

initrd /boot/initrd-2.6.9-22.0.2.EL.img

"initrd" will not have the raid driver listed for installation in the
RAM disk at boot time. This is a compressed file created with cpio. It
is recommended that it be recreated using mkinitrd rather than edit the
file. Because we have installed a RAID disk, mkinitrd will now pick up
the driver for RAID when it runs. It will also create nodes for each of
the RAID partitions in the RAM disk.

Use the command: mkinitrd /boot/initrd-2.6.9-22.0.2.EL.img
2.6.9-22.0.2.EL

NOTE: Use the same initrd file name and location given in grub.conf.
This should be the current kernel the system is booted from. You will
need to delete or move the current initrd file before running the
command. Copy the new file to both disks.

To see the contents of initrd, create a temporary directory and cd to
it. Then execute this command:

Gunzip --c /boot/initrd-2.6.9-22.0.2.EL.img \| cpio --icdumv

The contents of the initrd file will be uncompressed into the current
directory.

Reboot. Then add the hda partitions (The partition type was already
changed to Linux Auto in Part One) to the array as follows:

\[root@minos-spare \~\]# mdadm /dev/md1 --a /dev/hda1

\[root@minos-spare \~\]# mdadm /dev/md2 --a /dev/hda2

\[root@minos-spare \~\]# mdadm /dev/md3 --a /dev/hda3

\[root@minos-spare \~\]# mdadm /dev/md5 --a /dev/hda5

**Part Five: Make Both Disks Bootable**

After reboot, the system will have booted from md1 (in this example)
with working RAID. However, if hda1 fails and the system is restarted it
will not boot. It's necessary to install the boot loader to both disks.
This section will show the procedure for the Grub boot loader. LILO will
also work, but no example is provided.

Grub may be invoked from the command line while booted from the array.

NOTE: Do not be confused! The "hd0" notation within grub is not the same
as normal device notation when referring to devices in LINUX. The
"device" command specifies which disk Grub will operate on. Grub
addresses the entire disk at the hardware level.

\[root@minos-spare \~\]# grub

GNU GRUB version 0.95 (640K lower / 3072K upper memory)

\[ Minimal BASH-like line editing is supported. For the first word, TAB

lists possible command completions. Anywhere else TAB lists the possible

completions of a device/filename.\]

grub\> device (hd0) /dev/hdc

grub\> root (hd0,0)

Filesystem type is ext2fs, partition type 0xfd

grub\> setup (hd0)

Checking if \"/boot/grub/stage1\" exists\... yes

Checking if \"/boot/grub/stage2\" exists\... yes

Checking if \"/boot/grub/e2fs_stage1_5\" exists\... yes

Running \"embed /boot/grub/e2fs_stage1_5 (hd0)\"\... 16 sectors are
embedded.

succeeded

Running \"install /boot/grub/stage1 (hd0) (hd0)1+16 p
(hd0,0)/boot/grub/stage2

/boot/grub/grub.conf\"\... succeeded

Done.

If you are creating a new array, not cloning an existing disk, you will
have to boot from the rescue CD and use grub to install the boot loader
to both disks.

**Status example:**

To check the status of your arrays use "cat /proc/mdstat". mdstat is
dynamically updated with the current status of all arrays.

Below is an example file. This file indicates that array md1 is being
rebuilt, md3 and md5 are active and synchronized and md2 is waiting to
resynchronize after md1 finishes. Arrays are always resynchronized in
sequence.

The number in square brackets indicates the order of the disks. This is
the order indicated by the \[UU\]. If one disk fails, the one which
failed is indicated by an underscore. In the case of md1 the indicator
is \[U\_\] which tells us the second disc hdc\[2\] is failed.

The progress bar is a graphic representation of the resynchronization
progress. This line also shows the current percentage finished and makes
an estimate of the remaining time for recovery.

Personalities : \[raid1\]

md1 : active raid1 hdc1\[2\] hda1\[0\]

10241280 blocks \[2/1\] \[U\_\]

\[====\>\...\...\...\...\....\] recovery = 22.7% (2326016/10241280)
finish=4.9min speed=26860K/sec

md3 : active raid1 hdc3\[1\] hda3\[0\]

2048192 blocks \[2/2\] \[UU\]

md5 : active raid1 hdc5\[1\] hda5\[0\]

16546816 blocks \[2/2\] \[UU\]

md2 : active raid1 hdc2\[2\] hda2\[0\]

10241344 blocks \[2/1\] \[U\_\]

resync=DELAYED

unused devices: \<none\>

Monitoring is automatically performed. If an array fails, mail is sent
to root. To send notification, set up mail and put your email address in
root's .forward.

mdadm has a set of diagnostic and monitoring tools as well. See the man
pages for mdadm for more details.

**Additional Topics:**

Disk replacement:

To replace a disk that has failed follow Part One except you will not
modify the partition table of the existing disk. Then add the new disk
to the array. For a failure of hdc, you would use this command for each
array:

\[root@minos-spare \~\]# mdadm /dev/md1 --a /dev/hdc1

\[root@minos-spare \~\]# mdadm /dev/md2 --a /dev/hdc2

\[root@minos-spare \~\]# mdadm /dev/md3 --a /dev/hdc3

\[root@minos-spare \~\]# mdadm /dev/md5 --a /dev/hdc5

It's not necessary to dismount disks when manipulating an already
created array. The new disk may be added live. If your hardware supports
hot adding, you will not need to shut down to replace the physical disk.

Don't forget to follow Part Five for the new disk. The boot loader
information will need to be written to the MBR.

Disk geometry and fdisk:

When configuring multiple disks, fdisk may not initially report the
correct number of heads and cylinders. This makes partitioning
difficult.

Check the manufactures specifications for your disks before using fdisk
to create partitions. If fdisk is reporting the heads and cylinders
incorrectly you can force the correct geometry by using the fdisk expert
mode. Enter fdisk normally and delete any existing partitions. Then type
"x" to enter the expert mode. Type "m" for help. "h" will set the number
of heads and "c" will set the number of cylinders. Type "r" to return to
the normal mode and create your partitions as usual. After writing ("w")
fdisk will report the correct geometry.

NOTE: You must delete the old partition table and create at least one
new partition for this to work.

New Installs:

Use Disk Druid's RAID setup tools when doing a new install. Everything
will start up very nicely. You still need to do Part Five. The second
disk does not get the MBR installed by Disk Druid.