Diskless system

Related articles
NFS
NFS Troubleshooting
PXE
Mkinitcpio#Using net
iSCSI Boot
From
Wikipedia:Diskless node
:
A diskless node (or diskless workstation) is a workstation or personal computer without disk drives, which employs network booting to load its operating system from a server.
Server configuration
First of all, we must install the following components:
A
DHCP
server to assign IP addresses to our diskless nodes.
A
TFTP
server to transfer the boot image (a requirement of all PXE option roms).
A form of network storage (
NFS
,
Samba
or NBD) to export the Arch installation to the diskless node.
Note
dnsmasq
is capable of simultaneously acting as both DHCP and TFTP server. For more information, see the
dnsmasq
article.
You can also boot Arch Linux without using PXE at all. See the SKUF item below.
DHCP
Install ISC
dhcp
and configure it:
/etc/dhcpd.conf
allow booting;
allow bootp;

authoritative;

option domain-name-servers 10.0.0.1;

option architecture code 93 = unsigned integer 16;

group {
    next-server 10.0.0.1;

    if option architecture = 00:07 {
        filename "/grub/x86_64-efi/core.efi";
    } else {
        filename "/grub/i386-pc/core.0";
    }

    subnet 10.0.0.0 netmask 255.255.255.0 {
        option routers 10.0.0.1;
        range 10.0.0.128 10.0.0.254;
    }
}
Note
next-server
should be the address of the TFTP server; everything else should be changed to match your network
RFC:4578
defines the "Client System Architecture Type" dhcp option. In the above configuration, if the PXE client requests an x86_64-efi binary (type 0x7), we appropriately give them one, otherwise falling back to the legacy binary. This allows both UEFI and legacy BIOS clients to boot simultaneously on the same network segment.
Start ISC DHCP
systemd
service.
TFTP
The TFTP server will be used to transfer the
boot loader
, kernel, and initramfs to the client.
Set the TFTP root to
/srv/arch/boot
. See
TFTP
for installation and configuration.
Network storage
The primary difference between using NFS and NBD is while with both you can in fact have multiple clients using the same installation, with NBD (by the nature of manipulating a filesystem directly) you will need to use the
copyonwrite
mode to do so, which ends up discarding all writes on client disconnect. In some situations however, this might be highly desirable.
NFS
Install
nfs-utils
on the server.
You will need to add the root of your Arch installation to your
NFS
exports:
/etc/exports
/srv       *(rw,fsid=0,no_root_squash,no_subtree_check)
/srv/arch  *(rw,no_root_squash,no_subtree_check)
Next, start NFS services:
nfs-idmapd
nfs-mountd
.
NBD
Install
nbd
and configure it.
/etc/nbd-server/config
[generic]
    user = nbd
    group = nbd
[arch]
    exportname = /srv/arch.img
    copyonwrite = false
Note
Set
copyonwrite
to true if you want to have multiple clients using the same NBD share simultaneously; refer to
nbd-server(5)
for more details. Also use
chown
to change the ownership of the exportname directory to the user
nbd
.
Start
nbd.service
.
SKUF
You can boot Arch Linux using the
SKUF Network Boot System
project, where the root of the file system will be a
sparse file
located on
Samba
server.
To get started, install
samba
and create a configuration file:
/etc/samba/smb.conf
[global]
	workgroup = WORKGROUP
	security = user

[arch]
	path = /srv/samba
	valid users = @skuf
	write list = @skuf
	guest ok = no
	read only = no
	writeable = yes
	browseable = yes
Note
All your sparse images will lie in the specified directory (in this example
/srv/samba
).
Start
smb
systemd service
Then, create a
skuf
group and users who will be members of it and through whom
SAMBA
mounting on the client machine will happen.
# groupadd skuf
# useradd test -g skuf
# smbpasswd -a test
Client installation
Next we will create a full Arch Linux installation in a subdirectory on the server. During boot, the diskless client will get an IP address from the DHCP server, then boot from the host using PXE and mount this installation as its root.
Directory setup
Note
Creating a separate filesystem is required for NBD but optional for NFS and can be skipped/ignored.
Create a
sparse file
of at least 2 gibibytes, and create a btrfs filesystem on it (you can of course also use a real block device or
LVM
if you want).
# truncate -s 2G /srv/arch.img
# mkfs.btrfs /srv/arch.img
# export root=/srv/arch
# mount --mkdir -o loop,compress=lzo /srv/arch.img "$root"
Bootstrapping installation
Install
devtools
and
arch-install-scripts
, and run
pacstrap
to install the
essential packages
for the client:
# pacstrap -K "$root" base linux linux-firmware mkinitcpio-nfs-utils nfs-utils
Note
In all cases
mkinitcpio-nfs-utils
is still required.
ipconfig
used in early-boot is provided only by the latter.
Now the initramfs needs to be constructed.
NFS
Trivial modifications to the
net
hook are required in order for NFSv4 mounting to work (not supported by
nfsmount
– the default for the
net
hook).
# sed s/nfsmount/mount.nfs4/ "$root/usr/lib/initcpio/hooks/net" > "$root/usr/lib/initcpio/hooks/netnfs4"
# cp $root/usr/lib/initcpio/install/net{,nfs4}
The copy of
net
is unfortunately needed so it does not get overwritten when
mkinitcpio-nfs-utils
is updated on the client installation.
Edit
$root/etc/mkinitcpio.conf
and add
nfsv4
to
MODULES
,
netnfs4
to
HOOKS
, and
/usr/bin/mount.nfs4
to
BINARIES
.
Next, we
chroot
our installation and run
mkinitcpio
:
# arch-chroot "$root" mkinitcpio -p linux
Note
After recent changes, your system may fail to boot properly. If your diskless client repeatedly displays the following error during boot:
SIOCGIFFLAGS: No such device
this usually means that the
ipconfig
command in the
netnfs4
hook is being invoked with the
ip=dhcp
parameter, which is currently broken.
To fix the issue, change the parameter to:
ip=:::::eth0:dhcp
After applying this change, the system should boot normally.
NBD
The
mkinitcpio-nbd
AUR
package needs to be installed on the client. Build it with
makepkg
and install it:
# pacman --root "$root" --dbpath "$root/var/lib/pacman" -U mkinitcpio-nbd-0.4-1-any.pkg.tar.xz
You will then need to append
nbd
to your
HOOKS
array after
net
;
net
will configure your networking for you, but not attempt a NFS mount if
nfsroot
is not specified in the kernel line.
SKUF
To install Arch Linux on
sparse file
using
SKUF Network Boot System
, clone the
git
repository:
$ git clone
https://github.com/BiteDasher/skuf.git
$ cd skuf
$ ./switch-tag latest
Then, build the
skuf
package and
ISO image
which will later be used as a "kickstart" to start the main system using
kexec
First of all, you need to tune the method of encrypting your passwords for SAMBA (see
[1]
for more details):
tune.crypt
A B
X Y
I O
tune.password
1234 Test password!
Set up
defaults
:
defaults
SAMBA_USERNAME="testuser"
SAMBA_PASSWORD="pa33w0rd"
SAMBA_ADDRESS="192.168.0.5"
SAMBA_PORT="445"
SAMBA_VERSION="3.0"
...
Install required packages:
# ./install_deps.sh
And finally, build
skuf
package:
$ ./tune_crypt.sh
$ ./tune_password.sh
$ ./setup_defaults.sh
$ ./build_rootfs_tar.sh
$ ./build_package.sh
ISO image:
# ./setup_repo.sh
# ./build_iso.sh
And
sparse file
with Arch Linux:
# ./create_image.sh -s
SIZE_IN_GIGABYTES
additional_packages
Then, move
arch.ext4
in
/srv/samba
.
Client configuration
In addition to the setup mentioned here, you should also set up your
hostname
,
timezone
,
locale
, and
keymap
, and follow any other relevant parts of the
Installation guide
.
Boot loader
GRUB
This article or section is a candidate for merging with
GRUB
.
Notes:
(Discuss in
Talk:Diskless system
)
Though poorly documented, GRUB supports being loaded via PXE.
# pacman --root "$root" --dbpath "$root/var/lib/pacman" -S grub
Create a grub prefix on the target installation for both architectures using
grub-mknetdir
.
# arch-chroot "$root" grub-mknetdir --net-directory=/boot --subdir=grub
Luckily for us, grub-mknetdir creates prefixes for all currently compiled/installed targets, and the
grub
maintainers were nice enough to give us both in the same package, thus grub-mknetdir only needs to be run once.
Note
This example config assumes the NFS/NBD server's IP being
10.0.0.1
Now we create a trivial GRUB configuration:
# vim "$root/boot/grub/grub.cfg"
menuentry "Arch Linux" {
    linux /vmlinuz-linux quiet add_efi_memmap ip=:::::eth0:dhcp nfsroot=10.0.0.1:/arch
    initrd /initramfs-linux.img
}

menuentry "Arch Linux (NBD)" {
    linux /vmlinuz-linux quiet add_efi_memmap ip=:::::eth0:dhcp nbd_host=10.0.0.1 nbd_name=arch root=/dev/nbd0
    initrd /initramfs-linux.img
}
GRUB
will
set root=(tftp,10.0.0.1)
automatically, so that the kernel and initramfs are transferred via TFTP without any additional configuration, though you might want to set it explicitly if you have any other non-tftp menuentries.
Note
All GRUB files and initcpio files must be available through TFTP. For example, for an NBD install with TFTP root set to
/srv/tftp
,
/srv/tftp/grub/x86_64-efi/core.efi
,
/srv/tftp/vmlinuz-linux
must be present to boot successfully. You may copy all the
/boot
files inside the image to TFTP server's root.
You may generate grub config by
grub-mkconfig
to ensure video settings are set correctly. However, it is needed to edit
boot.cfg
afterwards, to remove
search --no-floppy ...
and ensure
linux
initrd
options (paths, NBD settings, NFS settings) are set correctly.
Modify your kernel line as-necessary, refer to
PXELINUX
for NBD-related options.
PXELINUX
PXELINUX is provided by
syslinux
, see
PXELINUX
for details.
Additional mountpoints
NBD root
In late boot, you will want to switch your root filesystem mount to both
rw
, and enable
compress=lzo
, for much improved disk performance in comparison to
NFS
.
# vim "$root/etc/fstab"
/dev/nbd0  /  btrfs  rw,noatime,compress=lzo  0 0
Client boot
NBD
If you are using NBD, the need to unmount
arch.img
on the server before booting a client depends on your configuration:
If
copyonwrite
is disabled, you must unmount
arch.img
before booting any client. Simultaneous read-write access to the base image will cause corruption.
If
copyonwrite
is enabled, clients only read from the base image and write to a separate cache, so a read-only server-side mount is theoretically safe. A read-write server-side mount is still discouraged while clients are active, as any modification to the base image during a live session risks inconsistency.
This makes things particularly interesting when it comes to kernel updates. You cannot have your client filesystem mounted while you are booting a client, but that also means you need to use a kernel separate from your client filesystem in order to build it.
You will need to first copy
$root/boot
from the client installation to your tftp root (i.e.
/srv/boot
).
# cp -r "$root/boot" /srv/boot
You will then need to umount
$root
before you start the client.
# umount "$root"
Note
To update the kernel in this setup, you either need to mount
/srv/boot
using NFS in fstab on the client (prior to doing the kernel update) or mount your client filesystem after the client has disconnected from NBD.
copyonwrite permissions
Warning
When
copyonwrite
is enabled,
nbd-server
runs as the user defined in  the
[generic]
section of
/etc/nbd-server/config
(commonly
nbd
). This user must have write permission on the directory containing the exported image, as COW cache files are written there by default. Without this, clients will fail to boot with the misleading error
received invalid negotiation magic
, which gives no indication that the root cause is a permissions issue.
To fix this, either transfer ownership of the export directory to the
nbd
user:
# chown nbd: /path/to/export/directory
Or redirect COW cache files to a dedicated directory that the
nbd
user already owns, by adding to:
/etc/nbd-server/config
cowdir=/path/to/cow/cache
Ensure that directory is also owned by
nbd
.
SKUF
Write
skuflinux-smth.iso
to your USB drive, plug it in client computer and select in
UEFI
/BIOS settings as a boot device.
Note
If the client computer has
UEFI
, you can install
SKUF
on a
ESP
partition so you don't have to use a USB flash drive/CD/DVD. To do this, mount
skuflinux-smth.iso
somewhere (like /mnt), then copy
/mnt/skuf/boot/x86_64/{vmlinuz-linux,initramfs-linux.img}
to
ESP
partition and execute
efibootmgr -c -d /dev/sdX -p Y -u 'initrd=\initramfs-linux.img' -l '\vmlinuz-linux' -L 'SKUF'
where
/dev/sdX
is the target disk and
Y
is the target
ESP
partition number.
See also
kernel.org: Mounting the root filesystem via NFS (nfsroot)
syslinux.org: pxelinux FAQ