# Presentation: 129591

## Notes from Installing a Mac G5 Cluster at SLAC

- Chuck Boeheim
- SLAC Computing Services

## Background

- New joint Stanford/SLAC department: Kavli Institute for Particle Astrophysics
- Choices for compute cluster included joining Opteron purchase, or Mac G5 cluster
- Astronomers like Macs
- Compatible with laptops and desktops they were buying

## Configuration

- 2 Mac G5 file servers
- 2 2.5TB Apple XServ Raid units each
- 2 Mac G5 interactive servers
- 10 Mac G5 compute nodes
- No graphics cards
- No CD drives
- Plus two development servers for us

## Physical

- Good overall engineering
- Good airflow; quiet
- Hot swap disks in server nodes
- Rails fit standard racks well
- Easy to mount

## Management

- Supports serial console (57,600 baud)
- No graphics card in G5 server
- Supports serial bios
- But must press button to boot in bios
- No power management
- Supports network install
- Initial disk partition and boot a little tricky
- Subsequent installs can be fully automated

## Installing the first server

- “Secret handshake”
- Install from CD
- Run Server Assistant on another Mac to start install, and again to configure
- Run Apple Remote Desktop to get graphic login
- Command line alternatives are available, though obscure

## The Partitioning Conundrum

- System has initial OS with one 250GB partition
- Cannot re-partition the boot drive
- One solution:
- Boot server from CD
- Ssh to server
- Partition the disk
- Diskutil partitionDisk disk0 3 “JournaledHFS+” System 10G “JournaledHFS+” Cache 1G “JournaledHFS+” Work 200G
- Netboot may also be possibility
- Fastest solution: insert disks in 2nd and 3rd bays of server and format

## Reference Server

- Install second server
- Mac OS X Server 10.3.5
- All patches
- Xcode tools
- Local directory configuration
- AFS
- Fink
- Shutdown, move hard disk to bay 2 of netboot server

## Make a Network Install Master

- Netboot serves disk images
- Uses DHCP, BSDP1, TFTP, and NFS to serve disk image to target machine
- Use Network Image Utility to make image of reference disk
- Turn on Netboot in Server Admin
- Boot client from network (different “secret handshake”)
- 1 Boot Server Discovery Protocol

## Network Install Sequence

- Target Server

- Netboot Server

- DHCP
- broadcast

- Configure with DHCP
- helper address

- DHCP
- response

- BSDP
- broadcast

- BSDP response

- Router

- TFTP bootloader

- NFS disk image

- DHCP Server

## Netboot Issues

- Our standard DHCP server worked just fine.
- BSDP is an extension of DHCP
- We put on same subnet as cluster for simplicity
- Should be able to use router helper to put on a different subnet
- Network Image Utility had some conflicts

## Users: Looks like Unix

- Use NIS, LDAP, Kerberos for accounts
- Most utilities present: ssh, bash, perl, emacs, X-windows
- Fink supplies most gnu tools

## Sysadmins: Sorta like Unix

- Strong BSD heritage
- Some things in different places than Linux
- GUI tools for configuring
- Most, but not all, can configure a remote server
- Serversetup and networksetup commands can configure most settings
- Directory Access (setup NIS or LDAP) seems not to be scriptable

## Startup Befuddlement

- Not inittab or /etc/rc.d based
- SystemStarter starts many daemons
- Ssh starts out of xinetd
- Ypbind started when needed by Directory Services
- Watchdog starts server processes
- /etc/watchdog.conf
- Sorta like inittab
- Mac OS X 10.4 (Tiger) is going to introduce yet another way

## Authentication Compatibility

- Can use NIS, LDAP, Kerberos, Active Directory
- Supports PAM, but not everything uses it
- LoginWindow (and a few other things) are directly kerberos-aware and don’t use PAM

## Infrastructure Fit

- Taylor (SLAC’s config tool) ran with little modification
- About half the config modules worked with little change
- The remainder took completely Mac OS X specific rewrites

## Nits

- /etc/passwd is there, but not used.  Uses netinfo instead for local accounts
- Shadow passwords in different place and format than linux, solaris
- afs permissions: default is to copy owner mode bits to group and other
- Copy file from afs to local, ends up world-readable
- Change ‘realmodes’ in /var/db/openafs/etc/config/settings.plist
- Mac HFS file system is not case-sensitive: Makefile and makefile are the same!

## Conclusions

- Good hardware package, but lacking power management
- Network install suitable for cluster operation, but still a few wrinkles
- Generally good configuration management tools, but some divergence from standard Unix tools