NixOS/nixops — pull request #119 by aszlig (2013-07-08 12:25:34+00:00)
Title: Backend for Hetzner dedicated servers

This adds a backend for the German data center provider [Hetzner](http://www.hetzner.de/en/), so unlike EC2 this isn't a target for virtual machines.

The installation/management is quite different from virtual backends as there obviously isn't an API for managing storage devices. But there is an [API](http://wiki.hetzner.de/index.php/Robot_Webservice/en) for providing an IPMI-like interface with a PXE-booted rescue system. And we're using this rescue system in order to install NixOS and the corresponding deployment on the target machine.

Basically it works like this:

- Boot machine into the rescue system using the API (which gives back a generated root password to connect to the rescue system).
- Build and copy [nixpart](https://github.com/aszlig/nixpart) onto the machine.
- Create and mount partitions.
- Copy Nix to the machine and create chroot wrappers for `/mnt`.

After that, the machine stays in state `RESCUE` and the deployment continues *almost* identical to other backends but using the wrapper scripts to provide seamless copying of closures. As soon as the closures are finished with copying, the activation script is run with the `boot` argument to ensure we don't accidentally activate the configuration within the rescue system (which won't work anyway).

So after this step a reboot is required. I'm currently not sure if we should do this by default for new machines or require to specify `--allow-reboot` in `nixops deploy`. Probably makes sense in this case.

Things that are done and working:

- [x] Put the machine into rescue system.
- [x] Partition disks.
- [x] Mount filesystems.
- [x] Bootstrap basic Nix closure.
- [x] Detect networking information.
- [x] Save rescue password in MachineState.
- [x] Add possibility to use passwords in `SSHMaster`.
- [x] Allow to deploy in rescue mode.
- [x] Copy SSH public key to remote system.

Issues still left to solve:

- [ ] Don't create Tar closures with leading `/`.
- [ ] Properly register Tar closures within /mnt (so we can remove the workaround of always returning `True` in `has_really_fast_connection()`).
- [ ] Consolidate installation state into one attribute.
- [ ] Suppress warning about adding the rescue system's key to known hosts.
- [ ] Figure out a way of how to do tests and implement them.
- [ ] Detect hardware/kernel options (`nixos-hardware-scan` maybe?).
- [ ] Add a command for putting system into rescue mode.
- [ ] Only mount filesystems when machine is already installed but in rescue mode.
- [ ] Refactor `ssh_util` into a small SSH library.
- [ ] Let [nixpart](https://github.com/aszlig/nixpart) write Grub configuration (aszlig/nixpart#1).
- [ ] Properly handle route for accessing own subnet.
- [ ] Add Hetzner-specific sysctl attributes.
- [ ] Implement support for IPv6 by default.
- [ ] Maybe always reboot after finishing installation?
- [ ] Remove/refactor unneded stuff from hetzner_util.
- [ ] Allow to pass Robot user and password using environment variables.
- [ ] Write documentation about how to use the backend.
- [ ] Implement hard (reset switch) and soft (ctrl+alt+delete) rebooting.
- [ ] Only Tar-copy store paths which do not yet exist on the machine.
- [ ] Add support for destroying a machine (possibly with an option to securely wipe data).

Wishlist to the folks at Hetzner:

- [ ] Use OAuth for interacting with the API.
- [ ] Provide a way in the API to store arbitrary data (in our case this would be the machines UUID or maybe even info from the physical specification).

For the curious people out there, here is how the deployment looks like:

http://shelr.tv/records/51daad1a9660807d76000031

CC for review: @qknight, @edolstra, @rbvermaa