nix/hetzner: Set grub timeout to 1.

This reduces reboot time by 4 seconds, so with a minimal system, reboot
took 65 seconds and it now (quite obviously because the timeout was 5
seconds before) takes 61 seconds.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
nixops reboot: Add flag to enable rescue system.

Whenever a machine has the possibility to boot into a rescue system,
specifying --rescue to "nixops reboot" should activate the rescue system
and wait until the machine has finished booting into it.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
hetzner: Implement reboot_rescue().

Also, document the implications on server.reboot("hard"), because we're
now not only using this for installation purposes only. Triggering a
hard reboot is definitely a bad idea on production systems.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
hetzner: Use rescue bootstrapper for mounting.

Move most of the stuff we used to have in _install_base_system() into
the new method _bootstrap_rescue(), which only does partitioning if the
install argument is set to true. This makes it possible to use the same
routine for "nixops reboot --rescue" as well as for installation of the
machine.

The downside is, that at the moment the full bootstrap installer is
moved to the rescue system, even in case we already have a fully
installed NixOS laying around. We already have a task for that so it is
on our roadmap and will be fixed soon[TM]. :-)

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
hetzner: Only do a hard reset if necessary.

Hard reboot is only necessary if we are neither in state UP nor in state
RESCUE, which means, that the machine has gone past lunch and it's okay
to virtually push the reset button.

If we are going to install, we're always using hard reset, as we don't
even know the current state.

For normal rebooting, we don't use self.reboot here, because it's going
to put us in self.STARTING state, which we want to avoid when booting
into rescue system.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
ssh_util: Add reset() method to reset SSHMaster.

This is needed when for example flags or passwords were changed in the
MachineState. We're using weakrefs here, to ensure that even when the
master is left referenced somewhere in the wild, it will still be the
only reference left within SSH.

Probably the best method to solve this is using context managers, but
that's entirely out of the scope of this branch and will be something to
work on afterwards.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
hetzner: Reset SSHMaster when booting into RESCUE.

If the rescue system is already active or the system is up and we're
booting into the rescue system (again), we still have the old SSH master
process active, so let's reset.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
hetzner: Install and reboot without allow_reboot.

The flag --allow-reboot is no longer needed for the first installation.
There is now a single state (justInstalled), which represents whether
we're finished with the installation and the rest of the deployment
could go on including the final reboot.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
MachineState: Always reset SSH after reboot.

In most cases the SSHMaster connection will be closed anyway, but let's
make sure that no reference is hanging around.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>