Add a new ResourceState called RESCUE.

This is needed for backends which are targeted towards datacenter providers that
provide a (usually PXE-booted) rescue system, which is not only useful in case
the server is unavailable but also for installation of NixOS :-)

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Add new basic stub Hetzner backend.

This backend currently doesn't do anything really useful* but just provides the
necessary boilerplate with a few options. The specification of the partitions
isn't final and serves no purpose at the moment.

One thing to note is the hetzner_utils module, which is essentially a stripped
down coppy of my python robot library from:

https://github.com/RedMoonStudios/hetzner

This library will be gradually stripped down and expanded to meet our specific
needs.

[*] Actually _one_ thing the current backend does: It reboots the corresponding
system into a 64bit Debian GNU/Linux rescue system on "nixops deploy".

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
SSHMaster: Allow to use keyboard-interactive auth.

This adds an additional keyword argument called "password" to directly supply a
password which then is passed as an environment variable NIXOPS_SSH_PASSWORD to
a temporary helper script, which in turn is passed to SSH using the SSH_ASKPASS
environment variable.

Thanks to @edolstra for suggesting this, he already has used this way of passing
passwords to SSH in NixOS/nix@bfec245efa68f010b8687c1eee8dcf909c4d1343.

Unlike to cygwin we don't have the luxury of most Unix things being unavailable
(such as PTYs or setsid maybe?), so SSH is trying harder to avoid what we're
trying to do.

Thus we setsid() after forking the main process and redirect stdin to /dev/null
in order to make sure SSH won't find anything that remotely looks like it could
be a TTY.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
hetzner: Use kickstart syntax for partitioning.

As mentioned a few times in commits from nixos/nixpkgs, this syntax is
preliminary at the moment. But it's at least better than what i had in mind so
far, so let's use it here.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
MachineState: Use SSH master in generate_vpn_key.

This method didn't use ssh_master.opts, so it will prompt for a password even
though we only can authenticate using keyboard-interactive authentication and
the temporary password is unavailable.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
hetzner: Handle rescue system as a machine state.

Using the rescue system as a normal machine state really makes sense, because
we can push all closures into the machine's storage devices and activate for the
next reboot.

This is especially useful if things went bad and the machine is unable to boot
into the normal system.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Drop now obsolete dependency on paramiko.

We're using a SSH_ASKPASS workaround now, so we don't need the additional
dependency.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Move SSHMaster class into nixops.ssh_util.

This is in preparation to factor out the whole SSH handling part into its own
class. We don't want to end in a bigger mess that we were at the beggining.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
MachineState: Supply master to copy_closure_to.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
backends.__init__: Move exceptions into ssh_util.

We want to correctly encapsulate all SSH specific stuff into ssh_util one day,
so moving exceptions over there makes perfect sense to me.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
MachineState: Use stdin fileobj in run_command.

The keyword argument stdin_string seems to be unused, so it's a great
opportunity to replace it with just "stdin", which takes a Python file-like
object.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
MachineState: Re-add stdin_string to _logged_exec.

We finally really need stdin_string, so let's add it back to _logged_exec. But
let's close stdin immediately after writing, because otherwise and unlike stated
in the FIXME, we _will_ get a deadlock even when the pipe buffer is large
enough.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
scripts/nixops: Add SSH master to ssh(for_each).

When the machine is in rescue state, we cannot ssh into the rescue system
without having the SSH master connection attached to it.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
hetzner: Install partitioner and run in rescue.

We already had the ability to boot into the rescue system, now we're actually
doing something, namely: partitioning. The partitioner itself spits out a Nix
expression with the created filesystems which can be used to provide a physical
base (fileSystem/swapDevices) specification of the machine. All we need to do
now is to deploy into the chroot and we're finished with the basics.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
hetzner: Comply to PEP8.

There was one overlong line in there, which annoyed me all the time.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
wait_for_ssh: Correctly handle RESCUE state.

If the machine already is in RESCUE state, don't try to set it back to UP state,
because this will change the way SSH is handling connections to the target
machine and thus it becomes inaccessible.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
run_command: Allow to pass env as a keyword arg.

We already have support for this kwarg in _logged_exec(), so it makes sense to
add it to run_command() as well, especially because we're going to need it in
the Hetzner backend.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
run_command: Unset locale when in RESCUE state.

Most likely we use chroot to access the real system from the rescue system, so
the locales on the host system are most likely unavailable within the chroot and
produce a ton of errors.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
hetzner: Add FS info to physical expression.

Our partitioner is going to output a Nix expression containing the fileSystem
and swapDevices attributes, so obviously we need to add the output to our
physical specification as well and store it into the machine state.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
deployment: Always activate for boot in RESCUE.

When the machine is in RESCUE state, we most obviously don't want to activate
the configuration within the rescue system, so we need to require a reboot in
order to activate the configuration.

This also fixes a few overlong lines to comply to PEP8.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>