dberzano/elastiq — pull request #2 by dberzano (2014-03-24 19:30:39+00:00)
Title: Deal with deployment errors

elastiq now deals with deployment errors. Errors are handled by keeping a permanent state of the instance IDs launched by elastiq: this state is saved to a text file (one instance ID per line). By default, the state file is located at:

* `$HOME/.elastiq/state` in case it is launched by an unprivileged user;
* `/var/lib/elastiq/state` in case it is launched by root.

Two types of deployment errors are considered.

### Virtual machines in "error" state

Some virtual machines might never boot and go to an EC2 "error" state. In some cases (like OpenStack) such virtual machines are not cleaned up automatically by the cloud, and take up quota.

* VMs are periodically checked for the "error" state
* VMs in "error" are redeployed (*i.e.*, they are terminated and a new one is requested)

The period of error check can be configured via the configuration file:

```{ini}
[elastiq]
check_vms_in_error_every_s = 20
```

### Virtual machines do not join the cluster in time

Some virtual machines correctly boot but they do not join the cluster "in time". After a time defined by the variable:

```{ini}
[elastiq]
estimated_vm_deploy_time_s = 600
```

if the virtual machine has not joined the batch cluster yet, a termination is triggered. If the termination fails, it will retry to terminate it in another 10 seconds.

**Note:** in this case no new virtual machine is requested to replace the terminated one. This is because a "late" virtual machine is allegedly due to a misconfiguration and not an infrastructure glitch. Redeployment will be anyway retriggered, if necessary, by the "scale up" mechanism of elastiq.