CA-85940: in VM.{start,resume} rely on the xapi<->xenops event sync in the error path

If an error happens during VM.start then we wish
1. xenopsd to be in a consistent state: a VM should be either completely
   intact or shutdown (see separate commit)
2. xapi to synchronise its state with xenopsd, not try to override it

To cause (2), we:
1. ensure that VM.resident_on is set in both success and failure cases
   -- this activates the event sycnrhonisation
2. inject a barrier into the event queue to block the XenAPI thread
   until the synchronisation completes.

Note that other VM lifecycle operations already worked like this becausee
VM.resident_on is set throughout the other operations. The difference here
is that VM.{start,resume} create a VM from scratch and, if we set
resident_on at the beginning, the event thread would immediately resynchronise
and delete the xenopsd metadata as soon as it saw a "Halted" VM.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: log when we're about to raise a task Cancelling exception

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: add "on failure" code to all the VM lifecycle operations

After a partial failure (e.g. arbitrary process restart) xenopsd always
leaves each VM in a state which is either
  * clearly marked as broken ("domain action request = poweroff")
  * intact (Running, Paused, Suspended etc)

Therefore all we need to do after detecting a failure (or a restart)
is to trigger a {VM,VBD,VIF,PCI}_check_state operation, which will
then enqueue the appropriate corrective action i.e.
  * VM_poweroff if broken
  * nothing if intact

In particular we should not try to perform bespoke, per-operation
cleanup (e.g. VM_start -> VM_poweroff). Instead we simply invoke
the correct trigger.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: rename Task.debug_info to Task.dbg for consistency

We use the argument name "dbg" to refer to the debug token passed
along with each API call. This value is then stored in the Task.dbg
field.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: add a Task.debug_info (string * string list) and return the number of "cancel points" seen

The Task.debug_info can be used to return arbitrary data without
officially extending (and polluting) the interface.

The number of "cancel points" refers to the number of times the
task could have been cancelled.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: Copy HTTP headers "X-Http-other-config-K=V" into Task.other_config/http:K=V

This is the first step towards allowing a XenAPI client to propagate
arbitrary information (tokens, debug configuration, fault injection)
through the whole stack.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: xenopsd can now be set to trigger the cancellation of a task at a numbered cancellation point

This allows a test harness to verify that cancellation is safe.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: on xapi start, startup storage before syncing with xenopsd

The xenopsd sync will require access to the storage layer, to detach
and deactivate VDIs.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: on error in suspend, clean up the xapi metadata and force-shutdown the domain if it has already 'suspended'

The xenops event thread in xapi now handles the 'power_state = Suspended' case by
pulling the metadata from xenopsd and calling VM.remove.

If a failure happens during suspend, xapi will wait for the event sync
and check the power-state. If the VM is suspended then it will be
force_shutdown because the failure will have left the VM in an invalid
state (probably through a missing or truncated suspend VDI). If the
VM is in any other state then it can be safely left.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: omit spurious error List.failure("hd") when a VM isn't running

We expect there to be no domids when a VM isn't running-- this is a
common case and doesn't deserve to generate an exception and a warning.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
Fix e3aaeabd04e959e85b66a7e7a2a777820f771974

VM.start:
 we set resident_on in the failure path so the cleanup happens properly

VM.resume
 we don't set resident_on until it succeeds in order to protect
 power_state, currently_attached, allowing us to try another resume later
CA-85940: VIF.unplug now passes the cancellation test

This required a redefinition of what "plugged" means.
We now consider a VIF to be "plugged" if the resources
are still allocated by the host, as reflected by
  /xapi/.../
in xenstore.

When the udev remove event fires and the "hotplug-status"
node is removed, we consider the device as needing an
explicit unplug via the "device action request" mechanism.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: if a VIF.plug fails (e.g. due to cancel) then remove the metadata from xenopsd

The principle is: xenopsd only knows about a VIF (or VBD) if it
is currently "plugged". Therefore if a VIF.plug fails, we should
make sure xenopsd forgets about it.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
Add awesome cancellation test
More infrastructure for the cancel test
Experimental plug/unplug work
VIF and VBD metadata should only be added/removed by the event thread; however its ok to refresh before a plug
cancel_test improvements
CA-85940: add the concept of 'active' VBDs and VIFs to xenopsd

During a VM.start we set all VBDs and VIFs to active. Within
xenopsd a {VBD,VIF}.plug will be ignored if the device is not
active. This means that on resume, when we bring back exactly
those devices which were marked as "active" at the time of
the preceeding VM.start.

We now distinguish between a VBD_plug as part of a compound
operation and a user-requested VBD_hotplug, where the latter
also makes the device active.

We set XenAPI {VIF,VBD}.currently_attached = active || plugged
 -- this means that on suspend, currently_attached stays true
    since we never set active to "false"

Unfortunately the active flags get wiped in the middle of a
reboot, so a reboot behaves the same way as a start i.e. it
connects all devices, not just those which were connected before
the shutdown.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
cancel_tests