CA-85940: in VM.{start,resume} rely on the xapi<->xenops event sync in the error path

If an error happens during VM.start then we wish
1. xenopsd to be in a consistent state: a VM should be either completely
   intact or shutdown (see separate commit)
2. xapi to synchronise its state with xenopsd, not try to override it

To cause (2), we:
1. ensure that VM.resident_on is set in both success and failure cases
   -- this activates the event sycnrhonisation
2. inject a barrier into the event queue to block the XenAPI thread
   until the synchronisation completes.

If an error happens during VM.resume we don't set resident_on because we
wish to protect the power_state, currently_attached fields, so we can try
again later.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: log when we're about to raise a task Cancelling exception

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: add "on failure" code to all the VM lifecycle operations

After a partial failure (e.g. arbitrary process restart) xenopsd always
leaves each VM in a state which is either
  * clearly marked as broken ("domain action request = poweroff")
  * intact (Running, Paused, Suspended etc)

Therefore all we need to do after detecting a failure (or a restart)
is to trigger a {VM,VBD,VIF,PCI}_check_state operation, which will
then enqueue the appropriate corrective action i.e.
  * VM_poweroff if broken
  * nothing if intact

In particular we should not try to perform bespoke, per-operation
cleanup (e.g. VM_start -> VM_poweroff). Instead we simply invoke
the correct trigger.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: rename Task.debug_info to Task.dbg for consistency

We use the argument name "dbg" to refer to the debug token passed
along with each API call. This value is then stored in the Task.dbg
field.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: add a Task.debug_info (string * string list) and return the number of "cancel points" seen

The Task.debug_info can be used to return arbitrary data without
officially extending (and polluting) the interface.

The number of "cancel points" refers to the number of times the
task could have been cancelled.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: Copy HTTP headers "X-Http-other-config-K=V" into Task.other_config/http:K=V

This is the first step towards allowing a XenAPI client to propagate
arbitrary information (tokens, debug configuration, fault injection)
through the whole stack.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: xenopsd can now be set to trigger the cancellation of a task at a numbered cancellation point

This allows a test harness to verify that cancellation is safe.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: on xapi start, startup storage before syncing with xenopsd

The xenopsd sync will require access to the storage layer, to detach
and deactivate VDIs.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: on error in suspend, clean up the xapi metadata and force-shutdown the domain if it has already 'suspended'

The xenops event thread in xapi now handles the 'power_state = Suspended' case by
pulling the metadata from xenopsd and calling VM.remove.

If a failure happens during suspend, xapi will wait for the event sync
and check the power-state. If the VM is suspended then it will be
force_shutdown because the failure will have left the VM in an invalid
state (probably through a missing or truncated suspend VDI). If the
VM is in any other state then it can be safely left.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: omit spurious error List.failure("hd") when a VM isn't running

We expect there to be no domids when a VM isn't running-- this is a
common case and doesn't deserve to generate an exception and a warning.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: VIF.unplug now passes the cancellation test

This required a redefinition of what "plugged" means.
We now consider a VIF to be "plugged" if the resources
are still allocated by the host, as reflected by
  /xapi/.../
in xenstore.

When the udev remove event fires and the "hotplug-status"
node is removed, we consider the device as needing an
explicit unplug via the "device action request" mechanism.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: never directly remove VIF/VBD metadata in failure paths

A background thread constantly synchronises XenAPI metadata with xenopsd
metadata, so it doesn't make sense to remove objects which will be
immediately re-added.

It is ok to 'refresh' the metadata in a {VIF,VBD}.plug operation.

Redefinite what 'plugged' means for VIFs so that a VIF is always
plugged while there is still pending cleanup action (e.g. in xenstore).
This avoids a cancelled unplug leaving stale state lying around.

Clarify the "epoch_{begin,end}" APIs -- these need to know the disk
but not the whole Vbd.t

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: add a test-case to check each "cancel point"

The test-case measures the number of "cancel points" traversed by
each lifecycle operation. It then checks that, if a "cancel point"
is triggered, that the system ends up getting into a valid state.

In particular it looks for:

 * xen domains and devices
 * xenopsd VM metadata
 * xapi VM metadata

all being in sync.

Run the test on an arbitrary test VM as follows (in dom0):

/root/cancel_tests -pw <password> -vm <VM name>

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: add the concept of 'active' VBDs and VIFs to xenopsd

During a VM.start we set all VBDs and VIFs to active. Within
xenopsd a {VBD,VIF}.plug will be ignored if the device is not
active. This means that on resume, when we bring back exactly
those devices which were marked as "active" at the time of
the preceeding VM.start.

We now distinguish between a VBD_plug as part of a compound
operation and a user-requested VBD_hotplug, where the latter
also makes the device active.

We set XenAPI {VIF,VBD}.currently_attached = active || plugged
 -- this means that on suspend, currently_attached stays true
    since we never set active to "false"

Unfortunately the active flags get wiped in the middle of a
reboot, so a reboot behaves the same way as a start i.e. it
connects all devices, not just those which were connected before
the shutdown.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85940: when "cancelling" a xenguest subprocess, raise the Cancelled exception

Signed-off-by: David Scott <dave.scott@eu.citrix.com>