[sparse_dd] add a "null:" output writer for speed testing

sparse_dd -src ... -dest null: -size ...

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
[sparse_dd] switch to pipelined NBD requests, increasing max throughput

On a disk with 90MiB/sec max bandwidth, a pipeline of length 1 achieved
30MiB/sec. A pipeline of infinite length achieved 89MiB/sec.

Unfortunately the pipeline code can't be activated because tapdisk
appears to be not sending the last few replies.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-84687: Fixing CORS headers that were added before HTTP status line.

Signed-off-by: Rabin Karki <rabin.karki@citrix.com>
CA-83837: lookup a domid from a VM uuid via xenstore rather than the domain info list

Unfortunately some libraries (such as libxenguest) will destroy a domain
beneath us, causing us to lose track of the domid corresponding to a uuid.
We are in control of xenstore, so we use the /vm/uuid/domains map to find the
lookup the domid. Note this also means that 'add' has to add such a key for
already-running VMs (such as dom 0)

Without this change a VM.suspend will leave VDIs attached.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-83837: if a cancellable xenstore watch simultaneously succeeds/fails/is cancelled, apply a prioritisation policy.

We consider that an operation which has succeeds (but simultaneously/historically has failed or been cancelled) has actually succeed.
We consider that an operation which has not succeeded but failed (but simultaneously/historically been cancelled) has actually failed.

So the event priority is:
  success > failed > cancelled

Without this, a cancel that arrives late may be said to have actually cancelled an operation which has already terminated with success/faulure (because cancel is always asynchronous).
Without this, an operation which failed in the past but then later succeeds (eg VBD.unplug) might be said to be still a failure even though it has now completed successfully.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-83837: watch the "hotplug-status" node as well as the "shutdown-done" node

The old XenServer/XCP-only protocol used the "shutdown-done" node, while
upstream in other kernels (such as PVops) they use the "hotplug-status"
key instead.

We need to watch the "hotplug-status" key and if the device is removed,
generate the correct event and clean up any resources allocated.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-83837: VBD_unplug is now much more robust

VBD_unplug may be called when the domain has been destroyed (in suspend
by libxenguest) and when the xenstore device has already been shutdown.

All the cleanup steps are now Opt.iters over option types, to do as
much cleaning as possible. Note this depends on the new 'domid_of_uuid'
function which uses xenstore rather than the domain info list.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
[sparse_dd] send NBD DISCONNECT after last reply has been received

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
[sparse_dd] wait for the last reply before closing the connection

Bug tracked down and fixed by: Jonathan Ludlam <jonathan.ludlam@eu.citrix.com>
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
[sparse_dd] allow the zero scanner's "quantum" to be set via the command-line

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
[sparse_dd] allow the read blocksize to be set via the command-line

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
[sparse_dd] record round-trip latency, and overall copy speed

Also remove some of the more egregious log-spam.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
[sparse_dd] read default configuration params from /etc/sparse_dd.conf

This makes it possible to tweak the values in the field if something
underperforms.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
[sparse_dd] switch between 'Buffer', 'Human' and 'Machine' logging modes

Initially we buffer all debug output until the command-line options
are parsed, at which point we can flush the logging buffer.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-84673: Metadata import should create a network if it can't find one

This already happens for "normal" VM imports. There is no good reason
why a metadata import should fail if it cannot find a network, rather than
creating an internal network.

This will prevent issues during, e.g., cross pool migration, where metadata
export+import is used to migrate VM metadata to the remote pool.

Signed-off-by: Rob Hoes <rob.hoes@citrix.com>
CA-83711: Stop generating spurious backslashes
Merge commit '7f2e39cd66cc84449485e990cefed1b8c741d18d'
Merge pull request #718 from djs55/nbd-pipeline

CA-84843: Add request pipelining to sparse_dd
Merge commit 'f8135d9d148b4b36e8b621904c0e2d1ea5988182'
Merge commit 'ca911fd9de6b9f69aeeee6a4dbe66524ac2240d2'