[xapi] avoid fetching unnecessary events from xenopsd when monitoring a task

If we call UPDATES.get None then we get an event on every object in the system.
Instead we only need to:

  last_id <- UPDATES.last_id
  TASK.stat
  UPDATES.from last_id

which will fetch only new events, and only in the case where the task hasn't
already finished.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
[xapi] Only update the storage.db if something has actually changed

We implement DP.destroy by iterating over every active VDI. If there
are 1000 active VDIs then we were pointlessly writing the database
1000 times. This saves 15s per DP.detach.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-83076: avoid xapi and xenopsd both "owning" the Vbd.backend xenopsd field

The xapi/xenopsd separatation needs to be strictly as follows:

1. all the static VM metadata (Vm.t, Vbd.t, Vif.t, Pci.t) is owned and
   updated by xapi, whenever the user requests it via the XenAPI

2. all the dynamic VM state (Vm.state, Vbd.state, Vif.state, Pci.state)
   is owned and updated by xenopsd. Xapi will receive events on this
   and update some data in the pool database.

In particular a xenopsd API call (Vbd.insert, Vbd.eject) must not write
any static metadata directly, since this will conflict with parallel
updates from xapi.

Furthermore all dynamic VM state updates from xenopsd need to be "complete"
and not require correlating with static VM state. For example the disk
"backend" which is currently associated with a CDROM must be completely
described in the Vbd.state.

The XenAPI VBD.insert/VBD.eject control flow is now:

1. xapi calls xenopsd with VBD.insert/eject
2. xenopsd performs the xenstore-level operation (including stashing away
   the VDI in xenstore)
3. xenopsd generates an event on the VBD
4. xapi receives the VBD event and calls VBD.stat
5. xapi sets VBD.{empty, VDI} in the pool database based on the VBD.stat
   results
6. xapi inserts a barrier event to ensure all events are processed

After step (5), a XenAPI event will be generated which will be received
by the event.from in xapi_xenopsd, which will cause xapi to push the
new Vbd.t configuration to xenopsd, ensuring that the insert/eject will
take effect on next reboot.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-67536: Remove unplug_force from vif_operations

The unplug_force operation was recently added to the vif_operations enum.
However, this causes problems during rolling pool upgrade, while the
master has been upgraded, but some of the slaves have not. Operations
such as Db.VIF.get_record on running VMs were be broken by this, which broke
live migrations between un-upgraded hosts.

After this patch, the "unplug" operation is used for both "unplug" as well
as "unplug_force" in VIF.allowed_operations, which is fine, since they are
always allowed at the same time.

Signed-off-by: Rob Hoes <rob.hoes@citrix.com>
networkd: split off config file functions into separate module

Signed-off-by: Rob Hoes <rob.hoes@citrix.com>
networkd: add a tool that returns networkd's configuration from

The tool reads /var/xapi/networkd.db and outputs the configuration
for a given bridge and/or interface. This is useful for the upgrade
path of the host installer.

Example:

$ networkd_db -bridge xapi0 -iface xapi0
interfaces=eth0,eth1
mode=static
ipaddr=10.80.228.25
netmask=255.255.240.0
gateway=10.80.224.1
dns=10.80.16.125,10.80.16.126
domain=uk.xensource.com
modev6=static
ipv6addr=fd00::666/64
gatewayv6=fd00::1

Signed-off-by: Rob Hoes <rob.hoes@citrix.com>
CA-83040: choose_suspend_sr ignores SRs not shared with host

When a VM was migrated within the pool from local storage to local storage, the
suspend SR was not reset, which caused suspend to fail because the VM's
suspend_SR was not visible to the host on which it was being suspended. We fix
the problem by modifying choose_suspend_sr to ignore a VM's suspend_SR field if
that SR has no PBD linking it with the VM's resident host.

Another commit will follow which blanks out the suspend_SR field on a VM when
it's suspend SR will no longer be visible after a migrate.

Signed-off-by: Mike McClurg <mike.mcclurg@citrix.com>
whitespace: delete trailing space

Signed-off-by: Mike McClurg <mike.mcclurg@citrix.com>
CA-83040: fix suspend_SR on intra-pool storage migrate

Signed-off-by: Mike McClurg <mike.mcclurg@citrix.com>
CA-83040: fix suspend_SR on VM metadata import

Signed-off-by: Mike McClurg <mike.mcclurg@citrix.com>
CA-78558: Be robust against absent PIF_metrics

Signed-off-by: Rob Hoes <rob.hoes@citrix.com>
CA-83076: quicktest now checks that VBDs have currently_attached = true after VM.start

It's useful to explicitly check this -- it's surprising how far you can get with a broken VBD.stat

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-83076: add the same xenstore key in VBD.plug as we do in VBD.insert

Without this VBD.stat is unable to determine the VDI from the VBD, breaking
VBD hotplug.

Signed-off-by: David Scott <dave.scott@eu.citrix.com>
CA-85168: Initialize dom0 memory_dynamic_min and memory_dynamic_max with some sensible values

Signed-off-by: Thomas Gazagnaire <thomas.gazagnaire@ocamlpro.com>
networkd: Improve exception handling when reading/writing config files

Signed-off-by: Rob Hoes <rob.hoes@citrix.com>
CA-83040: Simplify check_sr_exists_for_host

check_sr_exists_for_host had some nested-if spaghetti logic that was
unessessary. We simplify it by just calling host_has_pbd_for_sr directly, since
that function returns false if the SR isn't valid.

Signed-off-by: Mike McClurg <mike.mcclurg@citrix.com>
CA-85168: xapi deal with memory values in bytes, xenops in KiB

Signed-off-by: Thomas Gazagnaire <thomas.gazagnaire@ocamlpro.com>
Merge commit '6d5eb1806e500a6af9cc18bc0e80cd72adcd39b3'
Merge commit '259dd8f2e3a827a3dfe8485f7f8e0d3959da51c0'
CA-77238: prevent even broadcast traffic flowing to/from VM when locking_mode==disabled