Placate warnings on ganeti.outils_unittest.py

Change ovf.py from using 'if foo:' to 'if foo is not None:' checks as
suggested by the FutureWarnings the OVF library raises otherwise.
This fixes issue 557.

Signed-off-by: Santi Raffa <rsanti@google.com>
Reviewed-by: Jose Lopes <jabolopes@google.com>
Hotplug: objects and constants additions

Add pci slot in NIC/Disk objects. This slot will be used only
by hypervisor code. Currently only KVM will use it and store it
temporarily in runtime files.
Add HOTPLUG_* constants to define device types an hotplug actions.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Introduce _GenerateKVMBlockDevicesOptions()

Simply move logic that generates block device related
options out of _GenerateKVMRuntime().

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
block_devices = (disk, link, uri)

Do not pass BlockDev instance to hypervisor. Instead calculate the
drive_uri (if any) in backend level and pass only the string to
hypervisor.

Hypervisor should not be aware of the entire block device but only
the final path it can be reached.

This is done also for easy manipulation of block devices inside
runtime files since hotplug support enforces separate entry and
BlockDev is not serializable.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Prepare kvm options to support -device for disks

Newer kvm version support -device driver,... option where available
drivers can be found by running kvm -device ? command. Currently
ganeti uses `if` property of driver to define the disk type to use
which is also passed as hvparam. This patch prepares the use of
-device option only for paravirtual disks. To support other drivers
as well we must create a mapping between the values passed now as
disk_type hvparam and existing drivers. Note that if -device is used
`if` property of `drive` option should be `none`.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Prepare block_devices runtime entry

With this patch we add another entry in runtime files along with
kvm_cmd, kvm_nics, and hvparams. block_devices that used to be
encapsulated inside kvm_cmd, live now separately just like nics do
but in tupples of (L{objects.Disk}, dev_path).

Introduce also _GetExistingDeviceInfo() helper function to search
for runtime entries. This is going to be useful later in hotplug
methods.

Define new exception: HotplugError.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helper methods for PCI slots and device ids

Device naming:
QEMU monitor expects devices to be uniquely named. Device ids derive
from the following function:
kvm_devid = <device_type>-<part of uuid>-pci-<pci_slot>
Device ids must be reproduce-able when we want to remove them.
For that reason we store the pci slot inside the runtime file and
in case we want to remove a device we obtain its pci slot by
parsing the corresponding runtime enrty and matching the device
by its uuid.

Finding the PCI slot:
For newly added devices Hypervisor parses existing PCI allocations
(via _GetFreePCISlot() and eventually ``info pci`` monitor
command) and decides the PCI slot to plug in the device. During
instance startup hypervisor invokes _UpdatePCISlots() for every
device of the instance.  Initial PCI reservations derive from KVM
default setup, that allocates 4 slots for devices other than disks
and NICs.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Use -device with pci info if possible

Extend kvm_cmd with -device option for the case of paravirual
Disks and NICs.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Helper methods that check for hotplug support

Hotplug is currently *not* supported for:

 - KVM < 1.0
 - existing devices in the cluster
 - python-fdsend module is not installed (NIC hotplug)
 - chroot (Disk hotplug)
 - security mode other than None (Disk hotplug)

For the above reasons raise HotplugError exception and
let masterd handle it.

TODO: modify CallHotplugCommand() so that it parses
monitor output and reports whether the command
succeeded or not.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Extract socket related code from QmpMonitor class

Separate unix socket related code from QmpMonitor class and
make the latter extend the newly introduced one: MonitorSocket

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Introduce hotplug methods (hypervisor inteface)

4 new methods: HotAddDevice, HotDelDevice, HotModDevice, HotplugSupported
All these are the hypervisor interface with the backend.
In general they read current runtime info, do the actual hotplug
and update the runtime file.

NIC hot-add:
 - open a tap and get its file descriptor.
 - pass fd with SCM rights (using python-fdsend) via monitor socket
 - create netdev and device with id=kvm_devid and proper pci info

Disk hot-add:
 - create drive with id=kvm_devid
 - create device with id=kvm_devid and corresponding pci info

Use MonitorSocket in _PassTapFd in order to connect to monitor socket
and pass the tap's file descriptor using fdsend.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Hotplug: rpc support

Introduce new RPC that eventually invoke hypervisor specific
hotplug functions. In order to be generic it has the following
arguments: device type, action, device, extra info, seq.
Device type can be NIC or DISK, action can be ADD, REMOVE,
device is the NIC or Disk object, extra info is used by Disk
hotplug to point the device path and seq is the device index
(from the master perspective)

Add HotplugSupported() in all Hypervisors

Only KVM hypervisor supports this method. The other hypervisors
raise HypervisorError.

Before trying hotplug in backend layer invoke hyper.HotplugSupported
and abort RPC in case it is not implemented or for some other reason
not supported (disk+chroot, nic without fdsend, etc).

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Return link_name in blockdev_assemble rpc

Until now this RPC returned only dev_path. Since we use it in
hotplug we have to know the simlink of the device so that we
pass it to the corresponding hypervisor command and include it
in block_devices entry in runtime files.

Fix unittest to be aware of changed rpc

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Hotplug: cmdlib support

Hotplugging is done by functions invoked by ApplyContainerMods(). In
order hotplugging to take place the --hotplug option must be passed
otherwise the modifications will take place after reboot.

NIC hotplug supports add, remove and modify. The modify is done by removing
the existing NIC and adding a new one in the same pci slot.

Disk hotplug supports add and remove.  Before hotplugging a Disk it
must be assembled. Use blockdev_assemble RPC, get the device link_name
and then call hotplug RPC command.

In order to remove a disk (with blockdev_remove) it must be shutdown. So
after unplugging the disk ShutdownDiskInstanceDisks() must be invoked.

For both device types we use the generic RPC call_hotplug_device.

Also adapt unit tests to follow the change.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Hotplug: client support

Add --hotplug option. Only used in OpInstanceSetParams.
If this is omitted, modifications become effective after reboot.

Ask user confirmation in case NIC modify + hotplug because it will
be done via removing old NIC (and the corresponding tap) and adding
a new one in the same PCI slot.

Corresponding mods in haskell opcode definitions.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Hotplug: Update manpages and NEWS

Add --hotplug entry in gnt-instance manpage.
Add NEWS entry for hotplug support.
In both cases mention that hotplug is only supported for
the KVM Hypervisor version >= 1.0.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Extend qa to test hotplug support

Only if default hypervisor is KVM try all hotplug related actions.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Add unittests for Hotplug support

Try base InstanceSetParams tests that modify instance's
devices with hotplug=True.

Add some unittests for kvm hypervisor regarding runtime file
manipulation. Based on a sample runtime file search for
specific disk and nic entries based on their UUID.

Add unittest for kvm device id generation.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Some fixes in _GenerateKVMBlockDevicesOptions()

* Add docstring.
* fix missed usecase of boot_disk=False if kernel_path is given.
* Use devlist arg in GenerateKVMBlockDeviceOptions()
  This is previously calculated in _ExecuteKVMRuntime()
  so there is no need to rerun the same command.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>
Make hotplug related method raise HotplugError

Hotplug is not supported by default and thus we raise
HotplugError() for hotplug related methods.
If a hypervisor wants to support it, we must override VerifyHotplugSupport()
and implement the Hot* method.

Backend invokes VerifyHotplugSupport() and raises RPCFail if an
exception is raised.

Signed-off-by: Dimitris Aragiorgis <dimara@grnet.gr>
Signed-off-by: Thomas Thrainer <thomasth@google.com>
Reviewed-by: Thomas Thrainer <thomasth@google.com>