Fix double reboot issue during soft reboot

Using the ID of domain in Libvirt to recognize the hard reboot has been
implemented or not, if the ID changed, we believe the domain has been rebooted,
return True and break from soft reboot.

Fixes: bug #1046356

Change-Id: Iec2f9e8225cfe2779f84d2095667f3c0e621e935
libvirt: use tap for non-blockdevice images on Xen

This patch reverts 35c4962c0b97bae5b8751d316d5822fe22c1ab6a: 'use
file instead of tap for non-blockdevice images on Xen', which breaks
qcow2 disk working on Xen. If the method pick_disk_driver_name()
returns 'file' for qcow2 format, we would get the following disk XML
snippet:

    <disk type="file" device="disk">
      <driver name="file" type="qcow2" cache="none"/>
      <source file="/data/instances/instance-00000081/disk"/>
      <target bus="xen" dev="hda"/>
    </disk>

which produces configuration in the xen tools like:

    "file:/data/instances/instance-00000081/disk,hda,w"

Guest would not boot successfully, as Xen doesn't think of disk format
as qcow2. Instead, 'tap' works with both raw and qcow2 disk.

For LP#1084618, I think it is related with libvirt verion. In my
deployment(libvirt-1.0.1), 'tap' dirver could produce correct
configuration for both raw and qcow2 disk format.

Change-Id: I636b2fed366474e2ae8c3b52403e3085724b24f0
Add rootwrap filters for password injection with localfs

Allow to 'sudo cat' to read passwd and shadow.

bug 1098077

Change-Id: Ic734bd33223df879b5e1f144bb4c85702eb88dfa
ensure zeros are written out when clearing volumes

Note O_DIRECT is _not_ used at the end of devices to
avoid issues with odd sized blocks etc. so instead
we arrange to have dd issue an fdatasync()
to ensure the data is persisted, lest it be discarded
from the write cache when the device is unprovisioned.

* nova/virt/libvirt/utils.py (clear_logical_volume): Add 'conv=fdatasync'
to the dd option list if O_DIRECT isn't used when clearing.

Fixes bug: 1100363
Change-Id: I76789557754ebaeb6d52bb34548a2ef17808fbf6
Cells: Add support for compute HostAPI()

This will allow the hosts API extension to work. Calls to get service
information from the DB are proxied via cells.

When using the host extension, the 'id' and 'host' fields for
service/compute will be modified to have the cell prepended to it.  One
can do a 'show' on a 'path!to!cell@id' to see details for a particular
host in the 'path!to!cell' cell, for instance.

Implements bp/nova-compute-cells

Change-Id: I8137fa64d2578acfc3d2457c231de2b0d69b96d8
Update en_US message translations.

Fixes bug 1101587.

* nova/locale/en_US/LC_MESSAGES/nova.po: This "translation" was
woefully outdated, causing KeyError exceptions under a number of
circumstances for users of this particular locale when format
strings wound up containing incorrect substitution variables. This
change attempts to sync up the msgstr entries to match msgid given
this is the source language for the pot file. I did not see any
obvious instances where en_US should differ from C locale for these.

Change-Id: I81d1729fee17653b089f3c47db20f3fc073ab363
Generate coverage even if tests failed.

A test failing doesn't make coverage useless, so we should always
generate it.

Change-Id: I3ed9e54213c7fb6756906553920d19c5075e1adb
Add support for Option Groups in LazyPluggable

Use @markmc's suggestion to enhance LazyPluggable with an
optional config group. Also fix the baremetal database
backend option to use the "baremetal" config group.

Fixes LP #1093043

Change-Id: I28cf51a2962f516fcef4ced19e30c985220e86dc
Fix floating ips with external gateway

If dnsmasq is configured to use an external gateway, then floating
ips on other interfaces do not work properly. This is because
outgoing traffic is no longer snatted to the floating ip.

This patch fixes it by adding an ebtables rule to force traffic
from ips that have a floating ip associated to route instead
of bridge.

Fixes bug 1096985

Change-Id: I8e4904660d42fe51c44b66686bed9f5d622693bd
PXE driver should rmtree directories it created

Baremetal PXE driver was failing to delete the per-instance tftpboot and
image directories which it created when the instance was deleted. This
happened partly because of dangling files within the directory, and
partly because 'unlink' does not remove directories and the error was
squelched.

Now, when destroy()ing an instance, PXE driver will call shutil.rmtree()
on the per-instance directories which it created.

Fixes bug 1101048.

Change-Id: I682d0264288add488ea23e5d5200357b7be52dd9
Baremetal/utils should not log certain exceptions

unlink_without_raise was logging exceptions when it failed to unlink a
file, which is confusing since it is often called on non-existing files.

create_link_without_raise was also logging exceptions when it failed
to create a symlink.

This patch corrects this behaviour; both functions now explicitly check
for the type of error they expect, and then suppress that. If another
type of OSError is encountered, a warning is logged. This patch also
adds unit tests for both functions.

fixes bug 1097931.

Change-Id: Ie36f59dce34a6c67765770c5f968c003003acc88
Add version constraint for cinder

Cinder client usage requires the "retries" parameter introduced in
cinderclient-1.0.1. This should be a constraint in pip-requires.

Change-Id: I36e8a363c9d4bb7cc4ecfda85644fc4284f17471
ensure failure to inject user files results in startup error

This was the case in Essex but was inadvertantly changed in:
folsom-2-95-g0d166ca.

* nova/virt/disk/api.py: Refactor to allow specifying
mandatory injection items, that result in an exception
on failure to inject.
* nova/virt/libvirt/driver.py: Specify that user 'files'
are mandatory items and thus result in VM startup failure
unless injected successfully.
* nova/tests/test_virt_disk.py: A new test for the
separate warning and error cases.

Fixes bug: 1095744
Change-Id: Idab5c4294c1cb52098ce44a7aae957a44fb2674f
Avoid stuck task_state on snapshot image failure

Fixes bug LP 1101136

Previously if the glance interaction failed prior to an
instance being snapshot'd or backed up, the task state
remained stuck at image_snapshot/backup.

The normal task state reversion logic did not kick in,
as this is limited to the compute layer, whereas the
intial glance interaction occurs within the API layer.

Now, we avoid this problem by delaying setting the task
state until the initial initial image creation/retrieval
has completed.

Change-Id: Id498ae6b3674306743013e4fe99837da8e2031b5
Add support for memory overcommit in live-migration

Override the live-migration memory check in FilterScheduler and
make it use RamFilter for checking available memory on target
host.

Resolves bug 1068258.

Change-Id: I3002b8330e595ce71e2718b2091d3b0b99747707
Fix double reboot during resume_state_on_host_boot

Re-basing my commit introduced a mistake I did not catch before the
merge occurred.  resume_state_on_host_boot is calling both
_create_domain_and_network followed by _hard_reboot. It should only
call _hard_reboot.

Change-Id: I52be2c0d0c85abe6386842c980f26b3c5713e3e3
Add version constraint for coverage

Python coverage lower than 3.6 causes issues when running the whole
test suite. Some arguments are randomly mixed up with script names,
resulting in unittest failures like this:

UnexpectedMethodCallError: Unexpected method call.  unexpected:-  expected:+
- exists.__call__('/tmp/buildd/.../nova/virt/images.py') -> None
+ exists.__call__('/some/path') -> True

Change-Id: Iadb6d046e3acc75c6ed5abffcfc1abda82aeee4f
Provide creating real unique constraints for columns

Main issue is that we can't create unique constraint for columns, because
we are using soft deletion of entries (set `deleted` column to True).

The main idea is to use `deleted` columns to create unique constraint for
columns. For example (`col1`, `deleted`).

To make (`col1`, `deleted`) unique after entry deletion, we should assign the
value of `id` to `deleted` column.

Change type of `deleted` column from Boolean to table.id.type for all tables.

Change models.soft_delete() method to assign table.id instead of True to
`deleted` column.

Change query.soft_delete() method to assign literal_column("id") instead of True

Change in db.models all occurrences of Table.deleted == False =>
Table.deleted == correct_type (0 or "")

Value of `deleted` property of entries is used in public nova api. To keep API
as is, we should change XMLDictSerializer by converting the type of
`deleted` property to string representation of boolean.

Change db.api.model_query() method to make it work with different types
of `deleted` column.
If we are using as model that is not subclass of NovaBase we should set
parameter base_model as subclass of NovaBase that corresponds to model.

Change in db.api all occurrences of using model_query with model that is not
subclass of NovaBase.

blueprint db-unique-keys

Change-Id: Ie1f67f49a5d085e6371efb63fc23a1c8b25d9464
Remove compute api from scheduler driver

Remove unused instance of compute api and belonging import statement
from scheduler driver.

Change-Id: Ice0c3e7555cf4fb00a387c6cc9789515569bea9c
Bump the base rpc version of the network api to 1.7.

Change I2627e8b0f45c5c39bbbf1a4a12b448e1992fa4b4
bumped the RPC version of the API to 1.7
but did not bump the RPC version in the network
manager.

Change-Id: I1c719a1458ef5892e90aabee92a8b5e93cfa11c8