Correct the state for PAUSED instances on reboot

If an instance is in the PAUSED state before the compute node is
restarted, it will be left in an inconsistent state (impossible to
delete and/or restart it) after the node boots again.

This patch adds a check in the periodic task sync_power_state so when
this situation is detected the instance is stopped and left in a state
from where it can be started again or deleted.

Closes-Bug: #1265494
Change-Id: I7266ace596598522c28a3839091ef50faf4463c2
VMware ESX: Boot from volume must not relocate vol

ESX does not support relocate API. Moreover when working with ESX the boot
from volume logic need not perform relocation since the volume's datastore
is always visible to the ESX host.

The fix is to ignore relocation when working with ESX.

Change-Id: I227cf2ea5b6f681150ede0bf6257eb238f1452d2
Closes-bug: #1244918
Volume operations should be blocked for non-null task state

Currently volume attach, detach, and swap check on vm_state but
not task_state.  This means that, for example, volume attach is
allowed even when other operations such as migration are in
progress.  As with other operations the check should be against
a task state of "None"

This change also refactors the volume_attach() method a little
so that the cells api can include the state decorator checks.

Change-Id: I0c4d697cf9d9e431a050c3c1b03e46b8629d182a
Closes-Bug: #1274627
Make libvirt wait for neutron to confirm plugging before boot

This makes the libvirt driver use the instance event mechanism to
wait for neutron to confirm that VIF plugging is complete before
actually starting the VM.

Here we introduce a new configuration option of vif_plugging_is_fatal
which allows us to control whether a failure to hear back from
neutron is something that should kill the current operation. In
cases where consoles are not provided for the guest, failing is
the only reasonable action, but if consoles *are* provided, it may
be advantageous to still allow the guest to boot and be accessed
out of band.

In order to properly reflect the neutron failure in the instance's
fault property, this also extends manager to catch and re-raise
the VirtualInterfaceCreateException during spawn(). Also, since
the oslo.messaging merge, we weren't declaring expected exceptions
for run_instance() which this also fixes.

DocImpact: This requires a neutron that is aware of these events
           when using nova with neutron, or a configuration that
           causes nova not to expect/wait for these events.
Related to blueprint admin-event-callback-api

Change-Id: I2f771d8bade513d2e54d98a176069011950830aa
Add version 3.0 of scheduler rpc interface

This commit adds a new major revision to the rpc interface used to
talk to the scheduler service.  The old version will be retained in
Icehouse for Havana rolling upgrade capability.  2.x can be dropped
as soon as Juno development opens.

When 2.x gets dropped, we will be able to remove the following methods
from the scheduler manager, as well as all other supporting code for
these methods throughout the scheduler:

  - create_volume()
  - live_migration()
  - show_host_resources()
  - get_backdoor_port()
  - select_hosts()

We can also drop backwards compatibility handling for the absence of
any parameters added to other methods during 2.x.

Note that the client side will only send 3.0 messages, but that's
fine.  We expect the nova-scheduler service to get updated at the same
time as, or before anything that uses this client code.

Part of blueprint rpc-major-version-updates-icehouse

Change-Id: Icca32ff1defbab06f9e9cd726b3ffa25d9fedf1e
xenapi: Fix test_rescue test to ensure assertions are valid

The assertion in this test gathers the list of VDI references and then
checks if the string 'swap' isn't in the list. A VDI reference is a
UUID, so the string 'swap' will never match, even if the code being
tested is buggy.

Compare against an actual VDI reference. Also, add a positive assertion
to check that the root disk from the original VM is attached.

Change-Id: I3e3e9798ce63f2f260b7993620fde3749452bfc9
Closes-bug: 1290903
Reverts change to default state_path

Reverting the change to the default value of state_path introduced by
I94502bcfac8b372271acd0dbc1710c0e3009b8e1 for the reasons set out
in my -1 review of the same that seems to have been skipped when the
change was accepted.

As implemented the change will break any existing systems that are using
the default value of state_path with no warning period, which goes beyond
the scope of change for UpgradeImpact

A better approach would be to introduce a new config value that is used in
the code, but is over-ridden by "state_path" if it is still in the config file,
and log a warning that state_path is deprecated and will be removed in the future.

That way the change to default behavior can be introduced at the start of the
next release cycle by removing the deprecated state_path config value, and
users have until then to prepare.

It does mean that we keep the insane default for a bit longer, but I think that's
much better that breaking a running system.

Closes-bug: #1291515

Change-Id: I7bb70c551ec9616276552d2e32076b66a1d3ce01
Make nova-network use quotas object

This makes nova-network use the quotas object for its work.

Related to blueprint nova-network-objects
Related-bug: 1290568

Change-Id: I416f63d2a4e4aa8220931ccf8f386009902e653e
Make nova-network use conductor for security groups refresh

This makes nova-network use the existing conductor interface for
triggering security groups member refreshes.

Related to blueprint nova-network-objects

Change-Id: Ieca163d67cd005534acb225df83457d755295af9
Related-bug: 1290568
Block database access in nova-network binary

This mirrors the code we have in cmd/compute.py to wedge the database
module so that we can't add any more direct calls, and will highlight
any stragglers.

Related to blueprint nova-network-objects

Change-Id: Ib12415d84de950265626b4a804f33717203d9765
Related-bug: 1290568
Merge "Make nova-network use quotas object"
Don't sync [system_]metadata down to cells on instance.save()

A cell should be authoritative for its metadata and system_metadata.
rebuild relis on system_metadata not being updated until the cell
updates it itself.  update_instance_metadata relies on metadata not
being updated so that it can calculate the diff properly.

And before the objects conversion took place the compute.api.update()
method that instance.save() replaced was not used for syncing metadata
or system_metadata.  So this change goes back to previous behaviour for
cells.

Change-Id: I339da9d1ff6d0e59f6a4ab68fd396bdfca7a224b
Closes-bug: #1292181
Closes-bug: #1292185
Merge "xenapi: Fix test_rescue test to ensure assertions are valid"
Fix input validation for V2 API server group API extension

The server group extension create method allows the name
parameter to not be a string. The unittest which tests that an integer
passed as the name fails, but this is due to the policies parameter
missing from the test data, not because the name is invalid.

The internal nova code silently converts floats, integers, etc to strings
and so also does not pick up the validation issue.

The patch adds extra input validation to the API layer to pick up this issue
and also tightens the acceptable values of the string to reject trailing
or leading whitespace which if used can cause confusion for users. It uses
the regexp that the other parts of the API are converging towards to use
as a common acceptable name across the API rather than having different
criteria for names for different extensions.

Also ensures input validation is always properly applied to the policies
parameter and adds test cases. This part is backported from this
https://review.openstack.org/#/c/79312/5 patch.

The patch also fixes the broken unittests and adds new ones to test the
tightened criteria for the name parameter.

+DocImpact
Co-authored-by: Gary Kotton <gkotton@vmware.com>
Closes-Bug: 1290326

Change-Id: I9ddaef8a128ac6a1a448e47d6b3c9abcf719eb76
Merge "Make nova-network use conductor for security groups refresh"
Merge "Block database access in nova-network binary"
Move log statement to expose actually info_cache value

When debugging an info_cache issue I noticed the logs  display:

"Updating cache with info: None update_instance_cache_with_nw_info"

Though we are not actually updating the info_cache with None as if
nw_info is not isinstance(NetworkInfo we fetch the latest nw_info
from neturon. This patch just moves the log statement so the actual
value is exposed.

Change-Id: I3a74191c31a053682bb2558c59228025ceb0cf28
Closes-bug: 1292298
Imported Translations from Transifex

Change-Id: Ie9e29d4ee35fc224f9e2f114dfdc6c15ad708e81
Merge "Volume operations should be blocked for non-null task state"
Merge "Fix input validation for V2 API server group API extension"