Correct the state for PAUSED instances on reboot

If an instance is in the PAUSED state before the compute node is
restarted, it will be left in an inconsistent state (impossible to
delete and/or restart it) after the node boots again.

This patch adds a check in the periodic task sync_power_state so when
this situation is detected the instance is stopped and left in a state
from where it can be started again or deleted.

Closes-Bug: #1265494
Change-Id: I7266ace596598522c28a3839091ef50faf4463c2
VMware ESX: Boot from volume must not relocate vol

ESX does not support relocate API. Moreover when working with ESX the boot
from volume logic need not perform relocation since the volume's datastore
is always visible to the ESX host.

The fix is to ignore relocation when working with ESX.

Change-Id: I227cf2ea5b6f681150ede0bf6257eb238f1452d2
Closes-bug: #1244918
Make libvirt wait for neutron to confirm plugging before boot

This makes the libvirt driver use the instance event mechanism to
wait for neutron to confirm that VIF plugging is complete before
actually starting the VM.

Here we introduce a new configuration option of vif_plugging_is_fatal
which allows us to control whether a failure to hear back from
neutron is something that should kill the current operation. In
cases where consoles are not provided for the guest, failing is
the only reasonable action, but if consoles *are* provided, it may
be advantageous to still allow the guest to boot and be accessed
out of band.

In order to properly reflect the neutron failure in the instance's
fault property, this also extends manager to catch and re-raise
the VirtualInterfaceCreateException during spawn(). Also, since
the oslo.messaging merge, we weren't declaring expected exceptions
for run_instance() which this also fixes.

DocImpact: This requires a neutron that is aware of these events
           when using nova with neutron, or a configuration that
           causes nova not to expect/wait for these events.
Related to blueprint admin-event-callback-api

Change-Id: I2f771d8bade513d2e54d98a176069011950830aa
Add version 3.0 of scheduler rpc interface

This commit adds a new major revision to the rpc interface used to
talk to the scheduler service.  The old version will be retained in
Icehouse for Havana rolling upgrade capability.  2.x can be dropped
as soon as Juno development opens.

When 2.x gets dropped, we will be able to remove the following methods
from the scheduler manager, as well as all other supporting code for
these methods throughout the scheduler:

  - create_volume()
  - live_migration()
  - show_host_resources()
  - get_backdoor_port()
  - select_hosts()

We can also drop backwards compatibility handling for the absence of
any parameters added to other methods during 2.x.

Note that the client side will only send 3.0 messages, but that's
fine.  We expect the nova-scheduler service to get updated at the same
time as, or before anything that uses this client code.

Part of blueprint rpc-major-version-updates-icehouse

Change-Id: Icca32ff1defbab06f9e9cd726b3ffa25d9fedf1e
Reverts change to default state_path

Reverting the change to the default value of state_path introduced by
I94502bcfac8b372271acd0dbc1710c0e3009b8e1 for the reasons set out
in my -1 review of the same that seems to have been skipped when the
change was accepted.

As implemented the change will break any existing systems that are using
the default value of state_path with no warning period, which goes beyond
the scope of change for UpgradeImpact

A better approach would be to introduce a new config value that is used in
the code, but is over-ridden by "state_path" if it is still in the config file,
and log a warning that state_path is deprecated and will be removed in the future.

That way the change to default behavior can be introduced at the start of the
next release cycle by removing the deprecated state_path config value, and
users have until then to prepare.

It does mean that we keep the insane default for a bit longer, but I think that's
much better that breaking a running system.

Closes-bug: #1291515

Change-Id: I7bb70c551ec9616276552d2e32076b66a1d3ce01
Don't sync [system_]metadata down to cells on instance.save()

A cell should be authoritative for its metadata and system_metadata.
rebuild relis on system_metadata not being updated until the cell
updates it itself.  update_instance_metadata relies on metadata not
being updated so that it can calculate the diff properly.

And before the objects conversion took place the compute.api.update()
method that instance.save() replaced was not used for syncing metadata
or system_metadata.  So this change goes back to previous behaviour for
cells.

Change-Id: I339da9d1ff6d0e59f6a4ab68fd396bdfca7a224b
Closes-bug: #1292181
Closes-bug: #1292185
Fix input validation for V2 API server group API extension

The server group extension create method allows the name
parameter to not be a string. The unittest which tests that an integer
passed as the name fails, but this is due to the policies parameter
missing from the test data, not because the name is invalid.

The internal nova code silently converts floats, integers, etc to strings
and so also does not pick up the validation issue.

The patch adds extra input validation to the API layer to pick up this issue
and also tightens the acceptable values of the string to reject trailing
or leading whitespace which if used can cause confusion for users. It uses
the regexp that the other parts of the API are converging towards to use
as a common acceptable name across the API rather than having different
criteria for names for different extensions.

Also ensures input validation is always properly applied to the policies
parameter and adds test cases. This part is backported from this
https://review.openstack.org/#/c/79312/5 patch.

The patch also fixes the broken unittests and adds new ones to test the
tightened criteria for the name parameter.

+DocImpact
Co-authored-by: Gary Kotton <gkotton@vmware.com>
Closes-Bug: 1290326

Change-Id: I9ddaef8a128ac6a1a448e47d6b3c9abcf719eb76
Move log statement to expose actually info_cache value

When debugging an info_cache issue I noticed the logs  display:

"Updating cache with info: None update_instance_cache_with_nw_info"

Though we are not actually updating the info_cache with None as if
nw_info is not isinstance(NetworkInfo we fetch the latest nw_info
from neturon. This patch just moves the log statement so the actual
value is exposed.

Change-Id: I3a74191c31a053682bb2558c59228025ceb0cf28
Closes-bug: 1292298
Imported Translations from Transifex

Change-Id: Ie9e29d4ee35fc224f9e2f114dfdc6c15ad708e81
Merge "Fix input validation for V2 API server group API extension"
Merge "VMware ESX: Boot from volume must not relocate vol"
Remove aggregate_get() from conductor API

The aggregate_get() method of the conductor API is no longer used.
Remove it from the client code and leave a note on the manager side
for it to be removed in the next major version bump.

This is cleanup, but part of a blueprint being completed during
feature freeze.  It allows us to drop more code in the rpc version
bump.

Part of blueprint rpc-major-version-updates-icehouse

Change-Id: I893f58bbcae21112f186348cefcbfdd3bcbb8c06
Add security_group_rule to objects registry

This adds the security_group_rule module to the objects registry,
which allows a service to make sure that all of its objects are
registered before any could be received over RPC.

We don't really have a test for any of these because of the nature
of how they're imported. Refactoring this later could provide some
incremental steps to making this more testable.

Change-Id: Ie96021f3cdeac6addab21c42a14cd8f136eb0b27
Closes-Bug: #1264816
add support for host driver cleanup during shutdown

Provides a place for any drivers to potentially
setup graceful-shutdown code.

VMware driver's proper driver-lifecycle code included.
This is critical in environments where the the vmware
driver is setup and torn down at high frequency.

Prevents run-away vSphere by closing stateless
HTTP management sessions gracefully.

Change-Id: I67a91613643540243ab1210b333ed8e121f05802
related to bug: 1262288
Closes-bug: 1292583
Remove aggregate_get_by_host() from conductor API

The aggregate_get_by_host() method of the conductor API is no longer
used.  Remove it from the client code and leave a note on the manager
side for it to be removed in the next major version bump.

This is cleanup, but part of a blueprint being completed during
feature freeze.  It allows us to drop more code in the rpc version
bump.

Part of blueprint rpc-major-version-updates-icehouse

Change-Id: Ib28fab80adbe796ecaded040d4c33ea05659e509
Remove instance_get() from conductor API

The instance_get() method of the conductor API is no longer
used.  Remove it from the client code and leave a note on the manager
side for it to be removed in the next major version bump.

This is cleanup, but part of a blueprint being completed during
feature freeze.  It allows us to drop more code in the rpc version
bump.

Part of blueprint rpc-major-version-updates-icehouse

Change-Id: Id2ed674c2d4c491bc7b0b0c79a64bef84fb05db4
Remove migration_update() from conductor API

The migration_update() method of the conductor API is no longer
used.  Remove it from the client code and leave a note on the manager
side for it to be removed in the next major version bump.

This is cleanup, but part of a blueprint being completed during
feature freeze.  It allows us to drop more code in the rpc version
bump.

Part of blueprint rpc-major-version-updates-icehouse

Change-Id: Id385255cbfff8c7a42a9a1551044746b7f6344bd
Fix equal_any() DB API helper

equal_any() query helper is meant to produce a WHERE clause combining
a few conditions with OR operator in order to select the rows for
which the given column is equal to ANY of passed values. In fact,
it uses AND operator instead of OR, which means that an incorrect
WHERE clause will be returned, if more than one value is passed to
equal_any().

This is caused by the fact that the or_() function from SQLAlchemy
expression language is used incorrectly: it accepts unpacked
instances of BinaryExpression, but a list of ones is passed.
Unfortunately, in 0.7.x and 0.8.x branches of SQLAlchemy passing
a list of values to or_() won't raise an exception immidiately,
and the result will be an iterable, which explains why we didn't
see equal_any() was actually broken before. Besides, equal_any() is
rarely used in our code base and the test didn't cover the case when
more than 1 value is passed to the function.

This effectively fixes the error when running Nova unit tests
with SQLAlchemy 0.9.x releases.

Closes-Bug: #1292285

Change-Id: If4c0f1d3e016e2affb0d49293c62ca9df0d033f7
Merge "Make libvirt wait for neutron to confirm plugging before boot"
Fix sqlalchemy utils test cases for SA 0.9.x

To be able to run migration scripts on SQLite we've been providing
a few workarounds for sqlalchemy-migrate and SQLAlchemy 0.7.x and
0.8.x branches in the form of sqlalchemy utils module. In 0.9.x
releases a few workarounds become obsolete (due to improvements made
to schema reflection for SQLite in SA), but our test cases still
expect SA to raise exceptions in several cases.

As long as our utils can work well with 0.9.x SA releases with our
workarounds and without them, we can just skip the test cases
expecting exceptions to be raised when SA version > 0.9.0 is used
and continue to run them for older releases of SA.

Closes-Bug: #1292284

Change-Id: Ie6cdb017b64b393cff43957b2047b5aae142d54e